Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 11 min read

AI Gateway Performance Comparison (2026): Python vs Go vs Rust vs Envoy, and Why the Benchmarks Disagree

Published charts that crown a fastest AI gateway usually claim a lead by a factor of forty. They disagree with each other because they do not measure the same interval. One number is time spent inside the process, with J

Published charts that crown a fastest AI gateway usually claim a lead by a factor of forty. They disagree with each other because they do not measure the same interval. One number is time spent inside the process, with JSON and the network left out. Another is added latency at the client, with logging and spend tracking off. A third turns production features on, uses a slower mock, or runs a single pass of a few thousand requests. Those are not competing answers to one question. They are answers to different questions, and each vendor's published run tends to put that vendor first. None of that requires dishonesty. It requires a method, and a decision about what to measure, before any product gets a paragraph.

Claims below were checked against project repos and docs as of November 2026. Every benchmark figure is labeled with who ran it. No new numbers were added for this piece.

What a useful measurement actually records

Measure proportions before you measure products. A hosted chat completion usually takes hundreds of milliseconds, and often several seconds. A long stream takes longer still. Next to that, a gateway that adds 1 ms and a gateway that adds 5 ms are the same request.

Three cases make the overhead visible.

  1. Agent loops. Thirty model and tool calls in a row pay the gateway thirty times, and each payment is on the critical path.
  2. The tail, not the median. Median overhead is rarely what pages anyone. p99 at peak concurrency is, because a CPU-bound runtime queues there.
  3. The replica count. Ten replicas where another runtime needs one is a monthly bill for compute, memory, and on-call surface, even if no user can feel the latency.

So the question worth answering is narrower than "which gateway is fastest." How much CPU and memory does this gateway need to hold your p99 at your peak concurrency, with the features you will actually enable?

Where the time goes

Ignore the product name and the per-request work is roughly the same. A benchmark that skips a step is a benchmark of a different system. Time these eight:

  1. Key. Check the caller's gateway key.
  2. Decode. Parse the body. The usual formats are OpenAI, Anthropic, Responses, and Gemini.
  3. Model. Turn aliases and routes into a concrete model and provider.
  4. Limits. Apply budgets and rate limits.
  5. Provider key. Choose the upstream credential and decrypt it.
  6. Encode and send. Rewrite the body into the provider's format and open the upstream call.
  7. Reply. Rewrite the response, including a token stream.
  8. Log. Store tokens, cost, and metadata.

The language shows up most in decode, encode, and reply: JSON parsing, translation, and stream relay, all of it CPU. Auth, limits, and logging are where deployments actually get slow, because those steps often touch a database or Redis. A synchronous Postgres write on every request will lose to a slower language that logs after the fact. Architecture beats the runtime more often than a chart will admit.

What matters more than the language

Calls on the hot path. Write down every network hop before the provider: a key lookup in a database, a rate-limit check in Redis, a policy service. Then ask what a slow or dead hop does. Fail open, fail closed, or hang?

Whether logging blocks the reply. Synchronous request logging makes the log store's write time your latency. Confirm that property on the gateway you run before you argue about languages.

Memory per replica. Kubernetes packs pods by memory request. A process that idles in the tens of megabytes and a process that idles in gigabytes do not have the same fleet cost, even at the same latency.

State shared by replicas. Budgets and rate limits have to agree across three replicas. The store for that state (Redis, Postgres, an Envoy rate-limit service) is often the throughput ceiling.

What a rollback costs. A single static binary rolls back in seconds. A stack with schema migrations does not. In an incident, that difference matters more than 2 ms of overhead.

A k6 method you can rerun

A lab is unnecessary. You need a mock that acts like a provider, a load generator, and the discipline to turn on the features production will actually use.

1. A mock with delay and a stream. Any small server that answers /v1/chat/completions will do, if it can wait a configurable delay (start at 300 ms) and emit SSE chunks. Put it on a different machine from the gateway.

2. A k6 script that streams and varies concurrency.

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  scenarios: {
    ramp: {
      executor: 'ramping-vus',
      stages: [
        { duration: '2m', target: 50 },
        { duration: '3m', target: 300 },
        { duration: '3m', target: 1000 },
      ],
    },
  },
  thresholds: { http_req_failed: ['rate<0.01'] },
};

const body = JSON.stringify({
  model: __ENV.MODEL || 'mock-model',
  stream: true,
  messages: [{ role: 'user', content: 'Summarize this paragraph in one line.' }],
});

export default function () {
  const res = http.post(`${__ENV.GATEWAY_URL}/v1/chat/completions`, body, {
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${__ENV.GATEWAY_KEY}`,
    },
  });
  check(res, { 'status is 200': (r) => r.status === 200 });
}

3. Baseline, then gateway. Hit the mock with the same script, then put each gateway in front of it. Publish added latency, gateway minus baseline, at p50, p95, and p99 for every concurrency stage.

4. Repeat with production features on. Per-key auth, budgets, rate limits, request logging, and at least one fallback route. The gap between that run and the features-off run is usually the number that matters.

5. Write down the machines, not just the latency. CPU cores and peak resident memory for each gateway replica, plus the dependencies it drags along (Postgres, Redis, ClickHouse). Divide throughput by total cores if you want an efficiency figure that is fair.

6. Three runs, and publish the config. Numbers you cannot accompany with a config should not be trusted. Do not trust them.

Published numbers, and who ran them

This is every public figure found in the materials checked for this piece, with the context the headline usually leaves out.

Claim Who ran it Setup What to keep in mind
Bifrost adds ~11 Β΅s per request at 5,000 RPS Maxim (Bifrost's maker) AWS t3.xlarge; Bifrost's internal timing Excludes JSON marshalling and the HTTP call; peak memory reported at ~3.3 GB in the same run
LiteLLM Rust p99 added ~0.7 ms vs Portkey 2.3 ms, Bifrost 4.5 ms, LiteLLM Python 257.7 ms LiteLLM (July 2026) Local Rust mock upstream, one host, logging and spend tracking off Single runs, no error bars; the author says to read it as order-of-magnitude
Median added latency at 50 RPS: LiteLLM Rust 0.9 ms, Bifrost 1.2 ms, Portkey 2.1 ms, LiteLLM Python 79.3 ms LiteLLM (Nov 2026) Production features on, LiteLLM Rust from a feature branch Portkey ran without the listed features because it lacks them
Bifrost and LiteLLM's Rust beta tie on throughput (~2,720 RPS at concurrency 64); Python ~180 RPS Maxim (Bifrost's maker) Shared test files, 2,000 requests per cell Bifrost ran with logging to SQLite; LiteLLM with logging off. A second run reversed the p99 order
Kong "over 200%" faster than Portkey and "over 800%" faster than LiteLLM Kong (July 2025) EKS, WireMock mock, proxy-only, no plugins, Kong 3.10 vs Portkey 1.9.19 vs LiteLLM 1.63.7 Old versions; no auth, caching or limits enabled
Kong ~15,900 req/s at 1,000 VUs; Bifrost failed above 300 VUs; LiteLLM capped ~175 req/s; Portkey plateaued ~850 req/s Author of Ferro Labs, a competing Go gateway One 8-vCPU VM, 60 ms mock, default configs, no tuning Author has a stake and says so; Bifrost v1.0.0 was tested, now well behind current releases
Kaeso Nobody yet n/a No published benchmarks. An open benchmark, Gatebench, is in progress with no results

Read the table a second time and the pattern is obvious. The publisher of each row finishes first in that row. Bifrost's run favors Bifrost. LiteLLM's favors the Rust build. Kong's favors Kong. Ferro Labs' favors Ferro Labs. The rows are measuring different things, which is why they can all be careful and still disagree.

Why the rows are not comparable

  • Inside the process, or at the client. Bifrost's 11 Β΅s is time inside its own code. JSON and the network are excluded. The other rows measure added latency as the client sees it. Those units do not convert.
  • Features stripped out. Logging, spend tracking, and auth are exactly the database and Redis calls that dominate a real deployment. Turn them off and you have timed a different product.
  • Untuned defaults. Pool sizes, worker counts, and buffer sizes decide whether a gateway falls over at 300 concurrent users or at 3,000. A default-config run tests the defaults.
  • What the mock does. An instant mock rewards parsing speed. A mock that waits 60 ms or more rewards the ability to hold thousands of open upstream connections.
  • Streams or a single JSON body. Several published tests skip streaming. Most production LLM traffic streams, and relaying server-sent events chunk by chunk stresses a runtime differently than one JSON reply does.
  • How thin the tail sample is. A lot of these are one run of a few thousand requests. A p99 taken from 2,000 requests rests on about 20 points.

How each runtime spends the overhead

The method above is the comparison. The runtimes below are what you are actually deploying when you pick a project. They explain fleet cost and tail behavior. They do not crown a winner the table already refused to crown.

Python, with a Rust core growing inside LiteLLM

Stock CPython runs bytecode for one thread at a time inside a process, so a Python gateway grows by adding worker processes, not by sharing a core. LiteLLM's sizing guide is the concrete number: about 100 combined RPS per four-worker pod. On that same pod, at 200 RPS, the measured p99 overhead rose to roughly 650 ms. The 3,000 RPS showcase used 33 pods, each with 4 vCPU and 16 GiB. That is the cost of a CPU-heavy proxy in Python, not a verdict on the engineers.

The project is leaving that cost behind in stages. Since v1.110.0-rc.1 the router always runs on a Rust core, and the Python router is gone. A Rust server for /v1/messages is in beta. The roadmap aims to have that endpoint in Rust end to end by December 1, 2026. Other endpoints stay on Python for now. Call it Python with a growing Rust core. Do not call it pure Python.

Node.js and TypeScript: Portkey Gateway, LLM Gateway

The Node event loop is one thread, and it is good at concurrent input and output. JSON parsing and stream rewriting occupy that same thread. Once the CPU is busy, every request waits on every other request. Portkey's README advertises "<1ms latency" and a 122 kb footprint. Cloudflare Workers is also a supported target, which hands the scaling problem to the platform.

Go: Bifrost, Agent Router's processor, Higress's control plane

Goroutines spread work across every core in a single process. The garbage collector is tuned for short pauses. You get one binary, and vertical scaling is straightforward. The costs are memory from the collector and, in one published run, sensitivity to how the connection pool is sized once concurrency is high. That run is the Ferro Labs row above. The author of that test has a stake, and the Bifrost build under test, v1.0.0, is well behind current releases. Use the pool point as a tuning warning, not as a ranking.

Lua on NGINX and OpenResty: Kong, Apache APISIX

Plugin code in Kong and APISIX runs as LuaJIT inside NGINX's event-driven workers. API gateways have lived in that model for a long time, and the performance shape is well understood. For model traffic the open question is how much of the AI-specific work stays cheap in Lua, and, on Kong, which of those plugins the license lets you run.

Envoy's C++ data plane plus an external processor

Agent Router, the project formerly called Envoy AI Gateway, leaves connection handling on Envoy's C++ data plane and puts the AI work in a Go external processor: provider routing, format translation, token accounting. A separate rate-limit service enforces token-based limits. You inherit Envoy's networking, and you pay an extra call between Envoy and the processor on every request. Higress is a cousin of that design: Envoy, plus Wasm plugins written in Go, Rust, or JS.

Rust: agentgateway, AISIX OSS, Kaeso, and LiteLLM's new core

No garbage collector, memory use you can predict, and async input and output on Tokio-style runtimes. For a gateway the interesting part is the tail. Without collector pauses, and with little allocation per request, p99 stays near p50, and each replica stays small in memory. agentgateway, a Linux Foundation project, is Rust. So is API7's AISIX OSS, which ships as one static binary. LiteLLM's new core is Rust too. The language is not a maintenance plan. TensorZero's repository was archived on June 12, 2026. Helicone's standalone AI gateway has had no commit since November 2025.

Kaeso, with no benchmark to cite

The Kaeso gateway is one Rust process. The console is TypeScript. Its docs describe the request path as the eight ordered checks above. Logs go to ClickHouse after the reply has already been sent, so the write is off the critical path and does not hold the response. Budgets and limits are evaluated before any provider is called. None of those design choices is a benchmark. No independent numbers have been published, and Gatebench, the open benchmark, has no results to cite.

Language and footprint have to be read together. The default install is Docker Compose: Caddy, the gateway, the console, PostgreSQL, ClickHouse, and Valkey, about a 2 GB download. A Helm chart exists for Kubernetes. A small Rust process beside three stateful services is not a single static binary. The fleet bill includes the stateful services.

FAQ

What is the fastest AI gateway?

As of November 2026 there is no independent answer. Bifrost, LiteLLM's Rust build, Kong, and others each publish a setup in which they win. Run your own traffic, with the features you will leave on.

Is Rust faster than Go for an AI gateway?

LiteLLM's published tests and Maxim's published tests put a Rust gateway and a Go gateway close on throughput. The tail-latency order flipped between runs. Rust's clearer edge is memory, and the absence of garbage-collection pauses. Architecture still outweighs the language.

Why is LiteLLM slower than Go or Rust gateways?

The proxy grew up in Python. It scales by adding worker processes, and parsing plus translation cost more CPU per request. That is why the hot paths are moving onto a Rust core.

Does an AI gateway add noticeable latency?

On most single requests, no. The provider is slower by a wide margin. You notice the gateway in a long agent loop, and at peak concurrency, when a CPU-bound process watches its p99 climb.

Does Kaeso have benchmarks?

Not yet. Nothing has been published, and Gatebench is still in progress with no results.

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.