I Tried Respan Gateway: An Open-Source LLM Gateway Written in Rust
If you've built anything on more than one LLM provider, you know the drill. OpenAI's SDK here, Anthropic's there, a wrapper to make them look alike, and retry logic you wrote at 2am and don't fully trust. An LLM gateway
If you've built anything on more than one LLM provider, you know the drill. OpenAI's SDK here, Anthropic's there, a wrapper to make them look alike, and retry logic you wrote at 2am and don't fully trust. An LLM gateway is supposed to take that away: one endpoint, any model, and the plumbing handled for you.
Respan Gateway is a new open-source one, written in Rust and public since October. I installed it on my Windows PC, pointed it at Gemini, broke a provider on purpose, and checked whether it does what it says. Everything below is from that run: real commands, real timings, real screenshots.
What Respan Gateway is
Respan Gateway is a single binary that sits between your code and AI APIs. You point the SDK you already use (OpenAI, Anthropic or Gemini) at it, ask for any model, and it translates the request into whatever format that model's provider speaks.
What stood out before I ran it:
- Everything is a plugin. Providers, routing, pricing, MCP and tracing are all built on the same public Rust API you can use to write your own.
- 16 providers built in, including OpenAI, Anthropic, Gemini, xAI, DeepSeek, Mistral, Groq, OpenRouter, Ollama and vLLM, plus any OpenAI-compatible endpoint.
- MIT or Apache-2.0, self-hosted, free.
- A built-in web UI with a live request log, a playground and a route debugger.
Installing it on Windows (and the first snag)
The README's quickest path is pip install respan-gateway, which ships the compiled binary. On a Linux box that took about 3 seconds. You can also use cargo install, Docker, or a release binary.
On Windows I tried the official one-line installer first:
irm https://raw.githubusercontent.com/respanai/respan-gateway/main/install.ps1 | iex
It failed with respan-gateway-v0.3.0-x86_64-pc-windows-msvc.zip is not in SHA256SUMS. The zip is in that file. The script's checksum pattern expects one space between the hash and the file name, and the file uses two (the standard sha256sum format), so the match never succeeds. I downloaded the zip from the v0.3.0 release, checked its SHA-256 against the published one myself, and ran the .exe directly. Small bug, easy fix, but it's the first thing a Windows user hits.
My config
You can start with no config at all: export a provider key and run respan-gateway. For this test I wanted a fallback route I could break on purpose, so I wrote a small gateway.toml. It has Gemini as the real provider, plus a provider called down that points at a local port where nothing is listening:
[providers.gemini]
vendor = "gemini"
keyΒ Β Β = "env:GEMINI_API_KEY"
# Nothing listens on port 9: this provider is "down" on purpose
[providers.down]
formatsΒ = ["openai-chat"]
base_url = "http://127.0.0.1:9/v1"
# Any bare "gemini-*" model tries the dead provider first, then Gemini
[routes."gemini-*"]
toΒ Β Β Β Β = ["down/{model}", "gemini/{model}"]
triesΒ Β = 2
backoff = "200ms"
The key sits in a .env file next to the config, which the gateway reads on startup. Before starting, respan-gateway check validates the file and prints exactly what it will serve: providers, endpoints, routes and whether auth is on. That saved me a guess-and-restart loop.
The same config shows up in the web UI, with secrets hidden:
One default I liked: with no client keys configured, the gateway only answers on loopback and refuses to start on a public address. You can't accidentally expose your provider keys by forgetting a setting.
Any SDK, any model
This is the headline feature, so I tested it first. I sent the same prompt to gemini/gemini-3.5-flash-lite twice: once in Anthropic Messages format (POST /v1/messages) and once in OpenAI Chat format (POST /v1/chat/completions). Both came back correct, in 1.03s and 1.52s.
The gateway turned the Anthropic-format request into a Gemini call and translated the answer back, including usage in Anthropic's shape (input_tokens, output_tokens). The request detail in the UI shows it plainly: format anthropic-messages, provider gemini.
The Playground makes this even more visible. I picked the Anthropic Messages API and a Gemini model, and the reply is labelled "served by gemini" and "translated from Anthropic Messages":
In code, that's the whole trick: keep your client, change the base URL, change the model string.
from anthropic import Anthropic
client = Anthropic(base_url="http://localhost:8081", api_key="unused")
reply = client.messages.create(
model="gemini/gemini-3.5-flash-lite",
max_tokens=1024,
messages=[{"role": "user", "content": "In one sentence, what does an LLM gateway do?"}],
)
Two things I ran into on the way, neither the gateway's fault:
- Google retired a model under me. My first pick, gemini-2.5-flash, is closed to new users. The gateway passed Google's own 404 message straight through, so it was obvious what to change.
- Thinking tokens eat small budgets. With full Gemini 3.5 Flash and max_tokens=150, the model spent 140 tokens thinking and returned a cut-off answer. The gateway reported it honestly in the usage (thinking_tokens: 140). Raising the budget and using Flash Lite fixed it, and replies dropped to around a second.
Fallback: breaking a provider on purpose
Here's the test I cared most about. I asked for the bare model name gemini-3.5-flash-lite, which matches the gemini-* route. The gateway tries down first, fails, retries, then falls back to Gemini. I ran it three times:
Every call came back 200 with the right answer. The client never saw a failure. The interesting part is call 3: after four failures, the gateway put down on cooldown and skipped it, so the request went straight to Gemini in 0.79s.
The first two calls took about 5 seconds each. Most of that is likely Windows retrying the connection to a dead local port before giving up, with tries = 2 doubling it. A provider that answers with a fast error would fail over much quicker. Either way, it's worth tuning tries and backoff for your setup.
The gateway's own log tells the same story. The third line shows the bare model name routed to gemini:
)
My favourite debugging feature is on the Routes page. Type a model name into "Try a model" and it shows where the request would go and why. Here it shows down at 100% errors and Gemini at 682 ms, without sending anything:
The detail view for one of the failed-over requests shows the whole 4.97s and that gemini served it:
Routes can do more than plain fallback. They can pick cheapest, fastest, weighted or random, hedge a slow call against a second provider (hedge = "200ms"), filter with needs = ["tools"], and match globs like the one I used. Config changes reload without a restart (--watch or SIGHUP).
The web UI
Open http://localhost:8081/ui/ and you get a proper dashboard, not an afterthought. After my clean run: 5 requests, 0% errors, tokens in and out, and per-provider stats.
The Requests page is a live log with model, endpoint, provider, status, latency and tokens for every call, plus filters and export:
Providers shows each upstream, what it serves and the formats it speaks:
Models lists all 57 models my Gemini key could reach, with context windows:
Notice the empty Price / M column. The gateway only knows a price when the provider's model list includes one or you declare it, and Gemini's doesn't. So out of the box I got tokens but no dollar figures for Gemini. You can fix that with a few lines per model:
[models."gemini/gemini-3.5-flash-lite"]
pricing = { input = 0.30, output = 2.50 }Β Β # $ per million tokens, Google's list price (Nov 2026)
Gateway-side MCP
I didn't test this one, but it's unusual enough to mention. You can put MCP servers or built-in tool vendors (web search, browsers, code sandboxes, memory) behind the gateway, and it runs the tools inside the chat request. A plain chat client that has never heard of MCP can still use them.
Performance
Performance is Respan's main pitch. These are their published numbers from the README:
Two caveats, to be fair. These are Respan's own benchmarks, and LiteLLM was run with a single worker, so scaling its workers narrows the gap. I didn't run a load test. What I can say is that the gateway added nothing I could notice: my round trips were dominated by Gemini itself.
What I didn't love
- It's young. v0.3.0 at the time of writing. Expect a smaller provider list, plugin ecosystem and community than LiteLLM's for a while.
- No prices for Gemini out of the box. The request log shows tokens but not dollars until you declare pricing.
- Failed attempts are invisible on the Providers page. down shows 0 requests there, while the Routes page correctly shows it at 100% errors. The Routes page is where to look.
- Custom plugins mean Rust. The plugin API is clean (one trait, four hooks), but a Python-only team is limited to config and the built-ins.
- The docs are thin in places. The useful routing details (pick, tries, hedge) live in docs/configuration.md and gateway.example.toml in the repo. Read those first.
Who should use it
Respan Gateway makes the most sense if you want a self-hosted gateway that's light on latency and memory, you like having routing and fallbacks in one readable TOML file, and you want to mix SDKs and providers without rewriting client code.
If you need the widest list of integrations today, or a mature enterprise feature set, LiteLLM is still the safer bet for now.
For me, the parts that matter worked exactly as described on a real provider. Anthropic-format requests came back from Gemini, a dead provider was routed around and then put on cooldown, and check plus the Routes page made it easy to see what the gateway was doing and why. That's more than I expect from a v0.3.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.













