The Best Self-Hosted LLM Gateway for Small Teams in 2026
A team of two to ten engineers does not have a platform group, and that is the constraint that should pick the gateway. Ops time matters more than the length of a feature list. Nobody's title says platform. You are alrea
A team of two to ten engineers does not have a platform group, and that is the constraint that should pick the gateway. Ops time matters more than the length of a feature list. Nobody's title says platform. You are already paying three model providers. Two people have the production OpenAI key in their shell history. Someone in finance just asked which feature burned last month's Anthropic bill.
A box that can take 5,000 requests per second is the wrong target. The right one installs in an afternoon, sits quietly after that, cuts off runaway spend before the invoice, and does not force a sales call when engineer eleven starts. Four projects meet that bar. The case for not self-hosting at all comes after the shortlist.
Three tests that matter at this size
Can you forget about it for a month?
Count processes before you count features. Each database is a backup job, an upgrade, and a page. PostgreSQL plus Redis plus a log store is three extra duties next to a gateway that writes an embedded file. Heavier is not wrong. Those extra pieces buy real features. Pick them on purpose, not by default.
Count upgrades the same way. A team this size does not read a changelog every week. Prefer tagged releases, a clean upgrade path, and a pinned version.
Does the limit stop the request, or only send an email?
This is the requirement small teams skip. A soft budget notifies you once the money is already spent. A hard budget refuses the call. An agent loop that runs away at 2 a.m. is stopped by only one of those. Ask every candidate whether spend can be capped per developer, per service, and per key, and whether the cap is applied before the provider is contacted.
What happens at engineer eleven?
Today the console is a shared admin login. In a year you will have an identity provider, an offboarding checklist, and maybe a SOC 2 auditor asking how access gets revoked. This is where open-core licensing usually lands. SSO, SCIM, and audit logs are the features most often held back for a paid tier. Find out now whether growth means a license negotiation.
Four projects that fit without a platform engineer
Ten-plus open-source gateways exist in this category. Four of them fit a team with no platform engineer. Agent Router (formerly Envoy AI Gateway), Higress, Kong, and APISIX are excellent, and they are absent here for a different reason: they assume Kubernetes, or an API-gateway practice someone already likes maintaining. Quality is not the objection.
| Bifrost | LiteLLM | Kaeso | agentgateway | |
|---|---|---|---|---|
| License | Apache-2.0 | MIT, except a commercial enterprise/ directory |
MIT, no enterprise edition | Apache-2.0 |
| Written in | Go | Python with a growing Rust core | Rust gateway, TypeScript console | Rust |
| What you run | One container; config in embedded SQLite by default, Postgres optional | Container + PostgreSQL (needed for virtual keys); Redis optional | Docker Compose: Caddy, gateway, console, PostgreSQL, ClickHouse, Valkey (~2 GB download) | One binary plus a YAML config |
| Install |
npx -y @maximhq/bifrost or docker run
|
Docker | `curl -fsSL https://kaeso.app/install.sh \ | sh`, or a template for Render, Railway, DigitalOcean, Fly.io, Coolify, Dokploy, CapRover and others |
| Web console | Yes | Yes | Yes, with a playground | Yes, but configuration is mostly file-based |
| Free budgets | Yes (virtual keys, budgets, rate limits) | Yes; budget reservation rejects over-budget requests before the provider call (on by default) | Yes, hard and soft, per project, member, service account and key | Budget and spend controls are listed in the README |
| Free SSO | No, enterprise | Up to 5 users | Yes (OIDC), plus SCIM and 2FA | Not a multi-user console product |
| Biggest catch for a small team | SSO, RBAC and audit logs cost money later | Two supply-chain compromises in 2026; Python under load; SSO beyond 5 users is paid | Weeks old, short public history, heavier stack | Built for infrastructure teams, not per-developer budgets |
Bifrost, if day one has to stay small
Bifrost wins when the constraint is how little you want to run on Monday morning. It is one Go process. Config lives in embedded SQLite. A web UI is included. Virtual keys, budgets, and rate limits are in the open-source build, and the docs also treat fallbacks and semantic caching as open source.
The bill arrives when identity shows up. SSO, RBAC, audit logs, guardrails, and clustering live in Bifrost Enterprise. OIDC login and SCIM provisioning are documented on that tier. The split is a fair business model. If an identity provider is already on the roadmap, put that license in the cost model now.
LiteLLM, if you need the coverage and will do the homework
Coverage is the reason LiteLLM stays on a small-team list. It advertises 100+ providers, and agent and LLM frameworks integrate it more widely than the other three. The open-source proxy will issue virtual keys and enforce budgets, with PostgreSQL underneath. Budget reservation ships on by default: if the estimated cost would break the budget, the request stops before the provider sees it. From v1.76.0, SSO is free for as many as 5 users.
A team without a dedicated operator has two pieces of homework. The first is supply chain. March 2026 put 1.82.7 and 1.82.8 on PyPI under an attacker's account. Those builds shipped a credential stealer that executed on every Python start. The Docker image was not part of that March incident. October 2026 repeated the compromise and did reach containers. For a few hours, PyPI's 1.107.0 and the Docker Hub and GHCR tags latest, main-latest, main-stable, and v1.107.0 pointed at malicious builds. Pin a version and an image digest. Never float on latest. Do not colocate the gateway with cloud admin credentials. The second piece is sizing. LiteLLM's own benchmark docs tell you to plan on about 100 combined RPS for a four-worker pod. Most startups fit under that. The process is still Python, and it will want real CPU and memory as traffic grows. Past 5 users, SSO, SCIM, and audit logs require an enterprise license.
Kaeso, if the license conversation is the thing you want to skip
Kaeso is an open-source AI gateway you host yourself, written in Rust. It exists for the cell the table leaves empty on the others: SSO, SCIM, roles, two-factor authentication, budgets, and the console are MIT-licensed, and there is no enterprise edition waiting at engineer eleven. Adding an identity provider means pointing OpenID Connect settings at it. The license does not change.
Spend controls attach to a project, a member, a service account, or a key. A hard budget refuses the request before any provider is called. LiteLLM's budget reservation also blocks before the call. The difference is the identity layer around those caps, which Kaeso does not charge for. Each request is logged with key, member, model, tokens, and cost, using the prices in force at the time. The log write happens after the reply, so it does not sit on the request path. Provider keys are encrypted with AES-256-GCM under a master key you hold. The keys you hand to apps are stored only as hashes. An OpenAI or Anthropic SDK you already use keeps working once the base URL and the key change.
The costs are age and weight. Work started in October 2026, and v0.7.0 shipped on November 21, 2026. The community is small. The public commit history is concentrated, which is a bus-factor risk for a component in the request path. There is no hosted version. PostgreSQL, ClickHouse, and Valkey are the default data stores, about a 2 GB download, and that is more to back up than Bifrost's SQLite file. Independent benchmarks have not been published. If any of those are a deal-breaker, take Bifrost or LiteLLM instead.
agentgateway, if the product really is an agent platform
agentgateway is a Rust proxy, and a Linux Foundation project, aimed at LLM traffic, MCP, and agent-to-agent calls. It deserves a look when the product itself is an agent platform and you need MCP federation and guardrails in the proxy. If the actual ask is one key per developer, a spend cap, and a dashboard, this is more infrastructure than that ask requires.
When not to self-host
A name on the shortlist is not a reason to rent a server. A managed router is less work. OpenRouter is the plain example: nothing to deploy. Its pricing page lists a 5.5% platform fee on the Standard plan. With your own provider keys (BYOK), the first $25,000 of list-price inference per month carries no fee, then 5%.
Self-hosting is the right call only when at least one of these is true:
- Prompts or keys cannot leave your company. Customer data, health data, a contract, or an EU customer asking where prompts are stored.
- The request log has to be yours. Who called what, on which key, at what cost, queried without an export from someone else's dashboard.
- Limits have to live in your infrastructure, per developer and per service, not only as a monthly cap on a credit balance.
If none of those are true, stop at the managed router. If one is true, use the stage picks.
What fits, by stage
Two to four engineers, one product, no compliance pressure: Bifrost. It has the fewest moving parts and free budgets. Revisit the choice when the team grows.
Five to ten engineers, an identity provider in use or coming soon: Kaeso, if a young project is acceptable, because SSO and SCIM do not start a license conversation. LiteLLM if you need its provider breadth and you are still under five console users, or you are willing to pay later.
Heavy framework integration (LangChain-style stacks, many niche providers): LiteLLM, pinned to a known-good version and an image digest.
You would rather not run anything: a managed router. Come back when a customer asks where prompts are stored.
A first-week checklist, whatever you pick
-
Pin the version. A specific image tag or release, never
latest. Subscribe to that project's security advisories. - Put provider keys in the gateway, then rotate the old ones. A laptop should no longer hold a raw provider key.
- One gateway key per developer, and one per service. A shared key makes cost attribution impossible, and it makes offboarding impossible.
- Hard-cap anything that runs unattended. Agents, cron jobs, and evaluation runs need a ceiling that refuses the call, not a ceiling that emails you after the spend.
- Give the most important model a fallback route. A provider outage should not become your outage.
- Back up the database before the first real week. The store that holds keys and budgets is production infrastructure now.
- Write the offboarding step. Even if the step is only "delete their gateway key," it has to be on the list.
Mistakes small teams make with gateways
- Budgets that only report. Yesterday's overspend on a dashboard is not a control.
- Picking from benchmark charts. At startup volume, provider latency swamps gateway overhead. Every gateway on this list is fast enough. Choose the one you can operate.
- Skipping the license file. An enterprise directory inside an otherwise open repo, or an AGPL core, changes what you can build on. Read the LICENSE before a workflow depends on a feature.
- Sharing a box with other secrets. The LiteLLM incidents are the demonstration. A compromised dependency collected whatever credentials were sitting on that machine.
FAQ
What is the best self-hosted LLM gateway for a small team?
The lightest setup is Bifrost. The widest provider coverage is LiteLLM. Kaeso is the one if SSO and SCIM have to be free from day one, and only if a new project is acceptable.
How much does it cost to self-host an LLM gateway?
The software on this shortlist costs nothing to run. You pay for a small server and for time. The cost that shows up later is an enterprise license for SSO, SCIM, or audit logs. On this shortlist, Kaeso is the only project with no enterprise edition at all.
Is LiteLLM still safe to use?
The bad releases were 1.82.7 and 1.82.8 in March, and 1.107.0 plus several Docker tags in October. Pin a known-good version by digest. Rotate credentials on any machine that executed an affected artifact.
Do you need Kubernetes to run an AI gateway?
No. Bifrost, LiteLLM, Kaeso, and agentgateway all run without a cluster. Kaeso also ships a Helm chart if you want one later. The option designed around Kubernetes is Agent Router, formerly Envoy AI Gateway.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.