5 Best Open-source LLM Gateways 💎

TL;DR

• An open-source LLM gateway gives applications one place to connect to model providers. Depending on the project, it can also handle routing, retries, budgets, access controls, caching and request logs.
• The five projects here take different approaches: Bifrost, Kong, Agent Router, Apache APISIX and LiteLLM.
• Bifrost is our pick for teams that want a self-hosted gateway with model, MCP and agent traffic controls. Its commercial Edge product can also route AI traffic from employee devices.
• Open source does not mean every feature is free or self-hostable. Check each project's license and feature boundaries before adopting it.

A few years ago, many teams adopted gateways to centralize provider keys and billing. Those needs have not gone away, but teams now also have to think about model choice, reliability, usage limits and the growing number of tools connected to agents.

The projects in this list are not interchangeable. Some focus on model routing, while others build AI features on top of a broader API or service-mesh stack. I compared five open-source LLM gateways, their deployment models and the features available without a paid plan.

What an open-source LLM gateway does

At a basic level, an open-source LLM gateway is a reverse proxy for model requests. An application sends traffic to a shared endpoint, and the gateway handles provider-specific formats, routing and retries across services such as OpenAI, Anthropic, AWS Bedrock and Google Vertex AI.

Some projects add budgets, access controls, audit trails, caching or MCP support. Others focus on getting requests to the right model with low operational overhead. The right choice depends on which of those jobs you need the gateway to do.

How to choose an open-source LLM gateway

• Provider coverage. Confirm the providers and API formats your application needs today, then check how new providers are added.
• Reliability and latency. Look for retries, fallbacks and load balancing. Treat benchmark numbers as a starting point.
• Deployment and license. Check the license for the gateway itself, and distinguish open-source components from paid hosting, support or enterprise features.
• Operations. Consider how the project handles secrets, budgets, access controls, logs and upgrades. A gateway still needs someone to run it.
• MCP and agent support. If your applications call tools, check whether the gateway routes or governs MCP traffic, and how identity and permissions work.

The five gateways at a glance

Here is the short version. Bifrost and LiteLLM are model-focused gateways, Kong brings AI traffic into a broader API platform, Agent Router is built around Envoy Gateway and Kubernetes, and Apache APISIX combines AI plugins with a general-purpose API gateway. These five open-source LLM gateways differ in deployment and feature boundaries.

The sections below look at what each project does well, where it fits and what to check before adopting it.

1. Bifrost: A gateway for models, MCP and agents

Bifrost is an open-source LLM gateway for enterprise AI, licensed under Apache 2.0 and written in Go. It provides one OpenAI-compatible interface to more than 10,000 models across 25+ providers. With its drop-in setup, an existing app can usually switch over by changing its base URL. Bifrost ships as a single binary and can run on a VM, in Kubernetes, in a private cloud or on a disconnected network.

Core capabilities

Performance is one of Bifrost's main selling points. Its published benchmarks report about 11 microseconds of overhead at 5,000 requests per second on a single t3.xlarge, with a 100% success rate. In a separate AWS EC2 comparison with 500 concurrent users over 60 seconds, Bifrost handled 424 requests per second, while LiteLLM handled 44.84. The reported P99 latencies were 1.68 seconds and 90.72 seconds, respectively.

For governance, virtual keys let teams set budgets and rate limits by key, team, customer or model. The Enterprise tier connects to identity providers such as Okta, Microsoft Entra, Keycloak, Zitadel, Auth0 and Google Workspace, as well as other OIDC providers. It also adds SCIM 2.0 directory sync, detailed role-based access control, data access controls and reusable access profiles. Routing supports weighted load balancing, retries with backoff and automatic fallbacks. Semantic caching, backed by Redis, Qdrant, Weaviate or Pinecone, can return stored answers for similar prompts.

// Detect dark theme var iframe = document.getElementById('tweet-2024540241844224013-383'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2024540241844224013&theme=dark" }

The MCP gateway controls which MCP servers and tools agents can use. Access can be filtered at the client, request and virtual-key levels, with per-user OAuth so each person connects using their own account. Code Mode takes a different approach to large tool sets: instead of loading every tool definition into the model context, the model writes a short script to coordinate the tools it needs. Bifrost reports that this can reduce input token use by up to 92.8%. In a 508-tool test, the reported cost fell from $377 to $29, with all tasks passing. The MCP gateway write-up has the full results.

Bifrost Edge extends those controls to AI tools running on employee devices. Available for macOS, Windows and Linux, it routes traffic from desktop chat apps, browser-based AI, terminal coding agents and their MCP servers through the Bifrost gateway after a one-time SSO sign-in. Edge routes AI traffic only and does not inspect other device activity. Gateway virtual keys, budgets and guardrails can then apply to those tools, including content safety, PII and secret detection on requests and responses, as well as integrations with AWS Bedrock Guardrails, Azure Content Safety, Google Model Armor, CrowdStrike AIDR and Check Point Guard. Administrators can allow or block apps and MCP servers centrally. Rules are enforced on the device. Rollout can use existing device management tools such as Jamf, Intune, Kandji, Omnissa Workspace ONE or JumpCloud.

For regulated environments, the Enterprise tier adds clustering, adaptive load balancing with circuit breakers, signed audit logs, and integrations with AWS Secrets Manager, GCP Secret Manager and HashiCorp Vault. It also supports in-VPC, on-premises and air-gapped deployments, where prompts and responses can stay within your infrastructure. Bifrost lists SOC 2 Type II, ISO 27001, HIPAA and GDPR among its compliance credentials, and says Fortune 500 companies use the product in production across several industries.

Where it fits

There are a few things to check before committing. Identity-based governance, clustering, adaptive load balancing, guardrail and vault integrations, and in-VPC deployment are part of the paid Enterprise tier, not the open-source core. Edge also requires an Enterprise gateway and is priced per endpoint. Since it is still in early access, a limited pilot makes sense before a wider rollout.

Best for: Teams that need to manage model, MCP and agent traffic from one gateway, especially when they have strict deployment or governance requirements. Bifrost supports self-hosted, VPC-isolated, on-premises and air-gapped environments.

💎 Check out the Bifrost repository ☆

2. Kong AI Gateway: AI features in an established API platform

Kong AI Gateway adds AI-specific features to Kong's established API gateway. If your APIs already run on Kong, you can use the same control plane, plugin model and operational setup for model traffic. The AI features include provider routing, token-based rate limits, prompt guard plugins, semantic caching and semantic routing across models.

Core capabilities

Kong added MCP support in Gateway 3.12 in October 2025. Its AI MCP Proxy plugin can expose REST APIs as MCP tools or sit in front of an MCP server. Access controls apply both when clients discover tools and when they invoke them. A companion OAuth2 plugin handles authentication for MCP traffic.

Where it fits

The main trade-offs are licensing and operational scope. Kong's documentation lists the MCP plugins as part of its AI Gateway Enterprise offering, and labels the OAuth2 plugin as a tech preview. If Kong is not already in your stack, deploying the full API gateway platform just for model traffic may be more than you need.

Best for: Organizations that already use Kong and want to manage AI traffic through their existing control plane.

3. Agent Router: Envoy-based routing for Kubernetes

Agent Router, formerly Envoy AI Gateway, is an Apache 2.0 project for managing access to hosted models, self-hosted inference and MCP servers. It provides an OpenAI-compatible API and is built around Envoy Gateway. The repository has moved to theagentrouter, but project documentation says the API resources, API group, CLI name and image paths remain unchanged.

Core capabilities

The project describes a two-tier pattern: one gateway handles shared authentication, routing and global limits, while another can manage traffic to self-hosted inference. It also offers a standalone CLI for local use, which makes it possible to try the API before deploying the Kubernetes components.

Where it fits

The trade-off is its platform focus. Teams already operating Envoy Gateway and Kubernetes have a clear path to evaluate it. For a small application that just needs a lightweight proxy, a simpler gateway may involve less setup. Its documentation and release notes are the best places to check the current status of specific providers and features.

Best for: Platform teams using Envoy Gateway or Kubernetes that want an open-source control plane for model and tool traffic.

4. Apache APISIX: Open-source AI plugins in an API gateway

Core capabilities

Apache APISIX is an Apache Software Foundation project licensed under Apache 2.0. Its open-source API gateway includes AI plugins for connecting to model providers, routing requests across instances, balancing traffic, retrying failures and limiting usage by tokens. The ai-proxy plugin supports documented providers and OpenAI-compatible APIs, while ai-proxy-multi adds multi-instance load balancing and fallback behavior. The ai-rate-limiting plugin applies token quotas.

APISIX also has an openapi-to-mcp plugin that exposes operations from an OpenAPI document as MCP tools. This is a useful contrast with Kong: APISIX's AI plugins are part of its open-source project, while Kong documents its MCP Proxy plugins as Enterprise features.

Where it fits

Because APISIX is a full API gateway, teams adopting it for AI traffic also take on its broader gateway configuration and operations. Its plugin model suits teams that want to manage API and model traffic together, with the AI capabilities available in the open-source project.

Best for: Platform teams that want an open-source API and AI gateway with multi-provider routing, token limits and MCP support.

5. LiteLLM: Broad provider support and a Python-first workflow

Core capabilities

LiteLLM is a common starting point for teams building with language models. It combines an MIT-licensed Python SDK with a self-hosted proxy that supports more than 100 providers, virtual keys, budgets, routing, fallbacks, guardrails and an MCP gateway. Semantic caching supports Qdrant, Valkey and Redis. Its documentation recommends using that feature for single-shot prompts rather than multi-turn agent traffic. A paid enterprise license adds SCIM, audit logs and some guardrail integrations. Admin UI SSO is free for up to five users.

Trade-offs

Two considerations are worth a closer look. In Bifrost's published benchmarks, LiteLLM handled 44.84 requests per second with a P99 latency of 90.72 seconds in an AWS EC2 test with 500 concurrent users over 60 seconds. Bifrost handled 424 requests per second with a P99 latency of 1.68 seconds in the same comparison. These are vendor-published results; performance varies by workload.

There was also a supply-chain incident. Malicious PyPI versions 1.82.7 and 1.82.8 were published on March 24, 2026; LiteLLM says a maintainer's PyPI account may have been compromised, according to its security update. The affected versions were removed. The incident involved a compromised package release rather than a vulnerability in LiteLLM's gateway code, and highlights the value of pinning dependencies and mirroring packages internally.

Best for: Python-focused teams with moderate traffic that value a broad provider ecosystem and can operate and secure a self-hosted service.

Deployment and licensing compared

This table is a starting point, not a substitute for checking the current license and feature matrix. Several projects keep parts of their enterprise functionality outside the open-source gateway.

Which AI Gateway Is Right for Your Team

If you are choosing a gateway for production, start with the constraints that matter most to your team. Bifrost provides multi-provider routing, agent workflows and private deployment options. Its published benchmark reports 11 µs of overhead.

LiteLLM is a natural option for Python-centric teams. Kong fits organizations already running Kong, while Agent Router suits teams built around Envoy Gateway and Kubernetes. Apache APISIX adds AI capabilities to an open-source API gateway. The best choice is the one that fits your existing platform and the amount of gateway operations your team is ready to own.

Frequently asked questions

What is an open-source LLM gateway?

An open-source LLM gateway sits between an application and one or more model providers. Depending on the project, it can provide a shared API, route requests, retry failures, apply usage limits and log traffic. Check the license and feature list because commercial hosting or enterprise controls may be separate.

How is an LLM gateway different from an API gateway?

A traditional API gateway focuses on routing and authentication for general API traffic. An LLM gateway adds features specific to model requests, such as provider routing, token-based limits, model fallbacks and usage reporting. Some API gateways now offer AI plugins as well, so the distinction depends on the product.

Should we self-host or use a managed gateway?

Self-hosting gives your team more control over prompts, keys and logs, but also means operating the service. A managed gateway can be quicker to adopt, though traffic passes through the vendor's network. The right choice depends on your security requirements and the operational support available to your team.

Do LLM gateways support MCP?

MCP support varies. Bifrost has a native MCP gateway, Kong offers AI MCP Proxy plugins, Agent Router handles AI and MCP traffic, Apache APISIX provides an openapi-to-mcp plugin, and LiteLLM includes MCP support in its proxy. Compare authentication and tool-level permissions rather than treating support as a yes-or-no feature.

How much latency does an LLM gateway add?

The answer depends on the gateway, workload and enabled features. Bifrost reports about 11 microseconds of overhead at 5,000 requests per second on a single t3.xlarge in its published benchmarks. That is small next to the time many models take to respond, but added latency can matter when an agent makes a long sequence of calls.

Can we move off LiteLLM without rewriting code?

Often, yes. Both products expose OpenAI-compatible APIs, and Bifrost provides drop-in adapters for the LiteLLM SDK. Teams using the LiteLLM proxy can also migrate models, keys, teams and users. See the LiteLLM migration guide for the process and its details.

Can an LLM gateway govern employees using ChatGPT or Claude desktop apps?

Only when that traffic passes through the gateway. A standard gateway typically sees requests that developers have configured to use it. Bifrost Edge is designed to route traffic from desktop apps, browsers and coding agents through Bifrost so gateway policies can apply.

Is Bifrost open source, and what does the Enterprise tier add?

Bifrost's core is open source under Apache 2.0 and can be self-hosted for free. The Enterprise tier adds high availability, identity-based governance, guardrail and vault integrations, signed audit logs, and in-VPC or air-gapped deployment. It includes a 14-day free trial. Bifrost Edge requires an Enterprise gateway and is priced per endpoint.

Conclusion

There is no single gateway that is right for every stack. Choose based on the providers you use, how you plan to deploy, and whether you want a dedicated model proxy or AI features inside an existing API platform. Check which features are included in each open-source project, and test each of the five gateways with your own workload before committing. If Bifrost is on your shortlist, its documentation and demo cover its deployment and Enterprise options.

Resources:

• Bifrost GitHub: https://github.com/maximhq/bifrost
• Bifrost Docs: https://docs.getbifrost.ai
• Bifrost CLI: npx -y @maximhq/bifrost-cli

Thanks for reading this article! ❤️

I'd love to hear your thoughts on this in the comments!

💎 Check out the Bifrost repository ☆