AI Inference Benchmark: Keln vs Fireworks, Together AI, Baseten, Modal, DeepInfra & Z.ai

We ran the same agent sessions across seven inference platforms to see how they compare on speed, reliability, prompt caching, and cost.

The test covered DeepSeek V4.1 Flash 1017 and GLM-5.3 Flash, using real agent conversations with tool calling and maximum reasoning enabled.

Each platform received the same conversations, in the same order, with identical pauses between turns.

We included our own platform, Keln, alongside Fireworks, Together AI, Baseten, Modal, DeepInfra, and Z.ai.

Here's what we found.

DeepSeek V4.1 Flash 1017

GLM-5.3 Flash

Turns answered

Most sides answered everything we sent on both models. Baseten refused about half the DeepSeek turns with rate-limit errors: these prompts are tens of thousands of tokens each, and just 2 sessions at once put us over our account's limit. On GLM-5.3 Flash the breaks kept the load lower, so Baseten refused fewer turns there.

Modal cut 1 stream on DeepSeek, while on all the other providers answers ended normally. An app calling Modal directly gets the cut stream and has to resend the turn itself, because there is no second provider to send it to.

Cache hit rate

Running inference across multiple providers can hurt prompt caching when conversations move between them. On DeepSeek, Keln switched providers on just 1 of 1,440 turns, keeping its cache hit rate at 94.3%, close to the best direct provider at 95%. On GLM-5.3 Flash, Keln reached 90.5%, the highest of the 6 sides.

Fireworks had the most cold turns on DeepSeek: 165, compared with 11 on Keln. Each cold turn meant reading most of the prompt again instead of using cached tokens.

Based on each side's usage. A cold turn is any turn after the first with under 10% cached.

Cache after a break

We also tested how well providers retain cached prompts when a conversation goes idle. On GLM-5.3 Flash, we added breaks of 1–20 minutes to 15% of turns, using the same timing across all providers.

After a 15–20 minute break, Keln still had 63.0% of the prompt cached, compared with 38.4% on DeepInfra and 16.5% on Z.ai. Baseten, Modal and Fireworks had 0%.

Decode speed

Keln had the highest median decode speed on GLM-5.3 Flash at 432 tokens/s, 75% faster than the next fastest provider, Baseten, at 247 tokens/s.

On DeepSeek, Keln and Together were nearly tied at 432 and 437 tokens/s, both well ahead of the other providers.

First token is p50 / p95 and counts the first reasoning token.

Reasoning at max

We set reasoning effort to maximum for every request. All providers returned broadly similar amounts of reasoning tokens, including Keln.

In our previous benchmark, 2 gateways returned less than half as many reasoning tokens for the same requests. We didn't see that issue here.

Modal doesn't report reasoning tokens for GLM-5.3 Flash, so those results aren't included.

Cost

Fireworks was the most expensive on DeepSeek, largely due to its 165 cold turns. It processed 2.5 times as many uncached prompt tokens as Keln.

On GLM-5.3 Flash, Keln was among the more expensive options. Its pricing covers more than model inference: Keln manages capacity across providers, monitors performance and output quality, and handles failover and retries. You get all of that through one API, without managing provider accounts, rate limits or caching yourself.

Baseten's cost per 1,000 turns counts only the turns it answered.

What we learned

Going directly to an inference provider can be cheaper, and performance can be excellent when capacity is available. But you're limited to that provider's infrastructure and your account's rate limits.

We saw those limits in this benchmark. Baseten rejected requests due to rate limits, Modal cut off a response mid-stream, and Fireworks had to process previously cached prompts again.

Keln takes a different approach. It runs inference across multiple providers, managing capacity, performance, and reliability behind one API. In this benchmark, Keln completed every turn, matched or exceeded the fastest direct providers on decode speed, and retained more cached tokens after longer breaks.

Keln wasn't the cheapest option, but the price includes the infrastructure management, failover, and retries you'd otherwise have to handle yourself.

We post benchmarks with the method included. The previous one compared OpenRouter, Vercel, Ramp and Keln on DeepSeek V4.1 Flash for 24 hours: read it here.

Benchmark method

We used 8 sessions from the public armand0e Claude Code corpus on Hugging Face, replayed with native tool calls. Each recorded assistant turn was sent as a new request, with the original tool results appended, so conversations grew as they did in the original sessions.

Each run used up to 30 rounds per session, with 2 concurrent sessions per side and fresh salts and conversation IDs for each pass. The median turn contained 40,665 tokens on DeepSeek and 38,578 on GLM-5.3 Flash.

All requests used reasoning_effort: max in the format supported by each API, temperature 1, streaming enabled, and no max_tokens limit.

Pauses between turns followed the original sessions, capped at 60 seconds, with additional breaks to test caching. On DeepSeek, 5% of turns had breaks of 30 seconds to 2 minutes. On GLM-5.3 Flash, 15% had breaks of 1–20 minutes. We used the same random seed across all sides, so each received the same pauses on the same turns.

The DeepSeek benchmark ran on November 3, 2026, from 02:30 to 08:53 UTC, with 8,640 turns total (1,440 per side). The GLM-5.3 Flash run made 3 passes in 14 hours on November 4; a pass replays the 8 sessions once. This post reports the third pass, from 03:45 to 08:32 UTC, with 1,440 turns total (roughly 240 per side).

We tested Keln through its public API. For direct providers, we used Keln's own accounts and each provider's model identifiers, reasoning settings, and per-conversation cache hints. These included the documented user field or, for Baseten, the x-session-affinity header. Fireworks served DeepSeek through its standard endpoint.

Cache hit rate is the percentage of prompt tokens reported as cached in each provider's usage data. A provider switch is counted when consecutive turns in the same conversation run on different providers. Decode speed is calculated as output tokens divided by generation time for each turn.

We calculated costs using each provider's published model prices as of November 4, 2026, applied to its reported usage. Fireworks offers 2 DeepSeek pricing tiers; we used the standard tier corresponding to the endpoint tested. Modal shows its rates in its console, so we used the prices shown on our account.

For Keln, we used the November 4 price list: DeepSeek at $0.30 per million input tokens, $0.01 per million cached input tokens, and $1.20 per million output tokens; GLM-5.3 Flash at $0.15, $0.05, and $0.50 respectively. The DeepSeek run was billed at the previous cached-input rate of $0.0108 per million tokens, resulting in $2.09 per 1,000 turns.

Results reflect this workload, our account limits, and the conditions during the test. Performance may vary with other workloads or concurrency levels.

The raw benchmark rows and response bodies are not publicly available.