I ran 33 LLM models behind one endpoint for a month. Here is what broke.

I built a small gateway that pools free LLM tiers — NVIDIA NIM, Kilo, OpenRouter, a couple of
community endpoints — behind a single OpenAI-compatible URL. Thirty-three models, one API key,
no monthly bill. It ran real daily work for a month.

It also broke in six different ways, and every single break was something I would have avoided
if someone had written it down first. So I did.

The numbers first

That last row is the one that cost me the most sleep.

Break 1: the 180-second hang that was not my bug

Every request through one particular relay hung for exactly 178 to 194 seconds and then failed
with a connection error. My code was fine. The relay had a hard 180-second timeout of its own,
and it sat on the socket without sending a single byte until the deadline expired.

The fix is not a longer timeout. It is an inter-chunk idle deadline: if no SSE chunk arrives
within N seconds, abort and retry on a fresh connection. A total-duration timeout will not save
you, because a stalled connection still counts as in-flight.

Break 2: one 429 froze every model on the provider

Free tiers rate-limit at the channel level, not the model level. A single greedy request tripped
a 429 and the cooldown I wrote disabled every model behind that provider for two minutes —
including ones I was not even calling.

Two fixes: pace client-side at ~85% of the measured entitlement instead of bursting, and scope
any cooldown to the exact model that was limited.

Break 3: the retry stampede

Fifteen parallel workers hit the same failure and retried on the same second. One upstream
hiccup became an outage. Exponential backoff without full jitter is the bug — the workers
stay synchronized.

Break 4: a byte-order mark broke my config

A scheduled job started throwing Unexpected token ''. The JSON config was written by
PowerShell's Set-Content -Encoding UTF8, which emits a BOM. JSON.parse does not tolerate it.
The fix is one line, but finding it was not.

Break 5: the stream that connected and said nothing

Some proxies accept the connection and send zero bytes. No error, no data — the request just
sits there. This looks identical to a slow model until you instrument the difference between
connect-fail, idle-fail, and slow-success.

Break 6: I lost a batch at item 700

No checkpoints. A crash at item 700 of 1000 meant reprocessing everything, and the bill doubled.
An append-only checkpoint journal with content-hash dedupe keys makes any job resumable in
seconds.

What I actually built

The gateway is about 600 lines: a candidate selector that respects per-model cooldowns, a pacer
that keeps under the sliding window, a failover ladder (primary → fallback → local Ollama), and
one JSONL line per request so that a 3am question has an answer in the morning.

Three of the recipes — the pacer, the 15-second messaging window, and the postmortem template —
are MIT licensed in this repository: llm-ops-recipes.

The other 27 (JSON repair loops, provider failover ladders, model tiering, golden-set regression
gates, cost accounting, and two full postmortems) are in the
LLM Ops Cookbook.

The one thing I would tell myself a month ago

Measure the limit before you design around it. I spent two days building a cooldown system
before running a 45-request burst probe that told me, in 13 seconds, exactly what the real
constraint was.

If you'd rather not rebuild this from scratch, I packaged the exact scripts I use: LLM Ops Cookbook — 30 one-page recipes for running LLM workloads that don't fall over. There is also a free sample if you want to see the format first.