Two More Languages Should Have Been Cheap. My Estimate Was Off by a Lot

RealFeedApp writes a short summary and a plain-words explainer for every science and tech story it picks up. A language model does the writing, on our own server, not through someone else's API. Until this week it wrote in two languages.

On Monday I added German and Portuguese. I had done the arithmetic first. I had a measurement of how busy the model was, I had a per-language cost, and the sum said: a noticeable increase, comfortably within capacity.

By the evening the monitoring sent a message I hadn't seen in weeks: the worker isn't keeping up, a story has been waiting too long for its summary. A measurement over the hours since the release said the model was now busy almost all the time — several times the load I had estimated.

Nothing was broken. The estimate was. Here is how it was wrong, and what I changed.

The estimate counted the wrong thing

My mental model of the cost was: one call writes all the languages, so a new language costs whatever its share of the answer costs. Two more languages, a somewhat longer answer. Proportional, predictable.

That model was true before the change and false after it. Writing four languages in one answer had failed earlier quality tests — the answer grew, languages leaked into each other — so the new languages went into a separate call: the base pair in one request, the new pair in another, with the finished English headline passed along as a reference.

A separate call doesn't cost "the extra output". It costs reading the whole article again, plus the instructions, plus the output. I had priced the new languages as a few more lines of text. In reality each story got one more full trip through the model.

And explainers got the same treatment, in the same shape: a second call per story.

The lesson isn't "separate calls are expensive". They were the right call for quality. The lesson is that the cost model has to change when the call structure changes, and mine silently didn't.

Counting calls told me the wrong story too

The obvious first move was to look at what the model was doing: count calls by type. Explainers had doubled, summaries had grown — and explainers are the longest texts the system writes. Dropping the new languages from explainers looked like the cheap fix.

Then I measured seconds instead of calls. The worker logs each step before it starts, and each model call logs its duration, so a few lines of awk can attribute every second of model time to the step it belonged to:

journalctl -u my-worker --since '-6h' -o cat | awk '
  /worker: enriching/  {t="enrich"}
  /worker: explaining/ {t="explain"}
  match($0, /model .*: [0-9]+ s/) {
    split(substr($0, RSTART, RLENGTH), a, " "); s[t] += a[3]; n[t]++
  }
  END { for (k in s) printf "%s: %d calls, %.0f s\n", k, n[k], s[k] }'

Explainers turned out to be a small slice of the model's time. Almost all of it went to the summary step. The long texts were rare; the step that ran for every single story dominated. Cutting explainers would have saved almost nothing and made the product worse.

Count the resource you're short of. I was short of model-seconds, not of calls.

Why a small queue raised the alarm

The other surprise was the shape of the problem. The queue wasn't long. A handful of stories were waiting. But the oldest one had been waiting since roughly the release.

The worker always takes the highest-ranked stories first. That's what you want when there's spare capacity: the important news gets summarized first. When there's no spare capacity, it means the stories at the bottom never get a turn — fresher, higher-ranked ones keep arriving and jumping ahead.

That's why the alert watches the age of the oldest waiting item, not the length of the queue. A queue that stays short while its oldest member gets older is exactly what running at full capacity with a priority order looks like. Queue length would have stayed green all evening.

Two languages or one?

The options were: drop both new languages from generation, add hardware, or find something in between.

The in-between: keep German, drop Portuguese. But the pair was one call. Removing half of it wouldn't remove the call — it would only shorten the answer, while the expensive part, reading the article, stays.

So the question was empirical: how much cheaper is a call with one language than a call with two? And does one language on its own keep its quality? An earlier test had shown that a single-language call drifted into the language of the instructions — but those instructions were in Russian. The new calls used English instructions.

I ran the same comparison the earlier decision was based on: the same set of articles, the same prompts and sampling, the pair against German alone. German alone came out roughly a third cheaper per call, with no detectable quality loss: no wrong script, no drift into English, nothing cut off. (Checking for drift into English needed its own check — German and English use the same alphabet, so the usual script test can't see it. Counting common function words did.)

Portuguese came out of generation. German stayed.

Don't break the experiment you're running

There was a tempting further cut: stop generating German too, translate it lazily only for people who pick it in the menu, and add it back when there's demand.

The catch: the demand for a new language on a news site mostly comes from search, and search needs pages in that language to exist. Day pages and article pages are only built for languages that have finished translations. Remove generation, and the German pages disappear, and nobody arrives looking for German, and the demand test reads zero. The cut would have guaranteed its own justification.

So the menu now offers exactly the languages that are generated. A language in the menu either works fully or isn't there.

What I'd do before the next language

• Price the call structure, not the output. If the change adds a call, the cost is a full call — input included.
• Measure in the unit you're short of. Seconds of model time per step, not call counts.
• Watch the oldest item, not the queue length. With priorities, saturation looks like a short queue that ages.
• Test the cheaper variant instead of assuming it's half price. One language out of two wasn't half the cost; it wasn't the full cost either.
• Check that a cost cut doesn't sabotage the metric you'll judge it by.

And the obvious one, the one the estimate was supposed to replace: run the change for an hour and measure before calling it done. The arithmetic was fine. It was arithmetic about a system that no longer existed.