The real economics of a production LLM pipeline: resumability, cost-aware routing, and measuring when local beats the API

The gif, and the honest headline

Here's a multi-stage LLM pipeline being killed mid-run and started again:

It picks up exactly where it stopped. No duplicate model calls, no re-spend. That's the
engineering payoff. The headline I want to be honest about up front is the economics
one, because it runs against the usual pitch:

This is not "run it locally and save a fortune." At a real content pipeline's scale,
batched cloud inference is genuinely cheap. The valuable thing is a reproducible way to
measure the local-vs-cloud crossover, and the answer of when local does and doesn't
pay off.

Here's the real number, from an instrumented production run: one weekly run of this
pipeline moves 641,671 input and 77,409 output tokens across nine local model stages, plus
134 locally-generated images. Run it on batched cloud models instead and that's about
$0.88, under a dollar a week. So the interesting question was never "how much does local
save." At this scale, cloud is already pocket change. It's when that stops being true,
and how you would know.

The pains everyone hits on week two

The first version of any LLM pipeline is a for loop over a list. It works in the demo.
Then production happens:

• It dies at item 800 of 1000 and you do not want to pay for the first 800 again.
• The same run, re-triggered, re-does work and double-writes downstream.
• Near-duplicate inputs get processed twice, because nothing deduplicates them.
• You have no idea what it cost, because nobody recorded the tokens.
• A rate limit or a credit ceiling hits and the run dies dirty, mid-item.

None of these is hard on its own. Together they're the difference between a script and a
pipeline. resumable-llm-pipeline is the
small, framework-light reference that wires them together, extracted (patterns only) from
a real weekly content pipeline at Cedar & Bloom. It runs offline on the stdlib alone via a
deterministic mock router, so you can see all of it in five minutes.

The build

Two pieces carry the weight.

ProgressStore, for resumability. A crash-safe {item_id -> {status, ...}} written
with a temp-file-and-rename atomic swap after every item. Each stage filters to unfinished
items and reads the previous stage's output. That decoupling is the resumability: kill
the process anywhere and re-running recomputes nothing that already finished. The test
suite proves it in numbers. Across a killed run and its resume, total model calls equal
the number of kept items exactly. No item is processed twice.

ModelRouter, for cost-aware routing. complete(prompt, tier) and embed(text)
behind one uniform response that carries token counts. Two tiers (cheap and quality)
map to a backend (mock, local ollama, or hosted) by one config line. Because
both local and hosted return (text, tokens_in, tokens_out), cost and metrics are
backend-agnostic, and swapping local for hosted to re-run the benchmark is a one-liner. A
Budget(max_tokens, max_cost) guard checks the paid stage as it runs; on breach it
checkpoints and exits cleanly. Resume is the retry.

Routing between a cheap and an expensive model is not a new idea.
RouteLLM
(paper) showed a learned router can cut cost by around
85% on some benchmarks while holding quality. This router is the boring, explicit version:
you assign each stage a tier by hand. The goal is not to be clever, it is to make the cost
of that choice measurable.

The measurement: the tokens were being thrown away

Here's the detail that started this. The production pipeline it generalises calls local
models via Ollama, and Ollama's response includes
prompt_eval_count and eval_count, the exact input
and output token counts, in the final chunk. The original code threw them away. It did
real token work every week and recorded none of it.

So the router captures those counts on every call (and usage.input_tokens /
usage.output_tokens on the hosted path), and a fail-open metrics module writes one
JSON line per call: tokens, latency, backend, model. Fail-open matters. The thing that
measures the pipeline must never be the thing that breaks it. rlp report rolls it up.

Here's the actual per-stage scorecard from one instrumented weekly run (qwen2.5:14b
is the cheap tier, qwen2.5:72b the quality tier):

stage          model            calls   tok_in   tok_out
novelty        qwen2.5:14b        273   417647    12860
rewrite        qwen2.5:72b         67   112516    48673
image_prompt   qwen2.5:14b         67    52193    11094
enrich         qwen2.5:14b         67    43759     3289
cluster        qwen2.5:14b          4    10264      700
timelines      qwen2.5:14b         10     2568       60
companies      qwen2.5:72b         20     1445       661
edition_intro  qwen2.5:14b          1     1279        72
image_gen      FLUX.1-schnell     134        0        0   (images — timing only)
--------------------------------------------------------------
TOTAL                                    641671    77409

Until this run, every one of those token counts was None. The pipeline did the work
weekly and recorded nothing. Notice novelty on its own: 273 calls, 417k input tokens,
65% of the entire run's input. Hold that thought, it matters for the cost.

The benchmark: measure the crossover, don't guess it

rlp benchmark reads the tokens the run actually recorded, maps each tier to its hosted
equivalent, projects to an annual workload, and compares hosted cost (list price, and with
the Batch API's
50% discount) against a fixed local cost. Fed the real weekly run, with the cheap tier
priced as claude-haiku-4-5 ($1/$5 per 1M) and the quality tier as claude-sonnet-5
($3/$15 per 1M):

Measured (weekly run): 641,671 input + 77,409 output tokens across 9 stages + 134 images.
Modeled:  hosted prices as above, Batch 50% off, hosted image ~$0.003/img, local $560/yr
          amortised (hardware capex + power), 52 weekly runs/yr.

| line                         | per run | per year (x52) |
|------------------------------|---------|----------------|
| hosted text (list)           | $1.75   | $91            |
| hosted text (batch, 50% off) | $0.88   | $46            |
| hosted images (modeled)      | $0.40   | $21            |
| local (fixed, amortised)     | —       | $560           |

Break-even is about 640 weekly runs per year, roughly 12x the real cadence. Below
that, batched cloud wins on cash.

Add the monthly evergreen run (measured: 47,596 in / 33,920 out, so $0.47 a run at list,
$0.24 batched, roughly $3 to $6 a year) and the whole Cedar & Bloom LLM workload lands
around $50 to $70 a year on batched cloud (about $110 at list). Against a $560/yr
amortised local box, cloud is 8 to 12 times cheaper on cash at the real volume.

The discipline that makes this trustworthy, and the whole reason the tool exists: token
counts are measured; hosted prices, the per-image cloud rate, and the local yearly cost
are modeled and labelled. "The hardware's a sunk cost" is not a fair comparison. An
honest break-even needs the amortised capex line, which is why it's in there.

The counter-intuitive bit: the cost isn't where the tokens are

Look again at that weekly scorecard through a cost lens:

novelty moves two-thirds of the tokens but stays cheap, because it runs on the cheap
model and emits almost nothing. rewrite is the reverse: modest token volume, but its
output tokens on the quality model at $15/1M dominate the bill. The money is in output
tokens on the quality tier, not where the token volume is. Knuth's old warning that
"premature optimization is the root of all evil" has a neat LLM corollary: don't guess
which stage is expensive. The tokens pile up in novelty; the money leaves through
rewrite. You only see that by measuring, and it's what tells you which stage to move to
cloud first if you ever wanted a hybrid.

One genuinely measured cloud data point

Everything above prices local tokens as if they'd run on the cloud. But one stage in the
wider Cedar & Bloom stack already is hosted: SteadyPath's quiz runs on Anthropic Haiku, and
instrumenting it captured 4 real calls, 1,490 input and 1,861 output tokens, actual
usage from the API, not a projection. Same metrics module, same JSONL, local and hosted
side by side. That's the point of the backend-agnostic router. The moment you move a stage
to the cloud, its real cost lands in the same ledger as the local estimate.

The honest conclusion the data supports: at Cedar & Bloom's real scale, batched cloud
would cost tens of dollars a year, and a weekly run is under a dollar. Local's edge is
control, privacy, and no rate limits, and it only wins on cash at roughly 10 times
the volume, or once you bolt on a token-hungry agentic layer. The number that tells you
which regime you're in is one you can now measure instead of argue about.

Take the pattern

git clone https://github.com/uxdw/resumable-llm-pipeline && cd resumable-llm-pipeline
python -m venv .venv && . .venv/bin/activate && pip install -e ".[dev]"
rlp reset && rlp run && rlp run       # watch the 2nd run resume with 0 model calls
rlp benchmark                          # your own local-vs-cloud crossover

Point the router at your models, feed the benchmark your real per-stage tokens and verified
prices, and read your own break-even. Measure your crossover. Don't inherit someone
else's headline.

(Companion piece: stop shipping AI agents you can't
measure, on evals and observability as a CI gate. This
pipeline's metrics core is the same telemetry idea, pointed at cost.)

Built and measured at Cedar & Bloom. MIT-licensed. Written by Richard Atkins.