Cost per completed task is the only number that survives a workload
Every leaderboard ranks models by price per million tokens. That number is easy to measure and almost useless for a decision, because the thing you actually buy is a completed task, and the gap between the two is set by
Every leaderboard ranks models by price per million tokens. That number is easy to measure and almost useless for a decision, because the thing you actually buy is a completed task, and the gap between the two is set by your workload rather than by the price sheet.
Three places the gap shows up, each with real numbers from the last month of arguments about it.
1. Trials are not attempts
An eval scorecard that reports pass rate, cost, tokens and duration is already better than most. It is still missing the column that decides the purchase.
Suppose a target scores 3/3 at $0.0003 per run. Another scores 3/3 at $0.0003 per run. Same pass rate, same reported cost — and one of them needed two internal retries on every run to get there. The first is three times cheaper per completed task than the second, and the scorecard as printed cannot tell you which one you are looking at.
Pass rate over trials measures reliability. Attempts per trial is what reliability costs. They are different columns and only one of them is usually printed. Recording attempts is nearly free, because the loop already knows when it retried.
2. Per-generation is not per-accepted
The same compression happens with generative output. A vendor announces 100 ten-second videos for $5 — five cents a clip — and it is true.
It is also not the price you pay. If one seed in six produces something usable for a specific brief, the shot costs thirty cents. A loose prompt gets more usable variants per seed than a precise one, so the same model is cheap for mood boards and expensive for a shot you can describe exactly. The headline price is a property of the model; the accepted-clip price is a property of the brief.
At five cents a clip the bottleneck also stops being money and becomes review time. A hundred clips still costs an afternoon to watch. The metric that decides which tool wins is accepted clips per hour, not clips per dollar.
3. The metric that retires with the failure
The most expensive version of this is a metric that disappears when you fix the thing it measured.
Replace prose-and-parse with typed outputs and format failures drop to zero. The parse-error alert goes quiet, the dashboard turns green, and the remaining failure mode — a well-formed, schema-valid, wrong answer — has no metric at all. Nothing throws. Nothing queues. The ticket goes to the wrong team looking exactly like a correct routing.
The failure did not drop. It lost its instrument.
This is not an argument against typed outputs; it is an argument for naming the replacement metric before shipping the removal. When the loud failure goes away, the thing you were measuring still happens — you just stopped being able to count it. Retiring the parser without adding a confidence threshold or a spot-check is how a measurable problem becomes an invisible one, and schema validation makes it feel like the problem was solved.
Any rate you quote needs its bound
The reason these gaps survive scrutiny is that most reported numbers are point estimates with no denominator attached.
A clean example: zero regressions in 16 runs. The tempting summary is "0% regression rate". The honest one uses the rule of three — with zero events in n trials, the 95% upper bound on the rate is about 3/n, so 16 clean runs rules out rates above roughly 19%. That is still useful. It just cannot be multiplied by a cost per regression to produce a figure that looks precise.
The sample size then follows from the claim you want to make: about 30 clean runs to rule out 10%, about 300 to rule out 1%, and thousands to say anything about one in a thousand. At that point you have to sample deliberately, because production will not hand you the evidence fast enough.
The same logic applies to a confidence interval that straddles a floor — 74.3% measured against a 75% threshold is a breach by point estimate and an inconclusive result by interval. A gate that reports "inconclusive, would need N runs to resolve" is doing more for its users than one that reports fail, because the second trains people to ignore it within a month.
What to do instead
Two habits cover most of it.
Report the decision number next to the price number. Cost per completed task, per accepted clip, per passing run — with the workload that produced it stated. If the workload is not stated, the number is marketing rather than evidence.
When you remove a failure mode, name the metric that replaces it first. The removal is the easy part. Keeping the instrument is the part that determines whether anyone notices the next regression.
Numbers above come from published vendor pricing, a public eval scorecard, and two months of arguing about this with people who run these systems for a living. All of them are per-attempt or per-run figures unless stated otherwise — which is the whole point.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.