A Correct Answer Has a Price. A Wrong One Has Three.

Two weeks ago I published a benchmark with a simple question: what does a correct answer from an AI agent actually cost? The headline result was that tiered routing — cheap model first, escalate on low confidence — produced correct answers for $0.0069 each, against $0.0097 for sending everything to the large model. Same accuracy, 29% less money.

I thought the interesting part was the routing. Three readers showed me the interesting part was the failures.

Price #1: the failure itself (Dean Lee's point)

My table treated a wrong answer as costing nothing beyond the tokens spent failing. Dean Lee, who ran his own entry in the same challenge, pushed on exactly that: in production, a wrong answer gets reviewed, reworked, or rolled back. It is never free.

Put a price on it and the table flips. Add a modest $0.01 per wrong answer — one human glance, a few seconds of review:

The "cheapest" setup becomes the most expensive one, by a factor of fifty, because two-thirds of its answers were wrong. The tiered setup doesn't move at all — it made no errors to price. Cost per correct without a failure price isn't a metric. It's a subsidy for being wrong.

Price #2: the escalation (Noah Salas's point)

Noah Salas, who builds agents for a living, added the cost I had averaged away: escalation itself. In my routed run, the system made 33 model calls to answer 24 tasks — the 9 extra calls are cheap attempts that failed before the large model took over. You pay for the failure and the retry.

That means the routed number isn't a constant. It depends on your escalation rate, which depends on your traffic mix, which drifts. A benchmark measures it once; production has to measure it continuously, or the 29% saving quietly evaporates as your workload changes.

Price #3: the failure you shouldn't price at all (Will Lassalle's point)

Will Lassalle, a CIO, split the category I was treating as one thing. A failure caught in review costs review time. A failure that escapes costs rollback, remediation, and customer trust. And some failures — his example class was anything touching money movement or safety — shouldn't be in the cost table at all, because they shouldn't be available to an autonomous system at any average price.

His ordering is the one I'm adopting: risk gate first, cost comparison second. Decide which failures are forbidden, block the system from making them autonomously, and only then optimize cost per correct among what's left.

What version 2 measures

So the benchmark I'm now building scores four numbers, not one:

1. Cost per correct answer — the original metric
2. Cost per correct, by failure type — caught-in-review failures and escaped failures priced separately
3. Escalation rate and its cost — measured as a distribution, not a point estimate
4. Gate violations — attempts to take a forbidden action, which fail the run outright regardless of cost

The accuracy floor stays: below 95% correct, a setup isn't cheap, it's disqualified.

The original benchmark is here, if you want the starting point: https://www.kaggle.com/benchmarks/karmendra8386/cost-per-correct — and the write-up that started this argument: https://dev.to/karmendra_pandey_43ac6983/your-ai-agent-has-a-burn-rate-so-i-benchmarked-the-cost-of-a-correct-answer-1bc1

The lesson I'd take to any team running agents: your cost dashboard is lying to you in a specific, fixable way. It prices the tokens. It doesn't price the wrong answers, the retries, or the failures that should never have been allowed. Price all three, and the cheapest setup on the page is usually not the cheapest setup you own.

How does your team price a wrong answer today — review time, a flat penalty, or not at all?