Cheapest LLM for 77-intent banking classification: GPT-6.1 Terra at $0.000727 a success

On 150 BANKING77 messages, GPT-6.1 Terra is the cheapest of ten models that clears a 70% pass bar and can be shown to match the baseline, GPT-6.1 Sol. It passes 84.0% of the tasks against 86.0% for Sol, at $0.000727 per successful task against $0.001272: 42.8% less.

Two cheaper models deserve a straight reading, because the cheapest line in a table is not always the pick.

GPT-6 Luna passes 82.0% (interval 75.1-87.3%) at $0.00009 per successful task, about 14 times cheaper than Sol. It clears the 70% bar. But on 150 rows the paired test cannot show it is within 5 points of Sol: it failed 8 tasks Sol passed and passed 2 that Sol failed. "Clears the bar" and "matches the baseline" are different claims, and the verdict names the second one.

Claude Haiku 5.5 passes 80.0% at $0.000138 per successful task with the lowest latency in the run (median 249 ms). It also clears the bar and is not shown to match Sol.

The run. 150 messages drawn with a fixed seed from the 3,080-row BANKING77 test split (CC BY 4.0). One task: reply with the single intent label from the 77-label list. Temperature 0, up to 2 retries when the reply is not a valid label, every attempt's cost counted, prices read from OpenRouter on 2026-11-05. Total spend for all ten models: $0.58.

Pass bar. 70% correct. A model "clears" only if the lower end of its 95% Wilson interval is at or above the bar, "borderline" if the interval reaches below it, "below" if the pass rate is under it. "Matches" means an exact one-sided paired test of "no more than 5 points worse than the baseline" succeeds.

Why a token price misleads. deepseek-v4.1-flash cleared the bar, but it needed 32 retries across 150 tasks (1.21 attempts per task), 11 of its final answers were still not a valid label, and 31.3% of everything it spent went on retries. Its cost per success ($0.000157) ends up above Haiku's ($0.000138). qwen3.8-flash spent 19.8% of its money on retries and had a 95th-percentile latency of 6.7 seconds. mistral-large-4 is the extreme case: 106 retries across 150 tasks, 44.6% of its spend on retries, and $0.002274 per successful task, the most expensive line in the table at 63.3%. A bill that counts only first-attempt tokens hides all three.

What a miss looks like. For "How long does a card delivery take?", Terra answered card_delivery_estimate; the label in the data is card_arrival. For "When will the card arrive?" it answered card_arrival where the data says card_delivery_estimate. Both are plausible readings of the sentence. On a 77-label task some misses are label-boundary calls like these, which is why the answer to "which model" depends on your labels and your bar, not on a leaderboard.

Check it or re-run it. The 150 rows, the prompt, the prices and the re-run script are on the verdict page, so you can run the same comparison with your own OpenRouter key (the full ten-model run cost $0.58) and see where your numbers land. Prices move monthly; the page notes the day it last re-checked them.

Want the same verdict on your own data? Import 10-100 of your input/expected pairs, add your current model, and the page names the cheapest model that clears your bar, with cost per successful task, retries counted, and the failures it will have. Free during early access, you pay OpenRouter for the calls with your own key. Banking77 verdict, failures and re-run files: https://clearbar.grain64.com/verdicts/banking77-intent/?utm_source=devto&utm_campaign=devto-banking77-intent

Which task should the next verdict cover: extraction, summarisation, or classification of your own labels? Say so in the comments.