Is your AI pipeline paying for reasoning it never uses?

Quick question before we begin: which model is running your ticket classifier right now? If it’s the same frontier model you picked during the pilot, you’re in good company, but you’re probably also overpaying.

Imagine a situation where a team builds a support bot that reads incoming tickets and routes each one to the right queue. They choose the most capable model available to prove the idea works, the demo goes well, and nobody revisits the choice. Months later, the bot handles thousands of tickets a day, and the API bill has quietly become one of the biggest lines in the budget.

Does every AI workload need a frontier model?

In most cases, it doesn’t. Classification, field extraction, entity recognition, and short summaries need very little open-ended reasoning. What they definitely need is accuracy that holds across a huge number of runs, which is a different requirement, and not always an easier one. Compact models generally do well on retrieval, factuality, and coding benchmarks, while they still trail larger models on broad generalisation and complex task resolution.

Smaller models change more than the bill

Running inference locally keeps data on the device, removes round-trip latency, and drops the dependency on a network connection. For wearables, in-car systems or clinical devices in hospitals with patchy connectivity, that is often a difficult requirement. However, in regulated sectors, self-hosted models also give teams more control over where sensitive data is processed, which is why right-sizing AI models for enterprise workloads is as much of a governance decision.

A quick gut check for your own stack

Pick one workload and try answering these four questions:

• Does this task need reasoning, or does it mostly need consistency?
• How many times a day does it run?
• Where is its data allowed to travel?
• What does a single wrong answer cost?

You’ll likely find that those answers rule out far more options than you expected.

Why benchmark scores can mislead you

Published scores can be inflated, since many models have probably seen evaluation data during pre-training. Long inputs carry their own risk as well, because models often miss information buried in the middle of a large context window. In this scenario, testing against your own held-out data catches both problems well before production does.

None of this means you should shrink everything. Some workloads genuinely need frontier reasoning, and under-spending on those is the costlier mistake. The aim is to decide deliberately, with a routing layer and a named owner, instead of inheriting whatever the first pilot happened to use.

So, over to you: which tasks in your stack have you moved to smaller models, and did anything break along the way? Let me know in the comments.