LLM Routing: Stop Paying Premium Prices for Simple Prompts

Someone asked: can we end the Globex contract early? Seven words. The question looked short and simple, so the router sent it to the cheapest model. Half a second later the answer came back: yes, 30 days' notice. The contract says 12 months, plus a penalty. The answer is wrong, and your router picked the model that wrote it.

Routing can cut most of your LLM bill. In this article we build a router that saves money without making this mistake, and the check that catches it when it does.

One model for everything

Most teams start with one model for everything, usually the best one, because nobody wants to be blamed for a bad answer. But consider what goes through it: classifying a ticket, pulling the total from an invoice, translating a sentence. These are jobs a model at a tenth or a twentieth of the price does just as well.

You're paying frontier prices for autocomplete. So you route: each request goes to the cheapest model that can handle it. The hard part is "can handle it".

Small, medium or large?

• Classify a ticket: billing, bug or sales: small · one word from a fixed list
• Summarize a 40-page supplier contract: medium · long, but low risk
• Translate "your order has shipped": small
• "Can we end the Globex contract early?": large · 7 words, legal risk

For each of the four requests above, would you pick the small, medium or large model? Decide before reading on.

One: classify a ticket into three labels. Small. The answer is one word from a fixed list, and you can check it.

Two: summarize a 40-page supplier contract. Long, but it's a summary for a person to read. Medium is enough.

Three: translate "your order has shipped". Small, easily.

Four: the 7-word contract question from the introduction. Large. It's the shortest of the four, and the riskiest. A wrong yes has legal consequences. Length tells you what a prompt costs, not how hard it is. Request four comes back in the router code below.

Route first, then validate

Every request goes through a router before any model sees it. The router is not another LLM. It's plain code: the task type, a few risk rules, and a token count. It picks the cheapest tier that's allowed. Then the answer goes through a validator. Does it match the schema? Does it pass the checks for that task? If it fails, the request moves up one tier. That's a cascade: you only pay for the big model when the small one actually failed.

Step by step

The log below traces two requests. Acme sends an invoice. The task rule says small. The small model misses the total, so the schema check fails, and the request moves up to medium, which gets it right. One escalation, still a fraction of the large model's price. Then comes the contract question: the risk rule sees a legal task and goes straight to large. No guessing.

→ POST /ask  task=extract_invoice  tenant=acme
  route ..................... small    (task rule)
  small ....... 0.4 s  $0.0002   total=null       ✗ schema
  escalate .................. medium
  medium ...... 1.1 s  $0.0021   total=18,420.00  ✓
→ POST /ask  task=contract_qa  "Can we end the Globex contract early?"
  route ..................... large    (risk: legal)
2 requests · 1 escalation · 0 wrong answers

The router

The code below implements the router. Each task starts at a tier you tested. Classifying and translating: small. Summaries: medium. A task nobody has mapped yet goes to large. An unknown task should cost more, not risk more. Long inputs get at least medium. This is where request four is handled: anything tagged legal, payment or medical goes straight to large, whatever its length. That is the line the router from the introduction was missing.

// llm/route.ts
const TIERS = ['small', 'medium', 'large'] as const;
type Tier = (typeof TIERS)[number];

// Each task starts at the tier you tested, not the one you guessed
const TASK_TIER: Partial<Record<Task, Tier>> = {
  classify_ticket: 'small',
  translate: 'small',
  extract_invoice: 'small',
  summarize: 'medium',
};

const RISKY = new Set(['legal', 'payment', 'medical']);

export function route(req: LlmRequest): Tier {
  if (req.tags.some((tag) => RISKY.has(tag))) return 'large';

  const tier = TASK_TIER[req.task] ?? 'large';
  return req.inputTokens > 30_000 ? atLeast(tier, 'medium') : tier;
}

The cascade

Next comes the cascade. Start at the routed tier, call the model, validate. If it passes, return it. If it fails, move up one tier. And if even the large model fails, don't return its answer: hand it to a person. Validation is what makes cheap models safe: a schema with Zod, plus checks for each task (the label is in the list, the invoice lines add up to the total). Every attempt is also logged with its tier and its cost. You will need that log in the next section.

// llm/ask.ts
export async function ask(req: LlmRequest): Promise<ModelAnswer> {
  for (let tier = route(req); ; tier = next(tier)) {
    const answer = await callModel(MODEL_ID[tier], req);
    const check = validate(req, answer);
    log({ task: req.task, tier, ok: check.ok, cost: answer.cost });

    if (check.ok) return answer;
    if (tier === 'large') throw new NeedsHuman(req, check.reason);
  }
}

const validate = (req: LlmRequest, answer: ModelAnswer) =>
  req.schema.safeParse(answer.json).success
    ? TASK_CHECKS[req.task](req, answer) // label in list, lines add up…
    : { ok: false, reason: 'schema' };

Where this breaks

• Silent quality loss: re-run a daily sample on the large model
• Escalation storms: a task that escalates often belongs a tier up
• LLM routers: a model that picks the model adds a call to every request
• Managed routing: one model family, English-optimized, blind to your checks

Four ways this still breaks.

One: silent quality loss. Validation catches broken answers, not mediocre ones. Every day, re-run a small sample of cheap answers on the large model, and compare. That's what the log is for.

Two: escalation. If half the requests for a task escalate, you're paying twice. That task belongs one tier up.

Three: smart routers. A model that chooses the model adds a call to every request. Start with rules: most traffic is predictable.

Four: managed routing. Bedrock's prompt router picks between models of one family, is tuned for English, and can't see your validation results. Fine to start with. Keep your own log anyway.

Targets to hold

• 70%+ of requests on the small tier
• < 5% escalation rate per task
• 0 risky prompts on small

So what does the business get? Most requests answered by the cheapest tier: that's where the savings are. An escalation rate you can count and tune. And a hard rule that no risky question ever lands on the cheap model. The bill goes down, and you can prove quality didn't.

The numbers are targets for an illustrative scenario, not measurements.

▶️ Watch the video: https://www.youtube.com/watch?v=7kZ9mOlxuZo

💻 Full code: https://github.com/NodeNeuralAI/examples/tree/main/v5-llm-routing

🔔 More enterprise AI architecture every week: https://www.youtube.com/@NodeNeuralAI?sub_confirmation=1