Amazon Didn't Buy a Cheaper Model. It Bought a Decider.

Last week I argued that the price spread between models is an 80x gradient you route on, and that the saving is only real where your validator is deterministic.

Amazon just ran that experiment at a scale nobody else can. The result is stranger than "they used a cheaper model."

They replaced Claude Sonnet 4.6 with a 2-billion-parameter model on one of the largest internal ticketing systems in the world (20 categories, high volume, production traffic) and reported the same quality at 98% less cost. Cost per ticket went from $0.0222 to roughly $0.0005. Per million tickets, about $22,200 to about $500.

The obvious headline is "small model beats frontier model on cost". That is the wrong lesson.

They did not buy cheaper tokens. They deleted the generation step. The output went from 170 tokens to one.

What actually changed

Read the output row again. Sonnet produced 170 tokens per ticket: a JSON label plus a sentence of reasoning. The decider produces one token, the index of a chosen option.

Everything else follows. Input tokens fell 69% because the prompt no longer had to explain how to decide, only what the options were. Latency fell 78% because a 2B model scored twenty candidates instead of writing a paragraph. Cost fell 98% because nobody was paying output prices for text nobody read.

Do the arithmetic on the old setup. 170 output tokens at $15 per million is $0.00255. The input at 6,554 tokens and $3 is $0.0197. So the reasoning text was never the point. It was a side effect of how the model was asked to answer. Change the answer format, and the bill changes with it.

Why a 2B model can win a job a frontier model was doing

It is not doing the same job.

A decider takes three inputs: state, question, and a list of options. It returns a probability for each one, plus a confidence score. There is no free text anywhere in the output. In the 2B build described by the team: a Qwen3.5 torso with the language-model head removed and replaced by a roughly 1M-parameter pointer head, rank-16 LoRA, and a 4,096-token window.

Remove the ability to write sentences and you remove the ability to write a plausible sentence that is wrong. What is left is a scoring problem, and a 2B model can be very good at a scoring problem when the options are closed.

One more thing that rarely gets mentioned: you get a confidence number back. A generative model gives you a label and you hope. A decider gives you a label and 0.83, which you can threshold. The team tuned a confidence threshold before applying a label, so uncertain tickets go somewhere else instead of being silently mislabeled.

Most LLM pipelines don't have that. They have "the model said so".

The economics changed shape

The rule to remember: with token billing, your cost scales with traffic. With instance billing, it scales with variance.

Sonnet at $0.0222 per ticket means ten tickets an hour costs you 22 cents an hour. A g5.xlarge costs $1.408 an hour whether it processes ten tickets or five thousand six hundred. So there is a floor, and nobody published it:

Break-even = $1.408 / $0.0222 = 63 tickets per hour.

Below roughly 63 tickets per hour, one always-on instance costs more than just calling Sonnet. That is under 1.1% of the instance's capacity, so it is irrelevant at Amazon's volume. It is not irrelevant for a pilot, an internal tool, or a product that hasn't found traffic yet.

The second half of the shape is utilization. One instance saturates at about 5,600 tickets per hour, so at full utilization the cost is $1.408 / 5,600, roughly $251 per million tickets. Amazon reports about $500 per million on real traffic.

That 2x gap is not waste. It is the price of burstiness. Traffic arrives in spikes, and you provision for the spike, not for the average. So the honest way to evaluate this migration is never "cost per ticket". It is cost per ticket at your p95 rate, and that number only exists in your own logs.

I am keeping my arithmetic visibly separate from theirs for a reason. The 63.4 and the $251 are mine, derived from their published inputs. The 98% is theirs.

The three-way switch

This is the most reusable thing in the whole write-up. It is four lines of config.

labeling_mode lives in AppConfig and has three values:

• BEDROCK: Sonnet labels everything. The decider is deployed but never called.
• DUAL: both run in parallel on live traffic. Sonnet's result ships; the decider's choice and confidence are logged beside it.
• DECIDER: the decider labels everything. Bedrock is not called.

Rollback is a 45-second config change. That is the entire risk model.

# labeling.py — the migration is one config read and one diff
MODE = config.get("labeling_mode")  # "BEDROCK" | "DUAL" | "DECIDER"

def label(ticket, options):
    incumbent = lambda: sonnet_label(ticket, options)
    challenger = lambda: decider.choose(
        state=ticket,
        question=QUESTION,
        options=sort_keys(options),   # key order moves probabilities
    )

    if MODE == "BEDROCK":
        return incumbent()

    if MODE == "DECIDER":
        return challenger().choice

    # DUAL: ship the incumbent, log the challenger, judge the gap later
    shipped = incumbent()
    candidate = challenger()
    log_disagreement(ticket, shipped, candidate.choice, candidate.confidence)
    return shipped

DUAL is the actual contribution. Every disagreement is a labelled example you didn't pay an annotator for, drawn from the traffic that matters. The team's own phrasing is that it "told us more than any offline eval would have". You are not testing whether the small model is good in general. You are testing whether it is good on your queue, this week.

A clean switch plus a week of dual-run beats any benchmark you could build in a quarter.

Two traps worth stealing

1. The 4,096-token window rewrote the spec.

Twenty category descriptions, written for a frontier model, did not fit. They were rewritten as one-or-two-sentence criteria, plus an explicit "none" option.

That looks like a constraint and behaves like a gift. A lot of "the model got it wrong" complaints are really someone else's model being asked to infer a boundary the label never defined. Compressing a label into a sentence forces you to say what it means. My suspicion is that part of the reported quality gain came from the rewrite rather than the architecture. I cannot verify that, which is why it stays a suspicion.

2. JSON key order moved the probabilities by up to 0.19.

Same content, same options, different key order in the serialized object, and the returned probabilities shifted by as much as 0.19. They sort keys now.

If you A/B two prompts, two versions, or two vendors and the input serialization differs by so much as a field order, you are partly measuring the serializer. Before comparing two decision outputs, normalize: sorted keys, fixed whitespace, stable option ordering, canonical casing. Determinism is not a property of the model. It is a property of the entire call.

When this does not work

Their own list is refreshingly honest:

• labels need explanation text, not just a choice
• the catalog changes weekly
• the context will not fit in a few thousand tokens

I would add two:

• low volume: see the 63 tickets per hour
• you need a rationale: a probability is not an explanation. If a human reviewer, an auditor, or a regulator needs the why, a decider gives you a number and nothing to quote

That is the general shape. A decision model does not replace your LLM, it partitions your calls. Closed-set choice from a known catalog, high volume, latency-sensitive, no rationale required. That is a decider. Everything else stays on a generative model.

The Monday test

Do not start with the model. Start with the count.

Sample 200 calls from last week's logs and ask one question per call. Does the prompt end with "choose one of:"?

If that is true for a third of your traffic, you have a classification problem you are paying generation prices for. Run the open 2B decider against those logged calls, offline, and count agreements. One afternoon, no production risk, and it produces the only number that matters: how much of your bill is a decision wearing a costume.

What this post is not

I did not run this model and I have no access to Amazon's ticket queue. Every figure attributed to Amazon comes from their engineering write-up of 9 November 2026. The 63 tickets per hour and the $251 per million are my arithmetic on their published inputs.

It is also a vendor post about a vendor product used inside the vendor. There is no independent accuracy percentage in it. "Just as well" is not a benchmark. Treat the cost claim as solid, because it is arithmetic. Treat the quality claim as a claim.

Sources

• Amazon engineering write-up, 9 Nov 2026: Amazon saves millions by replacing Sonnet 4.6 with Strands Decider
• Model architecture and benchmarks: Introducing Strands Decider 2B
• Instance pricing used above: AWS SageMaker ml.g5.xlarge, us-east-1 on-demand, $1.408/hour. Verify before you quote it.

Your turn

The question I want answered is not which model you use.

What share of your LLM calls are actually decisions, meaning a closed choice from a list you control?

If you don't have the number, sample 100 calls and count. That percentage is your ceiling on this entire line of savings, and it decides whether the 98% is available to you or merely interesting.