Five few-shot examples nearly doubled the cost of my short LLM prompt

You write a prompt that turns messy expense notes into JSON. The model gets the format slightly wrong once, so you add an example of a note and the JSON you expect. Then a second one for euros, a third for dates, and two more for luck. The output looks great and the prompt ships. What doesn't show up in testing is that those five examples get sent and billed again on every call, and in a short prompt they end up costing more than the task itself.

What you're paying for

Model APIs bill per token, a chunk of text about three-quarters of an English word. Input tokens are everything you send; output tokens are what the model writes back, and they're usually priced five or six times higher.

A few-shot example is a worked sample in the prompt: an input plus the answer you want, so the model can copy the pattern. A prompt with no examples, just instructions, is called zero-shot. The API keeps no memory between calls, so examples aren't "learned" once. They're part of the input, and you pay for them on every request.

Measuring it

I wrote a realistic extraction prompt. It has 58 tokens of instructions (return amount, currency, category and date as JSON), five examples, and one new note to process: "train ticket berlin to hamburg, 79.90 EUR, 14 june". I counted tokens with tiktoken, OpenAI's open-source tokenizer library (other vendors count slightly differently). Each example came to 44–51 tokens. The note itself is 15 tokens and the JSON answer is 31.

PRICE_IN, PRICE_OUT = 2.00, 12.00   # GPT-5.6 list price, $ per 1M tokens
OUT = 31                            # answer tokens, the same in every variant
MONTH = 300_000                     # 10,000 notes a day for 30 days

for shots, tin in [(0, 78), (1, 128), (3, 226), (5, 321)]:   # measured input
    cost = (tin * PRICE_IN + OUT * PRICE_OUT) / 1e6
    print(f"{shots} examples: ${cost:.6f}/call  ${cost * MONTH:,.2f}/month")

The prices are list prices, meaning the ones each vendor publishes, taken from the catalog of the LLM gateway I work on on 8 November 2026. On GPT-5.6:

With five examples, 48% of what you pay goes to text the model has already seen 299,999 times that month. Each example costs about $30 a month by itself. The note you actually wanted processed is 15 tokens.

On other models the pattern is the same. GPT-5.6-mini has the same price ratio, so it lands on the same 1.92x, only it goes from $15.84 to $30.42 a month. Claude Haiku 5.5, Claude Sonnet 5.5 and Gemini 3.8 Flash price output at 5x input, which makes input a bigger share of the bill: five examples come out at 2.04x. A cheaper model gives you smaller dollar amounts, but it doesn't change the multiplier, because that comes from how much your examples weigh compared to the actual work.

That also tells you when you can stop worrying. Put the same five examples (243 tokens) in front of a 3,000-token document and they add 7.5%. Examples get expensive when the task is short, and extraction, tagging and routing prompts usually are.

Keep the example that fixes a mistake

Most examples get added for one of two reasons: to fix the format or to fix a judgment call. Those two have different fixes.

For format, most APIs now offer structured output, a setting where you pass a JSON schema and the model's answer is constrained to match it. That takes care of field names, types and "JSON only" without spending a single example token. Then look at what's left. In my set, only one example covers a case the model might actually get wrong: "38,90 GBP, 11/04", with a decimal comma and a day-first date. Keep that one and drop the rest. That puts you at 1.19x instead of 1.92x.

Don't decide this by feel. An eval set is a small batch of inputs where you've written down the correct answer by hand. Hand-label 100 real notes and run all four variants.

def accuracy(prompt_builder, labelled):
    """Share of hand-labelled notes the prompt gets exactly right."""
    hits = sum(call_model(prompt_builder(note)) == expected
               for note, expected in labelled)
    return hits / len(labelled)

Here, call_model is your own API wrapper that parses the JSON. Four variants times 100 notes on GPT-5.6 costs $0.30 in total, which is less than what one unneeded example costs you in a single day at this volume. If zero-shot with a schema scores the same as five examples, you have your answer.

What this doesn't solve

Sometimes examples really do work better. For fuzzy decisions, such as tone, categories with blurry edges, or a house style, a couple of good examples can beat a paragraph of rules, and dropping them can cost you more in errors than you save on tokens. My numbers are about cost only, since I didn't measure accuracy on this prompt. That's why the eval step isn't optional.

Prompt caching, where the provider charges less for a prompt prefix it has seen recently, can make a long, fixed block of examples cheaper. But providers set a minimum prefix length for caching, and a 321-token prompt like this one is usually below it, so check your provider's rules before counting on it. Examples also don't count against you only in money: they make the prompt longer to read and maintain, and outdated examples quietly teach the model outdated formats. And no gateway can trim them for you, including the one I work on: it sees your prompt, not which of your examples are actually needed.

The takeaway

An example in the prompt isn't a one-time lesson. It's a line item you pay on every call. Let a schema handle the format, keep only the examples that fix mistakes your eval set actually catches, and recheck when your prompt is short.