Going from top-5 to top-20 retrieval makes every RAG answer 2.7x more expensive

Your support bot answers from your docs. It gets a wrong answer in testing, so you raise top_k from 5 to 20 in the retriever settings, the next answer comes out right, and you ship it. Nobody notices that each answer now costs 2.7 times as much, because the setting lives in the search step, far from the line that calls the model.

Where a RAG answer's money goes

RAG (retrieval-augmented generation) means that before asking the model, your code searches your own documents and pastes the best matches into the prompt. The documents are cut in advance into chunks, pieces of a few hundred words. The search gives each chunk a score, how close it is to the question, and top_k is how many of the best-scoring chunks you paste in.

Model APIs bill per token, a chunk of text about three-quarters of an English word. Input tokens are what you send, output tokens are what the model writes back, and output costs more per token. A short question like "How do I change the email address on my account?" is about a dozen tokens. Everything else you send is your instructions and the retrieved chunks.

Here is one answer with 400 tokens of instructions and question, 500-token chunks, and a 250-token answer. Those sizes are my assumptions, typical defaults rather than a measurement.

PRICE_IN, PRICE_OUT = 2.00, 12.00        # GPT-5.6 list price, $ per 1M tokens
FIXED, CHUNK, ANSWER = 400, 500, 250     # tokens: instructions + question, one chunk, answer

for k in (3, 5, 10, 20):
    tokens_in = FIXED + k * CHUNK
    cost = (tokens_in * PRICE_IN + ANSWER * PRICE_OUT) / 1e6
    chunks = k * CHUNK * PRICE_IN / 1e6
    print(f"top_k={k:2}: {tokens_in:6,} in, ${cost:.4f} per answer, "
          f"chunks {chunks / cost:.0%} of it, ${cost * 300_000:,.0f} a month")

GPT-5.6 at its list price, the one the vendor publishes, is $2 per million input tokens and $12 per million output. The prices come from the catalog of the LLM gateway I work on, read on 6 November 2026. A month here is 10,000 questions a day for 30 days.

The question itself is about 0.3% of the bill at top_k = 5, and less as top_k grows. Each extra chunk adds $0.001 to every answer, or $300 a month at this traffic. Going from 5 to 20 adds $4,500 a month, and the answer the user reads is the same 250 tokens.

The multiplier doesn't care about the vendor

A cheaper model makes every row cheaper, but not the jump between rows. I ran the same numbers for five models: top_k = 20 costs 2.70 times top_k = 5 on GPT-5.6 and GPT-5.6-mini, and 2.81 times on Claude Haiku 5.5, Claude Sonnet 5.5 and Gemini 3.8 Flash. The ratio is set by token counts: once the chunks outweigh the answer, the call grows almost in step with top_k. On GPT-5.6-mini the same move costs $264 a month instead of $714. Smaller money, the same 2.7x.

So "use a cheaper model" and "retrieve less" are separate levers. You can pull both, and switching models doesn't undo a top_k someone raised in a hurry.

Budget the context in tokens, not in chunks

A fixed top_k pastes 20 chunks even when the first two already hold the answer, and it doesn't care that one chunk is 300 tokens and another 900. Two rules fix both: stop when the score drops below a cutoff, and stop when the chunks fill a token budget, a cap on how many tokens of context one answer may use.

import tiktoken
enc = tiktoken.get_encoding("o200k_base")   # an OpenAI tokenizer; others count a bit differently

def pick_chunks(ranked, budget=2_000, min_score=0.5):
    """ranked: (text, score) pairs from your search, best match first."""
    picked, used = [], 0
    for text, score in ranked:
        n = len(enc.encode(text))
        if score < min_score or used + n > budget:
            break
        picked.append(text)
        used += n
    return picked

ranked is what your vector search already returns, so the function sits between search and prompt. Ask the retriever for 20 candidates if you like: searching is cheap, and pasting is what you pay for. With a 2,000-token budget, the most an answer can cost on GPT-5.6 is about $0.0078, whatever top_k says. The 0.5 cutoff is a placeholder: scores mean different things for different embedding models, so look at the scores of chunks that did and didn't help on your own questions before picking a number.

What this doesn't solve

Fewer chunks can mean a missing answer. If the right paragraph ranks 12th, a budget cuts it off and the bot guesses, so check on a few dozen real questions how often the useful chunk lands in the top three or five before you tighten anything. The real fix for the wrong answer that started this is usually better chunking or search, not more context. And a gateway can't make this choice for you, including the one I work on: it sees one finished prompt, not which of its chunks mattered.

The takeaway

In RAG, you pay for what you retrieve, not for what the user asked. Treat top_k as a price setting: cap context in tokens, cut off weak matches, and raise the cap only when a measured question needs it.