Lost in the Middle: Find Your LLM's Context Dead Zone and Fix It Before the Call

One result from the "Lost in the Middle" paper should change how you build RAG prompts. GPT-3.5-Turbo was given 20 retrieved documents with the answer placed in the middle of them, and it scored lower than when it got no documents at all. Answering from memory beat answering with the documents. Here the extra context made the answer worse.

Liu et al. (Stanford, UC Berkeley, Samaya AI, 2023) tested multi-document QA and key-value retrieval. They moved the relevant passage from the first slot to the last and tracked accuracy. The result was a U-shaped curve: strong at both edges, down by 20+ points in the middle. Newer models have flattened that curve, but it hasn't gone away. You won't know how your model handles your prompt shape until you measure it.

This post covers three things you can add to a pipeline today: a test that finds your own dead zone, a reorder step that keeps important chunks out of it, and a pre-flight gate that blocks oversized prompts before they're sent.

Your context window is a budget

A 200K window is the most the model will accept. You don't have to fill it. Every token you add costs money, adds latency, and competes for attention with the tokens that actually matter.

The attention cost is the one people miss. Developers tend to treat the window like a bucket: if it fits, ship it. A 3,000-line file fits. Fifteen RAG chunks fit. Forty turns of agent history fit. The model takes all of it without complaint and gets worse at the one thing you asked.

No error tells you "your instruction is buried at token 61,000." You just get an answer that ignores it.

Step 1: Measure your own dead zone

Don't rely on a 2023 chart for a 2026 model. Run a needle test with your real prompt shape. It takes about ten minutes and a few dollars.

import anthropic, random

client = anthropic.Anthropic()
MODEL = "claude-sonnet-5-5"

NEEDLE = "The deploy freeze for the payments service ends on March 14."
QUESTION = "When does the deploy freeze for the payments service end?"

def build_haystack(filler_chunks, needle, position):
    """position runs from 0.0 (start) to 1.0 (end)."""
    idx = int(position * len(filler_chunks))
    chunks = filler_chunks[:idx] + [needle] + filler_chunks[idx:]
    return "\n\n".join(chunks)

def run(filler_chunks, positions=(0.0, 0.25, 0.5, 0.75, 1.0), trials=5):
    results = {}
    for pos in positions:
        hits = 0
        for _ in range(trials):
            random.shuffle(filler_chunks)
            doc = build_haystack(filler_chunks, NEEDLE, pos)
            resp = client.messages.create(
                model=MODEL,
                max_tokens=50,
                messages=[{
                    "role": "user",
                    "content": f"{doc}\n\n{QUESTION} Answer in one short phrase.",
                }],
            )
            if "march 14" in resp.content[0].text.lower():
                hits += 1
        results[pos] = hits / trials
    return results

Use realistic filler: your own docs, old tickets, whatever your production context actually contains. With lorem ipsum the needle stands out too easily. The honest test uses a needle that looks like the text around it.

One caveat: pulling back a single string is the easy case. If your task needs the model to combine a fact from the middle with something near the end, expect worse numbers than this test gives you.

Step 2: Put the best chunks at the edges

When a retriever returns chunks ranked by relevance, most pipelines join them in rank order. Chunk #1 ends up at the top, which is good. Chunks #2 through #5 land in the middle, which is bad. And your least relevant chunks sit at the end, right next to the question.

Reverse that. Alternate the placement so the strongest chunks sit at both edges and the weakest fill the middle:

def edge_order(chunks_by_relevance):
    """Input: most relevant first. Output: best at both ends, worst in the middle."""
    front, back = [], []
    for i, chunk in enumerate(chunks_by_relevance):
        (front if i % 2 == 0 else back).append(chunk)
    return front + back[::-1]

# ranks [1, 2, 3, 4, 5, 6] -> [1, 3, 5, 6, 4, 2]

If you'd rather not maintain this yourself, LangChain has it as LongContextReorder in langchain_community.document_transformers. Either way it's only a few lines.

Two more placement rules that cost nothing:

• Put the question after the documents. The end of the prompt is the second-best position you have.
• In long-running agents, repeat the core instruction near the end of each turn. Don't count on the model still following a system prompt from forty messages ago.

Step 3: Block the prompt before it ships

Reordering helps. Not sending 120K tokens of marginal context helps more. The simplest fix is a pre-flight check: count tokens before the call, and if the prompt is over budget, trim it or fail with a clear error.

MAX_INPUT_TOKENS = 30_000  # set this from your Step 1 results, not the model card

class ContextBudgetExceeded(Exception):
    pass

def guarded_create(client, *, model, messages, system=None, **kwargs):
    params = {"model": model, "messages": messages}
    if system:
        params["system"] = system

    n = client.messages.count_tokens(**params).input_tokens
    if n > MAX_INPUT_TOKENS:
        raise ContextBudgetExceeded(
            f"{n} input tokens > {MAX_INPUT_TOKENS} budget; trim retrieval or history"
        )
    return client.messages.create(**params, **kwargs)

Where should MAX_INPUT_TOKENS come from? Your own needle curve. If mid-prompt accuracy holds at 30K tokens and drops at 80K, your usable window is 30K, whatever the pricing page says.

When the gate trips, start by dropping the lowest-ranked chunks. Then summarize old agent turns into a short state block. For code, send the function you're working on plus its direct callers, not the whole file. A model that sees 400 relevant lines will usually do better than one that sees 3,000 lines where the class definition sits in the middle.

The same gate keeps the bill down

This gate helps with cost as well as accuracy. Context bloat is a common way agent costs run away. If an agent loop appends every tool result to its history, the prompt grows with every step. Step 30 costs many times what step 1 did, and the model reasons worse by then too. As a rough example, 30 steps where the input grows by 4K tokens each come to about 1.86M input tokens for a single task.

A token cap handles the accuracy problem. A dollar cap handles the cost problem. I wrote baar-core for the second one. It's an open-source Python library that puts a hard budget on every LLM call and returns a 402 before the provider is ever contacted. It reserves spend atomically, so twenty parallel agents that each look under budget can't overshoot the cap together. pip install baar-core. For teams, noburn.dev builds on it: once a user goes over their budget, it blocks their API calls before they're sent.

Both checks go in the same spot, right before create(), while you can still stop the call.

What the advertised window tells you

The number on the model card tells you how much the model will accept. It doesn't tell you how much it uses well. You have to measure that gap for each model, task, and prompt shape. It also changes whenever a provider ships a new version, so rerun the needle test after every upgrade.

Think of the advertised window like a car's top speed. It's useful to know, but it's not the speed you should drive at.

Have you measured the effective context size of your own pipeline, and how far was it from the spec sheet?

Originally published at https://www.robatdasorvi.com/chapters/ai-automation/why-context-window-size-is-the-thing-every-developer-should-care-about