Dev.to AI 🤖 Ai 👁 0 📖 8 min read

Why AI coding agents slow down in long sessions, and 5 prompt-caching mistakes to avoid

Originally published at tabishapps.com/blog/prompt-caching-coding-agents. If you use Claude Code, Codex or an open model through Ollama, you've probably seen an agent that was quick at the start get slower and more expe

Originally published at tabishapps.com/blog/prompt-caching-coding-agents.

If you use Claude Code, Codex or an open model through Ollama, you've probably seen an agent that was quick at the start get slower and more expensive as the session went on. Often the reason isn't the model. It's the prompt cache being thrown away, sometimes by something as small as a timestamp.

This week Ollama shared three lessons on this, from a longer write-up by Together AI and Ollama. I read it, then tested the main claim on my own Mac. Here's what's going on, in plain words, and what to do about it.

Every turn, the agent sends everything again

A coding agent doesn't "remember" your session the way a person does. On every step, the tool you use (the harness: Claude Code, Codex, Cursor, Pi and so on) sends the model the whole conversation again: the system prompt, every tool definition, your request, every file it has read and every command it has run. The model usually answers with a short tool call, like "read this file". Then the next turn sends all of that again, plus a bit more.

So for agents, the expensive part is reading the input, not writing the answer. Without help, turn 100 would mean re-reading 100 turns of history.

Prefix caching: only read what's new

Inference engines (Ollama, vLLM, MLX, and hosted APIs like Anthropic's and OpenAI's) keep the work they did on a prompt for a while. If the next request starts with exactly the same tokens, they reuse that work and only process the new part at the end.

The Together write-up gives the simple maths: if each turn adds about the same amount, ten turns without a cache process 1 + 2 + … + 10 = 55 units of prompt, and with a cache only 10. At a hundred turns it's 5,050 against 100.

But there's one rule that everything else follows from:

The cache matches from the very first token up to the first token that's different, and stops there. Adding text at the end is cheap. Changing anything earlier, even one character, means everything after that point is processed again.

I tested it on a Mac

I wanted to see this rule in real numbers, so I ran a small test with Apple's MLX and Google's Gemma 3 12B (4-bit) on a MacBook Pro with an M3 Pro chip and 36 GB of memory. The script builds a fake agent session: a system prompt with tool definitions, then a few turns where the agent reads files. It then times how long the model takes to read the prompt for the next turn in three cases:

Case Tokens the model had to read Time before the first word
No cache (fresh start) 10,275 (the whole prompt) about 2–4 minutes
Cache kept, new turn added at the end 2,513 (only the new turn) about 1 minute (66 s)
Cache kept, but the system prompt has a new timestamp 10,244 (only the first 32 tokens matched) about 2–4 minutes

The token counts are the important part, because they don't depend on what else the laptop was doing. With the cache kept, the model read 2,513 tokens instead of 10,275, about four times less, and that gap keeps growing as the session gets longer. With one changed timestamp near the top, only the first 32 tokens matched, so the cache saved almost nothing: the model read 10,244 tokens, the same as starting fresh. (The times are rough. My two full re-reads took 134 and 234 seconds because the Mac was busy with other apps, which is why I show a range.)

A cloud model on big GPUs is much faster than a laptop, but the shape is the same: a broken cache means paying again, in time and money, for the whole session.

The test script is short and at the end of this post if you want to try it.

5 things that break the cache

1. Anything in the system prompt that changes each turn

A current time, the git status, the current folder, "turn 14 of 40". It feels helpful to keep the model up to date, but the system prompt sits at the very top, so a new value there means the whole session is read again on every turn. That's the "timestamp" row in my test.

Fix: keep the system prompt identical for the whole session (and ideally across sessions). Put changing information in a new message at the end instead. If you write your own agent or prompt templates, check for datetime.now() or similar in the system prompt first.

2. Changing reasoning effort in the middle of a session

For some open models, the chat template writes the reasoning-effort setting into the first system block. Ollama found this with GLM. So switching from "low" to "high" halfway through a long task changes the prompt almost at the first token, and the whole session is processed again.

Fix: choose the effort level at the start and leave it. If you really need more thinking, it's often cheaper to start a fresh session (or compact the old one first) than to flip the setting on a long one.

3. Adding tools near the top

Newer harnesses load tools only when the agent searches for them, which saves context. But if the new tool definition is added to the tool list at the top, everything after it is re-read. The better place is the end of the conversation. Not every model's chat template accepts that, so this depends on the harness and model.

Fix (if you build tools or MCP servers): avoid changing the tool list during a session, and prefer harnesses that add late-loaded tools at the end.

4. Huge tool results

A file the agent reads becomes part of every request after it. If it reads 3,000 lines when it needed 40, you carry those 3,000 lines for the rest of the session. Even cached, they fill the context window and memory.

Fix: ask the agent to search before it reads, and to read line ranges. If you build tools, return less by default.

5. Cutting output without saying so

This one isn't about speed, but it's the third point in Ollama's post and it matters. Many tools trim long output. If a tool returns the first 100 lines of a 640-line file without saying it stopped, the model doesn't know the rest exists and will happily edit code it never saw.

Fix: when output is cut, add one line like [showing lines 1–100 of 640; search or ask for another range]. With that hint, the model goes and reads the part it needs.

A quick checklist

  • System prompt: no dates, times, git status or folder paths that change. Put those in the latest message.
  • Pick model, reasoning effort and tools at the start of a session and don't change them midway.
  • Long, messy session? Start fresh or compact, rather than changing settings.
  • Read files by searching and line ranges, not whole files.
  • Truncated output should always say it was truncated.
  • For subagents: give them a short brief instead of the whole history when they only need a few files. A cached prefix is cheap, but not free.

The test script

It needs a Mac with Apple silicon and Python. Install mlx-lm with pip install mlx-lm; the first run downloads the model (about 8 GB for the 12B, under 1 GB for mlx-community/gemma-3-1b-it-4bit).

"""Measure how prefix caching behaves for an agent-style prompt with mlx_lm.

Scenarios (all time only prefill + 1 token):
  cold      : whole prompt for turn N+1, empty cache
  append    : cache holds turn N, only the new turn is processed
  timestamp : same as append, but the system prompt has a new timestamp -> first
              differing token is near the top, so nothing after it can be reused
"""
import sys, time, random
from mlx_lm import load, stream_generate
from mlx_lm.models.cache import make_prompt_cache

MODEL = sys.argv[1] if len(sys.argv) > 1 else "mlx-community/gemma-3-1b-it-4bit"
TURNS = int(sys.argv[2]) if len(sys.argv) > 2 else 12
model, tok = load(MODEL)

TOOLS = "\n".join(
    f"- {n}(path: str, start: int = 1, end: int = 200): {d}"
    for n, d in [("read_file", "Read part of a file and return numbered lines."),
                 ("search", "Search the project for a string and return matching lines."),
                 ("run", "Run a shell command and return stdout and stderr."),
                 ("edit_file", "Replace an exact string in a file with new text."),
                 ("list_dir", "List the files in a directory.")])

random.seed(7)
def fake_file(i):
    rnd = random.Random(i)
    lines = []
    for j in range(60):
        lines.append(f"{j+1:4d}  def handler_{i}_{j}(event, tz='UTC'):  # export step {j}")
        lines.append(f"      return convert(event['start'], tz) + {rnd.randint(1, 999)}")
    return "\n".join(lines)

def messages(turns, stamp):
    sys_prompt = f"You are a coding agent.\nCurrent time: {stamp}\nTools:\n{TOOLS}"
    msgs = [{"role": "user", "content": sys_prompt + "\n\nTask: fix the timezone bug in the calendar export."}]
    for t in range(turns):
        msgs.append({"role": "assistant", "content": f'read_file(path="src/module_{t}.py")'})
        msgs.append({"role": "user", "content": f"Tool result:\n{fake_file(t)}"})
    return msgs

def ids(turns, stamp):
    return tok.apply_chat_template(messages(turns, stamp), add_generation_prompt=True)

def prefill(tokens, cache):
    t0 = time.perf_counter()
    for r in stream_generate(model, tok, prompt=tokens, max_tokens=1, prompt_cache=cache):
        pass
    return time.perf_counter() - t0

def lcp(a, b):
    n = 0
    for x, y in zip(a, b):
        if x != y: break
        n += 1
    return n

# warm up the GPU kernels so the first timing is fair
prefill(ids(1, "warmup"), make_prompt_cache(model))

S1, S2 = "2026-11-17 10:04:11", "2026-11-17 10:04:52"
prev = ids(TURNS, S1)
nxt_same = ids(TURNS + 1, S1)
nxt_new_stamp = ids(TURNS + 1, S2)
print(f"model {MODEL}")
print(f"turn {TURNS} prompt: {len(prev)} tokens; turn {TURNS+1}: {len(nxt_same)} tokens")
print(f"shared prefix, same system prompt: {lcp(prev, nxt_same)} tokens")
print(f"shared prefix, new timestamp:      {lcp(prev, nxt_new_stamp)} tokens")

# chat templates may re-render the end of the previous prompt (generation prompt),
# so reuse exactly the shared prefix: prime the cache with it, then send the rest.
def warm(prev_ids, new_ids):
    n = lcp(prev_ids, new_ids)
    c = make_prompt_cache(model)
    if n > 1:
        prefill(new_ids[: n - 1], c)       # what the server already has from turn N
        return prefill(new_ids[n - 1:], c), len(new_ids) - (n - 1)
    return prefill(new_ids, c), len(new_ids)

cold = prefill(nxt_same, make_prompt_cache(model))
app, app_n = warm(prev, nxt_same)
ts, ts_n = warm(prev, nxt_new_stamp)
print(f"cold      : {cold:6.2f}s  ({len(nxt_same)} tokens processed)")
print(f"append    : {app:6.2f}s  ({app_n} tokens processed)")
print(f"timestamp : {ts:6.2f}s  ({ts_n} tokens processed)")

Run it with python cache_test.py mlx-community/gemma-3-12b-it-4bit 3. The last number is how many file-reading turns to put in the session.

Sources

Written by Tabish Hassan, a software developer. Found a mistake or got different numbers on your machine? Let me know in the comments.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.