Prompt Caching Explained: How It Works on Claude, OpenAI, Bedrock and Gemini (and Why Your Cache Isn't Hitting)

Prompt caching lets an LLM provider reuse the work it already did on the beginning of your prompt. When a new request starts with exactly the same tokens as a recent one (the same tools, system prompt, and reference documents), the provider reads that prefix from cache instead of processing it again, and bills those cached tokens at a fraction of the normal input price. Output tokens are billed as usual. The cache only hits when the prefix is byte-identical, so a timestamp in the system prompt, a reordered tool list, or a switch to a different API key is enough to make every request a miss.

This post explains how prompt caching works on Anthropic Claude, OpenAI, Google Gemini, and Amazon Bedrock as of October 29, 2026, shows working code, and walks through the usual reasons a cache never hits.

How prompt caching works

Providers cache a prompt from the start up to a point, in a fixed order. Anthropic and Bedrock document the order as tools, then system, then messages, and a change at any level invalidates that level and everything after it. That gives one rule for structuring requests:

Static content first, variable content last.

1. Tool definitions (stable order, stable serialization)
2. System prompt (no timestamps, user names, or request IDs)
3. Static reference material and few-shot examples
4. Conversation history
5. The latest user message, retrieved chunks, and any per-request values

Everything above the first byte that changes can be cached. Everything after it is processed at full price.

Prompt caching is different from response caching. A prompt cache saves input-token cost and latency on the shared prefix, and the model still generates a fresh answer. A response cache (exact or semantic) returns a stored answer and skips the model call entirely. Embedding caches avoid re-embedding unchanged text. All three are useful, and they are often confused.

Provider comparison

Sources: Anthropic prompt caching, OpenAI prompt caching, Gemini context caching, Bedrock prompt caching. Numbers change with new models, so check the page for the model you use.

Anthropic Claude

Anthropic has two modes:

• Automatic caching. Add one top-level cache_control: {"type": "ephemeral"} field. The breakpoint goes on the last cacheable block and moves forward as the conversation grows, which is the simplest option for multi-turn chat.
• Explicit breakpoints. Put cache_control on specific tool, system, or message blocks, up to 4 breakpoints per request (the automatic breakpoint uses one of the four).

The cache lifetime is measured from the start of the request that writes or reads it, and it refreshes at no cost each time the cached content is used. The input total is cache_read_input_tokens + cache_creation_input_tokens + input_tokens, where input_tokens counts only the tokens after the last breakpoint.

import anthropic
client = anthropic.Anthropic()

resp = client.messages.create(
    model=MODEL,
    max_tokens=800,
    system=[
        {"type": "text", "text": LONG_STATIC_INSTRUCTIONS_AND_DOCS,
         "cache_control": {"type": "ephemeral"}},
    ],
    # Per-request values go at the end, never in the system prompt
    messages=history + [{"role": "user", "content": f"{user_msg}\n\n(current time: {now_iso})"}],
)
u = resp.usage
print(u.cache_creation_input_tokens, u.cache_read_input_tokens, u.input_tokens)

import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic();

const resp = await client.messages.create({
  model: MODEL,
  max_tokens: 800,
  system: [{ type: "text", text: LONG_STATIC_INSTRUCTIONS_AND_DOCS, cache_control: { type: "ephemeral" } }],
  messages: [...history, { role: "user", content: `${userMsg}\n\n(current time: ${nowIso})` }],
});
console.log(resp.usage.cache_creation_input_tokens, resp.usage.cache_read_input_tokens);

Changing tool definitions invalidates tools, system, and messages. Toggling web search or citations invalidates system and messages. Changing tool_choice or adding or removing images invalidates messages. Thinking parameters and effort settings invalidate messages, and on some models more.

OpenAI

Nothing to enable. Make the prefix stable and read the counter:

resp = client.chat.completions.create(
    model=MODEL, max_tokens=500,
    messages=[{"role": "system", "content": STATIC_SYSTEM_PROMPT}, *history,
              {"role": "user", "content": user_msg}],
)
print(resp.usage.prompt_tokens_details.cached_tokens, "of", resp.usage.prompt_tokens)

OpenAI's caches live on individual machines, and routing depends on a hash of the first tokens of the prompt (including tool definitions) and the optional prompt_cache_key. On models before GPT-5.6, a stable prompt_cache_key for requests that share a prefix improves hit rates; OpenAI suggests aiming for about 15 requests per minute per key, because heavier traffic overflows to other machines. Caches are not shared across organizations.

Google Gemini

Implicit caching is on by default for Gemini 2.5 and newer models and needs no configuration. The minimum prefix is 2,048 tokens on Gemini 2.5 Flash and Pro, and 4,096 on newer models listed in the docs. Explicit caching (a cached-content resource with a TTL) is available through the generateContent API on supported Gemini 3 and 2.5 models, for large shared context that you want to cache deliberately.

Amazon Bedrock

Add a cachePoint after the content you want cached. Checkpoints are processed in the order tools, system, messages, and the minimum size is evaluated across all three combined:

"system": [
  { "text": "You are an assistant for the internal billing team. <long static instructions>" },
  { "cachePoint": { "type": "default" } }
]

Many models support a 5-minute TTL that resets on each hit, and some support "ttl": "1h" in the checkpoint. Prompt caching works on on-demand inference only, not the batch inference API. Total input tokens are inputTokens + cacheReadInputTokens + cacheWriteInputTokens.

Why your prompt cache is not hitting

When the cached-token counter stays at zero, the cause is almost always one of these:

1. Something changes near the top of the prompt. A timestamp, request ID, user name, A/B flag, or randomly selected few-shot example in the system prompt makes every request unique. Move per-request values into the last user message.
2. Tool order or JSON serialization is not deterministic. Building the tool list from a dict or set, or serializing with unstable key order, changes the prefix. Sort tools by name and use json.dumps(..., sort_keys=True).
3. The prefix is below the minimum. Anthropic processes shorter prompts without caching and returns no error, so nothing tells you.
4. The gap between requests exceeds the TTL. A user who pauses longer than the cache lifetime starts cold.
5. Requests land on different providers, keys, or organizations. A load balancer that spreads one conversation across several API keys or providers sends each turn to a cold cache.
6. History is rewritten every turn. Re-summarizing the conversation on every turn changes everything after the summary. Summarize in larger, infrequent steps.
7. Retrieved chunks sit inside the system prompt. RAG results change per query. Put them after the stable prefix.
8. Model settings change. On Anthropic, changing thinking parameters, effort, tool choice, or images invalidates part of the cache.

To find the cause quickly, log the first 2,000 tokens of two consecutive requests and diff them. The first difference before the end of your intended prefix is the cause.

When caching costs more than it saves

On providers that charge a write premium, a cache write that is never read costs more than no cache at all. On Anthropic, a 5-minute write costs 1.25x the base input price and a read costs 0.1x, so a cached prefix pays for itself on the first reuse within the lifetime and saves about 90% on each reuse after that. A 1-hour write costs 2x, so use it only when turns are minutes apart and the prefix is reused often. Cache only prefixes that actually repeat within the TTL.

Prompt caching in Claude Code

Claude Code uses prompt caching automatically. Run /usage to see a Prompt cache (main) line with the share of input tokens served from cache, the number of misses, a likely cause for the last miss (for example "tool definitions changed"), and whether the cache is warm. The Claude Code cost docs note that the first message after a break longer than the cache lifetime reprocesses the full context, which is one reason long idle sessions cost more than expected.

Keep cache hits when you use a gateway

Gateways that load-balance across keys or providers can break caching by scattering one conversation across caches. Two features in Bifrost, an open-source AI gateway, address this:

Session affinity. Send a stable x-bf-session-id per conversation or agent run, and Bifrost keeps that session on the provider and key that last served it, so each turn lands on the warm cache. Claude Code, Codex CLI, and OpenCode session headers are picked up automatically.

client = OpenAI(base_url="http://localhost:8080/openai", api_key="sk-bf-your-virtual-key")
resp = client.chat.completions.create(
    model=MODEL, messages=msgs, max_tokens=500,
    extra_headers={"x-bf-session-id": conversation_id},
)

Auto prompt caching. Many clients never send Anthropic-style cache markers, so every turn pays full input price for a prefix that barely changed. With "prompt_cache": {"auto_inject": true} on a provider, Bifrost adds a cache breakpoint to requests that arrive without one. It is off by default, and caller markers always win.

{
  "providers": {
    "anthropic": {
      "keys": [{ "name": "anthropic-key-1", "value": "env.ANTHROPIC_API_KEY", "models": ["*"], "weight": 1.0 }],
      "prompt_cache": { "auto_inject": true }
    }
  }
}

Details are in the Bifrost prompt caching docs and session affinity docs.

Install the llm-cost-reduction skill

Prompt caching is one fix in a larger cost audit. The open-source llm-cost-reduction skill lets your coding agent find the prompt-cache killers in your code and apply the rest of the fixes in order of impact.

• GitHub: github.com/build-with-ai-hq/llm-cost-reduction
• skills.sh: skills.sh/build-with-ai-hq/llm-cost-reduction

npx skills add build-with-ai-hq/llm-cost-reduction

Ask your agent "prompt caching isn't hitting" or "our Anthropic bill spiked." When the skill loads, the agent:

1. Runs a grep audit for every model call site, the models in use, missing output limits, uncapped loops, retries, unrecorded usage, and prompt-cache killers such as timestamps or UUIDs at the top of prompts.
2. Instruments calls so cost is known per feature and per request, then ranks features by spend.
3. Applies fixes in order of impact: right-sized models with an eval set, output and loop caps, prompt and context trimming, provider prompt caching with stable prefixes, exact and semantic response caching, Batch APIs, and embedding reuse.
4. Verifies that cost per 1,000 requests dropped, quality held, and caches hit.

The grep it starts with for cache killers:

grep -rnE "(system|instructions).{0,80}(datetime\.now|Date\.now|new Date\(|uuid|time\.time)" \
  --exclude-dir={node_modules,.git,dist,build,.venv} .

Related reading

• Semantic Caching for LLMs: How to Stop Paying for the Same Answer Twice
• How Much Does Claude Code Cost for a Team? Real Numbers and Per-Developer Budgets

FAQ

Is prompt caching free?

Reads are discounted, not free. On Anthropic, writes cost more than normal input (1.25x for the 5-minute TTL, 2x for 1 hour). Gemini's implicit caching needs no setup and passes the discount through on hits. For OpenAI and Bedrock, write pricing depends on the model, so check the pricing page for the one you use.

Does prompt caching change the model's answer?

No. The cache stores the processed prefix, not the response. The model still generates a new output for each request.

How do I know if prompt caching is working?

Read the usage fields on each response: cache_read_input_tokens on Anthropic, cached_tokens on OpenAI, cacheReadInputTokens on Bedrock. A value above zero on repeated prefixes means it is working.

What is the difference between prompt caching and semantic caching?

Prompt caching reuses the provider's processing of a shared prefix and still calls the model. Semantic caching returns a stored answer when a new question is similar enough to an old one, and skips the model call.

Why does my cache miss after a few minutes?

The default lifetime on Anthropic and many Bedrock models is 5 minutes, refreshed on each hit. Longer gaps start cold. Use the 1-hour option where supported, if the reuse pattern justifies the higher write price.