Semantic Caching for LLMs: How to Stop Paying for the Same Answer Twice

Semantic caching stores LLM responses and serves a stored answer when a new request means the same thing as an earlier one, even if the wording differs. It works by turning each request into an embedding (a vector that represents its meaning), searching a vector store for a previous request whose embedding is close enough, and returning that request's response instead of calling the model. "How do I reset my password?" and "I forgot my password, how do I change it?" can share one cached answer. A hit costs one embedding call and a vector lookup instead of a full model call, so for repetitive traffic such as FAQ bots, support assistants, and documentation search, it removes a large share of model calls.

This post explains how semantic caching works, when it is safe and when it serves wrong answers, how to build it in your app or turn it on at a gateway, and how to tune the similarity threshold.

Prompt, response, and embedding caches

Exact caching is safe and cheap but only hits on identical requests. Semantic caching hits on paraphrases too, which raises the hit rate on natural-language traffic and also raises the risk of serving an answer to a different question. Most production setups do both: an exact hash lookup first, then a semantic search on a miss.

When semantic caching is safe

Safe:

• Answers that depend only on the prompt: FAQs, documentation Q&A, product lookups, policy questions.
• Classification of repeated inputs and deterministic transforms.
• Eval reruns and dashboards that ask the same questions repeatedly.

Not safe without a per-user cache key:

• Anything that uses private user data or account state ("what is my balance?", "summarize my tickets").
• Time-sensitive facts ("what is the status of the outage?").

Not safe by default:

• Responses that contain tool calls. Replaying a cached tool call can run a tool with another user's arguments.
• Long conversations. Late turns depend on earlier context, and they rarely repeat.

The failure mode to design against is one user receiving another user's personalized answer. Scope every cache entry to the identity and context that produced it.

Build an exact cache first

Exact caching is a few lines and has no false positives. Hash the model, the normalized messages, and the parameters, store the response with a TTL, and skip responses with tool calls:

import hashlib, json, redis
r = redis.Redis()

def cached_call(feature: str, ttl: int = 3600, **kwargs):
    key = "llm:" + hashlib.sha256(json.dumps(kwargs, sort_keys=True).encode()).hexdigest()
    if (hit := r.get(key)) is not None:
        return json.loads(hit)
    resp = client.chat.completions.create(**kwargs)
    if not resp.choices[0].message.tool_calls:
        r.setex(key, ttl, resp.model_dump_json())
    return json.loads(resp.model_dump_json())

import { createHash } from "node:crypto";
import { createClient } from "redis";
const redis = await createClient().connect();

export async function cachedCall(params: any, ttlSec = 3600) {
  const key = "llm:" + createHash("sha256").update(JSON.stringify(params)).digest("hex");
  const hit = await redis.get(key);
  if (hit) return JSON.parse(hit);
  const resp = await client.chat.completions.create(params);
  if (!resp.choices[0].message.tool_calls) await redis.set(key, JSON.stringify(resp), { EX: ttlSec });
  return resp;
}

Use temperature: 0 for cacheable calls so a cached answer is representative. In TypeScript, JSON.stringify key order follows insertion order, so build request params consistently.

Add semantic matching

A minimal semantic layer embeds the latest user message, compares it with stored embeddings, and returns the stored answer above a threshold. This sketch keeps vectors in memory to show the logic; production code uses a vector store with a TTL:

import numpy as np

THRESHOLD = 0.9
entries = []  # (scope, vector, response)

def embed(text: str) -> np.ndarray:
    v = np.array(client.embeddings.create(model=EMBED_MODEL, input=text).data[0].embedding)
    return v / np.linalg.norm(v)

def semantic_lookup(scope: str, question: str):
    q = embed(question)
    best, best_sim = None, 0.0
    for s, v, resp in entries:
        if s != scope:            # never match across tenants, users, or models
            continue
        sim = float(q @ v)        # cosine similarity on normalized vectors
        if sim > best_sim:
            best, best_sim = resp, sim
    return (best, best_sim) if best_sim >= THRESHOLD else (None, best_sim)

def answer(scope: str, question: str):
    hit, sim = semantic_lookup(scope, question)
    if hit is not None:
        return hit
    resp = client.chat.completions.create(model=MODEL, temperature=0, max_tokens=300,
                                          messages=[{"role": "user", "content": question}])
    entries.append((scope, embed(question), resp))
    return resp

scope carries everything that must match for an answer to be reusable: the tenant or user for personalized flows, the model, and the system prompt version.

Turn on semantic caching at a gateway

Building the cache into every service means reimplementing scoping, TTLs, and metrics in each one. A gateway does it once for all traffic. Bifrost, an open-source AI gateway, has a semantic_cache plugin that does a direct hash lookup first, then an optional semantic vector search on a miss (docs).

Start a vector store locally:

docker run -d --name redis-stack -p 6379:6379 redis/redis-stack:latest

Configure the plugin in config.json:

{
  "vector_store": {
    "enabled": true,
    "type": "redis",
    "config": { "addr": "localhost:6379" }
  },
  "plugins": [
    {
      "enabled": true,
      "name": "semantic_cache",
      "config": {
        "provider": "openai",
        "embedding_model": "text-embedding-3-small",
        "dimension": 1536,
        "ttl": "5m",
        "threshold": 0.8,
        "conversation_history_threshold": 3,
        "cache_by_model": true,
        "cache_by_provider": true
      }
    }
  ]
}

Weaviate, Qdrant, and Pinecone are also supported. For exact-match only (no embedding calls), set "dimension": 1 and omit provider and embedding_model.

Caching engages only on requests that carry a cache key, so nothing is cached by accident:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/openai", api_key="sk-bf-your-virtual-key")

resp = client.chat.completions.create(
    model=MODEL, messages=msgs, max_tokens=300, temperature=0,
    extra_headers={"x-bf-cache-key": "support-faq", "x-bf-cache-ttl": "1h"},
)

const resp = await client.chat.completions.create(
  { model: MODEL, messages: msgs, max_tokens: 300, temperature: 0 },
  { headers: { "x-bf-cache-key": "support-faq", "x-bf-cache-ttl": "1h" } },
);

Per-request controls:

• x-bf-cache-ttl sets the lifetime for this entry.
• x-bf-cache-threshold can only make matching stricter than the configured threshold.
• x-bf-cache-type: direct|semantic picks one lookup type.
• x-bf-cache-no-store: true reads from the cache without writing to it.

Safety defaults worth knowing:

• Entries are scoped to the requesting virtual key (and user, where resolved) on top of the cache key, so two virtual keys never share entries unless disable_identity_scoping is set.
• Tool-call responses are not cached unless cache_tool_call_responses: true.
• Conversations longer than conversation_history_threshold messages are not cached.
• A default_cache_key in the plugin config turns caching on for requests without the header. Leave it empty unless every request through that gateway is safe to cache.

Responses report hits in headers: x-bifrost-cache-hit, x-bifrost-cache-hit-type (direct or semantic), and, on semantic hits, x-bifrost-cache-similarity. Prometheus exposes bifrost_cache_hits_total, and DELETE /api/cache/clear-by-key/{cacheKey} clears entries for a key after you update the underlying content.

Tune the similarity threshold

The threshold is the trade-off between hit rate and wrong answers. Too low and "How do I cancel my order?" returns the cached answer to "How do I cancel my subscription?". Too high and only near-identical phrasings hit.

A procedure that works:

1. Start at 0.8 to 0.9 for FAQ-style traffic, scoped to one feature with its own cache key.
2. Log every semantic hit with its similarity score, the original cached question, and the new question.
3. Review a sample each day for the first week. The moment you see a hit that answers a different question, raise the threshold above that pair's similarity.
4. Keep separate cache keys (and thresholds) for features with different risk. A product FAQ can tolerate a lower threshold than a billing assistant.

Questions that differ by one word carrying the meaning ("cancel" versus "pause", "monthly" versus "annual") are where semantic caches fail. Include a few of those pairs in the review set.

Measure the savings

Report savings per 1,000 requests, so traffic changes do not hide the result:

• Hit rate: cache hits divided by total requests, per cache key. With Bifrost, sum(rate(bifrost_cache_hits_total[5m])) / sum(rate(bifrost_upstream_requests_total[5m])).
• Saved cost: hit count × average cost of a model call for that feature, minus the embedding cost on every request (hit or miss).
• Quality: the share of reviewed semantic hits that answered the right question. This number decides whether the threshold stays.

Latency usually improves too, because a hit skips generation, but an embedding call and vector lookup run on every miss. Check p95 latency before and after.

Common mistakes

• A shared cache key across users for personalized answers.
• A low threshold with no review of semantic hits.
• Caching responses that contain tool calls.
• Caching long conversations, where late turns depend on earlier context.
• Forgetting to clear entries when the source content (docs, prices, policies) changes.
• Counting only model-call savings and ignoring the embedding call on every request.

Install the llm-cost-reduction skill

Semantic caching is one part of the open-source llm-cost-reduction skill, which lets your coding agent measure where LLM spend goes and apply fixes in order of impact.

• GitHub: github.com/build-with-ai-hq/llm-cost-reduction
• skills.sh: skills.sh/build-with-ai-hq/llm-cost-reduction

npx skills add build-with-ai-hq/llm-cost-reduction

Ask your agent "cache LLM responses for our FAQ bot" or "reduce token costs." When the skill loads, the agent:

1. Audits every model call site, the models in use, missing output limits, uncapped loops, retries, and unrecorded usage.
2. Instruments calls so cost is known per feature, then ranks features by spend.
3. Applies fixes in order of impact: right-sized models, output and loop caps, prompt trimming, provider prompt caching, exact and semantic response caching, Batch APIs, and embedding reuse.
4. Moves central controls to a gateway (Bifrost) where that helps: semantic caching, a complexity router that sends simple requests to cheaper models, budgets, and cost attribution.
5. Verifies that cost per 1,000 requests dropped, quality held on an eval set, and caches hit where expected and never on personalized answers.

Related reading

• Prompt Caching Explained: How It Works on Claude, OpenAI, Bedrock and Gemini (and Why Your Cache Isn't Hitting)
• LLM Observability: What to Track in Production (Tokens, Cost, Latency, TTFT and Agent Traces)

FAQ

What is a semantic cache in LLM applications?

A store of previous requests and responses, indexed by embedding, that returns a stored response when a new request is similar enough to an old one. It skips the model call on a hit.

What similarity threshold should I use?

Start at 0.8 to 0.9 for FAQ-style traffic and raise it the first time you see a hit that answers a different question. Use separate thresholds for features with different risk.

Is semantic caching the same as prompt caching?

No. Prompt caching is done by the provider: it reuses the processing of a shared prompt prefix and still generates a new answer. Semantic caching is done by your app or gateway: it returns a stored answer and skips the model.

Which vector database should I use for semantic caching?

Any store with fast similarity search and TTL support works. Redis is a common choice because many teams already run it; Weaviate, Qdrant, and Pinecone also work with Bifrost's plugin.

Can semantic caching leak data between users?

Yes, if the cache key is shared across users for personalized answers. Scope entries to the user or tenant, keep tool-call responses out of the cache, and only enable caching on features whose answers depend on the prompt alone.