Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 7 min read

Semantic Cache Poisoning in RAG Pipelines: Embedding Collision, Staleness Windows, and the Cache Invalidation Trap

Semantic Cache Poisoning in RAG Pipelines: Embedding Collision, Staleness Windows, and the Cache Invalidation Trap Semantic caching is the obvious cost-control lever in a RAG pipeline. Instead of hitting your vector st

Semantic Cache Poisoning in RAG Pipelines: Embedding Collision, Staleness Windows, and the Cache Invalidation Trap

Semantic caching is the obvious cost-control lever in a RAG pipeline. Instead of hitting your vector store and LLM for every query, you embed the incoming question, search a cache of prior query embeddings by cosine similarity, and return the cached answer if the score clears a threshold. The latency win is realβ€”sub-millisecond Redis lookup versus hundreds of milliseconds of retrieval plus generation. The problem is that the cache becomes a vector attack surface that most teams do not model, and the invalidation semantics are subtler than any key-value cache you have operated before.

What Semantic Caching Actually Stores

A typical implementation stores three things per cache entry: the raw query text, its embedding vector, and the serialized LLM response (which already encodes the retrieved chunks at the time of generation). In Redis, this usually means a hash holding the text and response alongside an entry in a separate vector index (HNSW or flat, depending on your redis-stack build) keyed by the same entry ID.

The retrieval path at request time:

type SemanticCacheEntry struct {
    QueryText  string
    EmbedVec   []float32
    Response   string
    SourceHash string // hash of chunk IDs used during generation
    CachedAt   time.Time
}

func (c *SemanticCache) Lookup(ctx context.Context, queryVec []float32, threshold float32) (*SemanticCacheEntry, error) {
    results, err := c.redisClient.VectorSearch(ctx, c.indexName, queryVec, 1)
    if err != nil {
        return nil, fmt.Errorf("vector search: %w", err)
    }
    if len(results) == 0 || results[0].Score < threshold {
        return nil, nil
    }
    var entry SemanticCacheEntry
    if err := c.redisClient.HGetAllInto(ctx, results[0].ID, &entry); err != nil {
        return nil, fmt.Errorf("cache fetch: %w", err)
    }
    return &entry, nil
}

The SourceHash field is the one most implementations omit. It encodes which document chunks contributed to the cached answer. Without it, you cannot reason about whether the cached response is still valid after a corpus update.

The Three Failure Modes

1. Embedding Collision at Threshold Boundaries

Cosine similarity in high-dimensional space is not intuitive. Two queries that are semantically adjacent but factually divergent can produce similarity scores above a 0.92 threshold. Classic example: "What is our refund policy for digital goods?" and "What is our refund policy for physical goods?" differ by one token, share most of their semantic geometry, and may score 0.97. If the corpus has distinct answers for both, returning the cached digital-goods response to the physical-goods query is a correctness failure, not just a cache miss.

The mitigation is not purely threshold tuningβ€”lowering the threshold increases cache miss rate, directly eroding the cost benefit. The real fix is query fingerprinting: extract named entities and domain-specific tokens from the query before the similarity lookup, and require that the cache hit's stored query share a minimum token overlap. This is a hybrid retrieval check, not a replacement for vector similarity:

func tokenOverlap(a, b string, stopwords map[string]struct{}) float64 {
    tokA := tokenize(a, stopwords)
    tokB := tokenize(b, stopwords)
    intersection := 0
    for t := range tokA {
        if tokB[t] {
            intersection++
        }
    }
    union := len(tokA) + len(tokB) - intersection
    if union == 0 {
        return 0
    }
    return float64(intersection) / float64(union)
}

Require both cosineSimilarity >= 0.92 AND tokenOverlap >= 0.6 before accepting a cache hit. This rejects the digital/physical refund collision without abandoning semantic caching for genuine paraphrases.

2. Staleness Windows After Corpus Mutation

Document ingestion pipelines update the vector store asynchronously. A support article gets revised, the embeddings are recomputed and upserted, but the semantic cache still holds an LLM response generated against the old chunk. The cached answer is now factually wrong, and it will be served with higher confidence than a fresh queryβ€”because it cleared the similarity threshold for all semantically equivalent future queries.

This is worse than a stale CDN edge cache. A stale CDN response is at least consistent with a prior correct state. A stale semantic cache response may cite a policy, price, or procedure that no longer exists.

Key-based invalidation is not applicable here because queries are not keyedβ€”they are embedded. The correct model is source-side invalidation: when the ingestion pipeline upserts or deletes a chunk, it must publish the affected chunk IDs to a fanout channel. The cache layer subscribes and scans entries whose SourceHash references any of those chunk IDs.

func (c *SemanticCache) InvalidateByChunkIDs(ctx context.Context, chunkIDs []string) error {
    affected, err := c.findEntriesBySourceChunks(ctx, chunkIDs)
    if err != nil {
        return err
    }
    pipe := c.redisClient.Pipeline()
    for _, id := range affected {
        pipe.Del(ctx, id)
        pipe.ZRem(ctx, c.indexName, id) // remove from vector index
    }
    _, err = pipe.Exec(ctx)
    return err
}

This requires findEntriesBySourceChunks to maintain a secondary index from chunk ID to cache entry IDsβ€”a set stored per chunk in Redis. The ingestion pipeline writes to this index during cache population. This is overhead, but it is the only mechanism that gives you correctness guarantees after corpus mutation.

3. Threshold Poisoning via Adversarial Queries

If your RAG pipeline is exposed to untrusted users, an attacker can deliberately craft queries that are semantically similar to high-value cached entries (policy answers, pricing, access control information) and observe whether the response latency is lower than a fresh retrieval would produce. Sub-millisecond responses betray cache hits. The attacker then uses timing to infer whether a cached entry exists for a particular topicβ€”a side-channel that leaks which queries your system has processed before.

More aggressively, if the cache write path is also accessible (the attacker can get answers cached), they can inject a poisoned response by submitting a query that embeds near an existing high-value query cluster, receiving a fresh LLM response, and having that incorrect response cached for future victims.

Mitigations:

  • Add jitter to all cache-hit response paths so latency does not reveal hit/miss status.
  • Require authentication scoping: cache entries should be partitioned by tenant or role, preventing cross-tenant cache hits.
  • Never cache responses to queries that contain PII, user-specific context, or that produce responses referencing authorization state.

The Invalidation Architecture

[ Ingestion Service ]
        |
        |-- upsert chunk --> [ Vector Store (Qdrant / pgvector) ]
        |-- publish chunk_ids --> [ Redis Pub/Sub or SQS ]
                                        |
                              [ Cache Invalidation Worker ]
                                        |
                              scan SourceHash index
                              delete affected cache entries
                              delete from HNSW vector index

The invalidation worker must be idempotentβ€”chunk IDs may be published multiple times during reingestion. Deletion of a non-existent key is safe in Redis; deletion from the HNSW index requires checking whether the entry exists first, or accepting the benign error.

The worker should also enforce a maximum TTL on all cache entries independent of invalidation events. Corpus mutation events can be dropped (worker crash, queue backlog). A TTL of 24–72 hours depending on corpus update frequency provides a backstop. This is not a substitute for event-driven invalidation; it is defense in depth.

Observability Requirements

A semantic cache without observability is worse than no cacheβ€”you cannot distinguish between correct hits, stale hits, and collision hits from metrics alone. Instrument:

  • Cache hit rate by similarity score bucket (0.90–0.94, 0.94–0.97, 0.97–1.0): collisions cluster in lower buckets.
  • Staleness age at hit time: time since CachedAt. Distributions with long tails indicate invalidation lag.
  • Post-hit user correction rate: if downstream feedback exists, correlate corrections with cache hits to estimate incorrect-hit rate.
  • Invalidation lag: time between chunk upsert and cache entry deletion. Alert if P99 exceeds your staleness tolerance.

All four metrics are instrumentable in Go with standard Prometheus histograms and counters without requiring changes to the LLM call path.

Decision Framework

Before enabling semantic caching in production, answer these questions:

  1. Can you instrument SourceHash on write? If your cache layer does not track which corpus chunks contributed to each cached response, you cannot do source-side invalidation. Do not ship without this.
  2. What is your corpus mutation frequency? High-frequency corpora (hourly updates) require aggressive TTLs and reliable invalidation infrastructure. Caching may cost more in operational complexity than it saves in token spend.
  3. Is the query surface trusted? Untrusted, multi-tenant query surfaces require tenant-scoped cache partitions and response-path jitter. Shared caches across trust boundaries are a correctness and information-leakage risk.
  4. Do you have hybrid validation? Pure vector similarity thresholds produce collision false positives. Add token overlap or keyword extraction as a second gate before shipping to production traffic.
  5. Can you tolerate a stale answer? Some domains (static FAQs, versioned documentation) tolerate bounded staleness. Others (pricing, compliance, access policy) do not. Apply semantic caching selectively by query category, not uniformly across all RAG traffic.

Semantic caching is a legitimate cost-control mechanism. It also introduces failure modes that have no analog in conventional key-value cachingβ€”embedding geometry, corpus provenance, and cross-query information leakage all require explicit design. The teams that operate it well treat it as a distributed data structure with its own consistency model, not as a transparent optimization layer.

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.