Why Your RAG Answers Drift: The Document Layer Nobody Talks About

If you have built a retrieval-augmented generation system, you have probably had this moment: the demo works, the prototype impresses, and then in production the answers start to drift. They're confident. They're plausible. They're wrong in ways that are hard to pin down. The instinct is to blame the model, swap in a larger one, lengthen the prompt, add a reranker. Sometimes that helps at the margins. Often it doesn't, because the ceiling you're hitting was set long before the model ever ran.

This article walks through the document layer of a RAG pipeline — the preprocessing that happens before anything reaches a vector store or a model. We'll cover why compression works, why semantic retrieval beats filename matching, what local-first deployment buys you on compliance, how a pipeline fits together conceptually, and where token cost actually comes from. No internal stack, no secret sauce — just the concepts, explained the way an engineer would explain them to a teammate.

The first principle: retrieval quality is bounded by input quality

Before any technique, one idea governs everything. A RAG system's answer is only as good as the chunk of text the retriever actually pulls. The model does not "know" your documents; it reads whatever the retriever hands it, for a few thousand tokens, and reasons from there. If the right passage isn't retrieved, no model skill saves you. If the retrieved passage is bloated with boilerplate, the model spends its attention budget on noise.

This is why teams obsess over the model and under-invest in the document layer. The model is visible and purchasable. The document layer is invisible plumbing. But the plumbing sets the floor. You can raise the ceiling all you want; if the floor is low, users still get wrong answers.

A useful framing: model choice solves "is the ceiling high enough?" Document preparation solves "is the floor stable?" Most RAG failures are floor problems.

Why compression works (and why it's safe)

Consider a legal contract as a single PDF, a few hundred megabytes. Feed it raw and you immediately hit two walls: storage and retrieval. The file is huge because it contains repeated objects, embedded media, redundant structural information, and re-encodable assets — not because the meaningful text is that large. The actual contractual language, the clauses, the signatures and seals, occupy a small fraction of the bytes.

Compression here is not "make the text smaller by throwing words away." It's re-encoding the document by purpose. Redundant objects get deduplicated. Media gets re-encoded at a fidelity appropriate to its role. The text layer and the seals — the things a human or a model needs to read and verify — are preserved losslessly. The result is typically a 60% to 80% reduction in size with the information that matters intact.

Why does this help retrieval? Two reasons. First, a smaller document fits into context windows and vector representations more cleanly, with less boilerplate competing for the model's attention. Second, large files often simply fail to load into certain models or ingestion paths — they get truncated or rejected. A trimmed document loads everywhere. The benefit is not just cheaper storage; it's that the document can actually be read at all, by more systems, more reliably.

The key engineering principle is purpose-aware encoding: decide what must be lossless (text, seals, tables of figures) versus what can be re-encoded (decorative media, redundant structures). Get that split right and compression is free correctness, not a risk.

Semantic retrieval vs filename matching

This is the single biggest qualitative gap in a RAG system, and the easiest to misunderstand.

Filename matching is what most naive setups do. You search "2024 supplier agreement" and the system looks for files named like that. If the file is called "scan_0042.pdf" or "final_final_v3.pdf," you find nothing. Even within a file, keyword search matches characters, not meaning. It cannot connect "who did we sign with" to a clause headed "Party B." It has no notion of intent.

Semantic retrieval is different. The document is split into chunks, each chunk is turned into a vector that captures meaning, and a query is turned into a vector too. Retrieval then becomes "find the chunks whose meaning is closest to the query's meaning." Now "who did we sign with" can surface the "Party B" clause even though the words don't match. The system reads meaning, not characters.

But here is the trap that burns teams: semantic retrieval is only as good as the chunks it searches. If the document was never cleaned, never cut at sensible boundaries, never deduplicated, then the semantic index is searching garbage. Semantic power amplifies the input — it does not clean it. A beautifully tuned embedding model over a messy, over-long, boilerplate-heavy corpus will still return the wrong chunk, just confidently. That's why preprocessing (clean, split, normalize) is the prerequisite, not the competitor, of semantic retrieval.

A mature pipeline often blends approaches: lexical signals for exact terms (invoice numbers, part codes) and semantic signals for intent, then a reranker to order the combined candidates. Query rewriting helps too — expanding "supplier agreement" into the forms it actually appears in. None of this rescues dirty input, but all of it compounds on clean input.

Local-first compliance: keep the data where it lives

There's a second axis that gets discussed less in technical posts but dominates in regulated industries: where does the data go?

The default cloud path is convenient. You send your documents to a hosted service, it ingests and indexes them, and you query through an API. For a marketing FAQ, that's fine. For contracts, medical records, or financials, it's a different conversation. The risk with those documents isn't that the model answers wrong — it's that the data left your network at all. Once it's outbound, your compliance posture depends on someone else's controls.

Local-first deployment runs the entire preparation and retrieval stack inside your own environment. Documents never leave the internal network. The compliance story becomes much cleaner by construction: the data simply doesn't travel. This is a different road from "buy a bigger cloud model." One trades data for cleverness; the other keeps the cleverness at home.

This matters most in industries where an answer must be auditable. If a response can't point to which file and which page it came from, it can't be audited and can't be attributed. That's not a nice-to-have; for some workflows it's a hard acceptance line. The fix is to attach provenance at preparation time — tag each chunk with its source as it's cut and indexed, then carry that provenance through to generation. When the model answers, it answers "according to page 14 of contract X." That single habit turns an unverifiable chat into a traceable system.

The RAG pipeline, conceptually

Strip away branding and a RAG document pipeline is a sequence of stages, each of which degrades the final answer if done poorly:

1. Ingestion — collect the raw documents from wherever they live. Messy formats, mixed languages, scanned images.
2. Cleaning — normalize formats, strip boilerplate, deduplicate, handle encoding.
3. Splitting — cut documents into chunks sized for retrieval. Too big and retrieval is imprecise; too small and context is lost.
4. Vectorization — turn each chunk into a meaning vector.
5. Retrieval — given a query, find the closest chunks.
6. Generation — feed the retrieved chunks plus the query to a model and produce an answer with citations.

Every stage has a failure mode that looks, from the outside, like "the model is dumb." Ingestion misses a folder → answers ignore that source. Cleaning mangles a table → numbers come out wrong. Splitting cuts mid-clause → retrieval grabs half a thought. Vectorization on the wrong granularity → semantically adjacent but irrelevant chunks rank high. Retrieval without reranking → plausible but wrong top results. Generation without provenance → unverifiable claims.

Good teams treat this as a measurable system, not a vibe. They keep a fixed set of real question-answer pairs as an evaluation collection, run retrieval quality as a quantified metric, and regression-test it regularly so they notice the week a class of questions got worse. Frameworks for offline evaluation exist and are public engineering common sense. The differentiator isn't knowing the technique; it's actually making each step reproducible and watched.

Making preprocessing observable: the underrated step

Here is the part that deserves more attention than it gets. When you prepare a handful of documents, you do it interactively and you see what happens. When you prepare thousands — compress them, convert formats, run long batch imports that take hours — the work becomes a wait, and the wait is where trust breaks.

The old pattern: start a job, walk away, come back to a black screen or an unread wall of logs, and discover too late that some files silently failed. You can't tell which step is slow. You can't tell which file choked. You can't easily undo a half-finished run.

The better pattern — and the one a RAG data-engineering base has started enabling — is to make preprocessing observable. Concretely, three reader-visible capabilities:

• Progress is visible. You can see how far a long batch job has gotten, in real steps, not a vague spinner.
• Failures are locatable. When one document fails, you know exactly which one, instead of finding the gap weeks later in production.
• Rollback is possible. If a batch goes wrong, you step back to a known-good point and redo only what's needed, rather than discarding everything.

This is not a model feature and it's not a retrieval trick. It's operational maturity applied to the document layer. And it's the line between "the pipeline runs in a demo" and "I'd trust it in production." A pipeline you can watch is one you'll actually use, because the frightening part of long batch work was never the compute — it was the blindness.

Token cost: where the money actually goes

A common misconception is that token cost is dominated by the generation step — the model writing the final answer. In reality, a large and recurring cost is retrieval and context. Every query pulls chunks into the model's context window, and those chunks are billed as input tokens. If your chunks are bloated with boilerplate, redundant across duplicates, or simply larger than necessary because the source was never compressed, you pay for that bloat on every single query, forever.

Walk the cost through:

• Uncompressed source means larger chunks, more tokens per retrieval, higher per-query cost, repeated across every user question.
• Duplicate content means the retriever may pull near-identical chunks, wasting context on redundancy.
• Poor chunking means you retrieve more text than needed to answer, again paying for tokens that don't help.
• Cold, untouched data sitting in expensive primary storage adds infrastructure cost with no retrieval benefit.

Komprise's customer research puts a number on that last point: 60 to 70% of data on primary network storage hasn't been accessed in over 90 days, yet still consumes the most expensive tier. Most hoarded data is cold. If you never trim or tier it, the cold pile grows and your retrieval gets slower and costlier for no return.

The economic argument for document preparation is therefore straightforward: spend effort once, at ingestion, to shrink and clean the source, and you pay less on every query afterward. Preparation is a one-time cost that lowers a recurring cost. Skipping it is a recurring tax.

Three data points worth keeping in mind

These aren't model stats; they're document and risk stats, and they reframe why the document layer matters:

• IBM, 2025 cost-of-data-breach report: among stolen data, customer personal information was 53%, employee personal information 37%, and intellectual property 33%. Intellectual property was also the most expensive per record, at 178 US dollars each. Raw-fed documents leak exactly these fields.
• McKinsey: knowledge workers spend roughly 19% of the work week searching for information. A knowledge base is supposed to recover that time; bad document prep just relocates the frustration.
• Komprise: 60 to 70% of primary storage data is untouched after 90 days, still on the costliest tier.

None of those are solved by a bigger model. They're solved by how you handle documents before the model ever sees them.

Chunking: the quiet determinant of recall

Retrieval is only as precise as the chunks it searches, and chunking is where most systems quietly lose quality. Too large a chunk, and a single retrieved block may contain the answer buried among ten unrelated paragraphs — the model has to dig, and often misses. Too small a chunk, and the answer's context gets split across blocks; no single retrieved piece carries enough to answer, and the model can't reconstruct the whole.

The art is matching chunk size and boundaries to the document's natural structure. A contract chunks well by section and clause. A manual chunks well by heading and procedure. A transcript chunks well by topic turn. Fixed-size character splits are the easy default and the frequent mistake — they slice mid-thought and scatter meaning. Structure-aware splitting, where the cut points follow the document's own logic, keeps each chunk coherent enough to retrieve cleanly and answer completely.

There's also overlap to consider. A small overlap between adjacent chunks prevents an answer from being cut exactly at a boundary where neither chunk contains the full thought. Used sparingly, overlap recovers recall at a modest token cost. Used heavily, it just duplicates context and inflates bills. The right amount is an empirical setting, not a dogma — which is exactly why measuring retrieval on a fixed question set matters.

Hybrid retrieval and reranking in practice

Pure semantic retrieval is powerful but not sufficient. It can miss exact-match signals: an invoice number, a part code, a legal citation. Those are character-exact facts, and a meaning-based search may rank a semantically similar but wrong chunk above the one with the literal string. Lexical search (the old inverted-index, term-frequency kind) handles exact terms beautifully and semantic search handles intent beautifully. A mature pipeline runs both and merges.

The merge needs ordering, and that's where a reranker earns its place. After the hybrid step returns a broad candidate set, a reranking model reorders those candidates by relevance to the specific query, often using a finer signal than the initial retrieval. This two-stage approach — broad recall, then precise ranking — is consistently more robust than leaning on either method alone. Query rewriting complements it: expanding "supplier agreement" into the phrasings it actually appears in recovers cases where the user's wording and the document's wording simply don't align.

None of these techniques rescues dirty input. They compound on clean input. A well-reranked search over duplicated, boilerplate-heavy, poorly split chunks still returns the wrong thing, just higher up the list. Preprocessing remains the prerequisite.

Evaluation: how to know it's actually better

The trap in this field is shipping based on feeling. "It seems more accurate now" is not data. Treat retrieval quality like any other engineering metric: define a fixed collection of real question-answer pairs — the questions your users actually ask, with known correct source passages — and measure against it on every change.

A few practices separate serious setups from vibes:

• A golden set. A fixed, independently built question-answer collection that doesn't change between runs, so comparisons are meaningful.
• Offline metrics. Retrieval recall and answer faithfulness measured automatically, not by someone reading outputs and nodding.
• Regression suites in CI. The evaluation runs on every change, so the week a class of questions degrades, you know immediately rather than via a user complaint.
• Blind A/B. Where possible, compare configurations without knowing which is which, to avoid confirmation bias.
• Real usage signal. Production queries and whether users accept answers are the ultimate ground truth, and they should flow back into the eval set.

Frameworks for offline evaluation of retrieval and generation exist and are public engineering common sense. The differentiator isn't knowing them; it's actually running them as a standing practice. That's what turns "the model feels smarter" into "retrieval recall improved 11 points and held."

Long-context models vs RAG: a false dichotomy

A common question now: if models can read entire books in one context, why bother with retrieval at all? The answer is that long context and RAG solve different problems and fail differently.

A model with a huge context can, in principle, read everything. But stuffing a massive corpus into one prompt reintroduces the noise problem: the relevant passage competes with thousands of irrelevant tokens, attention dilutes, and exact citation becomes harder. You also pay for every one of those tokens on every query, forever — the recurring cost problem from earlier. And a single giant context doesn't help you find which of ten thousand documents answers a question; it just guarantees the model saw all of them, including the wrong ones.

RAG keeps the working set small and relevant per query, which is cheaper per question and easier to cite. The two aren't enemies. A reasonable design uses retrieval to narrow to the right few chunks, then gives the model enough context to reason over them. The document layer — clean, compressed, well-chunked — is what makes that narrowing accurate. Long context is a tool; retrieval is the discipline that tells it what to read.

A concrete cost sketch

To make the token-cost argument tangible, sketch a simple case. Suppose a source corpus, uncompressed, averages chunks of 1,200 tokens, and a typical query retrieves five chunks — 6,000 input tokens per query. Now suppose preparation compresses and dedupes such that average chunk drops to 600 tokens (boilerplate and duplicates removed), and good chunking means three chunks suffice — 1,800 input tokens per query. Same question, answered from cleaner context, at roughly a third of the token cost, repeated across every user query indefinitely.

Multiply that by query volume and the saving dwarfs the one-time preparation effort. And that's before counting the storage tier: Komprise's finding that 60–70% of primary storage data is untouched after 90 days means most of what you hoard sits on the costliest tier doing nothing. Trimming and tiering the cold mass isn't just tidiness; it's a line item.

A selection framework

If you're evaluating how to handle the document layer, a generic framework helps cut through vendor noise. Score each candidate on dimensions that actually affect outcomes:

Use it as a checklist, not a scorecard to hand to a salesperson. The point is to force the conversation onto the document layer, where the real reliability lives.

Closing thought

The next time a RAG answer drifts, resist the reflex to swap the model. Look one layer down, at the documents. Were they trimmed, or fed raw? Were sensitive fields handled, or exposed? Can you trace each answer to a page? And — the question I now treat as essential — when the long batch job runs, can you actually watch it?

The model is the last step. Make the earlier steps observable and clean, and the model you already have will start looking far smarter than it did.