RAG Chunking Is a Schema Decision, Not a Preprocessing Step

TL;DR — Most RAG teams treat chunking as a tunable preprocessing knob and lean on re-ranking to clean up the mess. But re-ranking can only reorder what's in the candidate set — it can't recover information that chunking already severed. The fix is to stop conflating the unit you index with the unit you generate from, and to build eval that specifically targets boundary-spanning failures instead of aggregate recall.

Every RAG postmortem looks the same. Retrieval quality is bad, so someone swaps the embedding model. Still bad, so someone adds a re-ranker. Still bad, so someone tunes top_k and chunk overlap for the fourth time this quarter. Nobody questions the chunking strategy itself, because chunking got filed away months ago as a preprocessing detail — something you set once in a config and move on from.

That's the mistake. Chunking isn't preprocessing. It's schema design. It decides, permanently and irreversibly, what unit of meaning your system is allowed to retrieve. Every embedding model, every re-ranker, every eval harness downstream is operating inside the box that chunking drew. If the box is wrong, nothing downstream can draw outside it.

The Myth of the Preprocessing Step

The reason chunking gets demoted to an afterthought is that it looks like a mechanical problem: split documents into pieces of roughly N tokens, maybe with some overlap, maybe respecting sentence or heading boundaries. Treat it as solved and move on to the "real" ML work — embeddings, re-ranking, prompt design.

But chunking is a compression decision with no inverse function. Once a document is split, the information that lived across a boundary is gone. Not degraded — gone. If a contract clause defines a term in paragraph three and applies it in paragraph nine, and your chunker splits at paragraph five, you now have two independently retrievable fragments, neither of which contains the complete thought. No embedding model will recover what wasn't preserved. No re-ranker will surface an answer that was never assembled into a candidate in the first place.

This is the part teams consistently underweight: re-ranking is a selection operation, not a reconstruction operation. It assumes the correct answer already exists, intact, somewhere in the retrieved set, and its only job is to find it and move it to the top. When chunking has already fragmented the answer across two or three chunks, there is no single candidate that contains it. The re-ranker is being asked to solve a problem it was never designed to solve, and it will silently fail — not by erroring out, but by confidently promoting the least-wrong fragment it has.

Why Re-Ranking Can't Recover What Chunking Destroyed

Think about what a cross-encoder re-ranker actually computes: a relevance score between a query and a candidate passage, taken in isolation. It has no visibility into the candidate's neighbors, no notion that the passage it's scoring is half of a larger unit. You can have the best re-ranker available, and it will still score a severed fragment on its own flawed merits, because that fragment is the only thing it's been shown.

This is why teams see a frustrating pattern in production: swapping re-rankers moves the needle a little, then plateaus hard. The plateau isn't a re-ranker limitation. It's a ceiling imposed by the index itself. You cannot re-rank your way past information that was discarded at indexing time, any more than you can sharpen a photograph that was never in focus.

The failure is especially insidious because it doesn't show up as an obvious miss. The system still returns something plausible-sounding, grounded in a real fragment of the source document, scored highly by a re-ranker that's doing exactly what it was built to do. The output passes a quick eyeball check. It just happens to be subtly, confidently incomplete.

Decoupling Retrieval Granularity from Generation Granularity

The structural fix is to stop assuming that the unit you search over has to be the same unit you generate from. These are two different jobs with two different optimal sizes, and conflating them is the root of the problem.

Retrieval wants small, dense, semantically coherent units — a sentence, a bullet, a short paragraph — because small units embed more precisely and compete more fairly in vector space. A 50-token chunk about one specific claim doesn't get diluted by three unrelated claims sitting next to it in the same embedding. Generation wants the opposite: enough surrounding context that the model can actually answer the question, including the parts of the thought that live just outside the retrieved unit.

The architectural pattern that resolves this tension already exists under different names — parent-document retrieval, windowed expansion, hierarchical indexing — but it's worth stating as a principle rather than a technique: index atomically, expand contextually. Retrieve the smallest useful unit, then, at generation time, pull in its surrounding window, its parent section, or its linked definitions before handing anything to the model. The retrieval step finds the needle; a separate expansion step reconstructs the haystack around it on demand.

This only works if the index carries structural pointers — not just a vector and a blob of text, but an explicit link from each chunk back to its position in the source document, its siblings, and its parent. Most vector database schemas don't store this by default, because it wasn't considered part of the retrieval problem. It needs to be.

Evaluating the Boundary, Not Just the Answer

This is also where most RAG eval harnesses go quiet. Standard evals ask whether the retrieved set contains an answer or whether the generated response matches a reference. Neither question is built to catch boundary-spanning failures, because both assume the relevant information lives in a single, intact retrievable unit. If your eval set was built by writing questions against whole documents rather than against your actual chunk boundaries, it will systematically miss the exact failure mode chunking introduces.

A better eval deliberately targets boundaries. Take documents where you know a fact spans two adjacent chunks — a definition in one, its application in the next — and ask questions that require both. Measure not whether the system retrieves a relevant chunk, but whether the generation step actually received the complete thought. This is a different metric than recall or faithfulness; it's closer to a completeness check at the unit-of-meaning level, and almost nobody is instrumenting for it.

A Different Way to Think About the Index

None of this means embeddings and re-ranking don't matter. They do — but they're operating on a playing field that chunking already laid out, and they can't redraw its lines. The teams getting real gains in production RAG aren't the ones running another embedding bake-off. They're the ones who went back and rebuilt their indexing layer to separate "what gets searched" from "what gets read," with explicit structural links connecting the two.

Chunking earned its reputation as a boring setup step because it's invisible when it's wrong. It doesn't throw an error. It just quietly decides, once, what your entire retrieval pipeline is capable of ever finding — and every model you bolt on after that inherits the decision whether it knows it or not.