Day 02/90: Chunking Strategies in RAG — Why Splitting Documents Matters

You upload a document, ask a question, and get an incomplete answer.

You check the source. The information is there.

So why did the RAG application miss it?

Sometimes, the problem starts with how the document was split.

For Day 2 of my AI challenge, let’s look at how chunking affects RAG retrieval, when different approaches help, and where they can fall short.

The answer is there, but part of it is missing

Imagine a company policy says:

Employees may carry forward up to five unused annual leave days. This option is available only after completing one year of employment.

Someone asks:

“Can I carry forward my unused leave?”

If the app retrieves only the first sentence, it may answer yes without mentioning the one-year requirement.

The model received a relevant passage. It just didn’t receive everything needed to explain the policy correctly.

This is one of the challenges chunking strategies in RAG need to address.

Source:Daily Dose of Data Science

Why smaller chunks aren’t always better

In many RAG systems, documents are split into passages called chunks. Those passages become the pieces the system searches through.

Smaller chunks can make it easier to find a specific fact. But a fact may lose its meaning when separated from a heading, definition, or exception.

Larger chunks keep more context together, but they can bring unrelated information into the search results.

There’s also an embedding issue. When a system creates one vector for a chunk containing several topics, that vector has to represent all of them. A question about one small detail may be harder to match.

The useful question is:

How much text does the system need to find the answer, and how much does the model need to explain it correctly?

Those amounts don’t always have to be the same.

When the document has headings, use them

Policies, contracts, and technical manuals often tell us how information belongs together.

For example, these headings change how we should interpret the text below them:

• Permanent Employees
• Contract Employees
• Employees on Probation

If we remove those headings during processing, a passage may look like it applies to everyone.

Structure-aware chunking uses headings and sections to help preserve that context.

A chunk might carry a heading path such as:

Employment Policy → Permanent Employees → Annual Leave

One detail matters here: storing that path as metadata doesn’t automatically make it part of the embedding.

Including it in the text being embedded can help the search system match the passage. Including it in the context sent to the answering model helps the model interpret the passage.

Both can be useful, but they happen at different points in the pipeline.

This also explains why document parsing matters. A chunker can’t reliably preserve headings or table relationships that were lost during extraction.

When the structure is messy, start with recursive splitting

Not every document has clean sections.

For text with inconsistent formatting, recursive splitting is a practical starting point. It tries larger boundaries, such as paragraph breaks, before falling back to smaller ones.

It’s relatively straightforward to run and doesn’t require another model to decide where every split should happen.

The limitation is simple: a paragraph break doesn’t guarantee that an explanation is complete.

A rule may be in one paragraph and its exception in the next.

Chroma’s research report found that appropriately configured recursive splitting performed competitively in its evaluation. That’s a useful reminder to give a simple baseline a fair comparison before adding complexity.

When topics change without headings, semantic chunking may help

Some documents move between topics without marking the change clearly.

Semantic chunking uses signals such as sentence similarity to help decide which passages belong together.

This can help avoid putting unrelated discussions into the same chunk.

But there’s a catch: similar wording doesn’t always mean the information belongs together.

Two products may have nearly identical descriptions but different limits. A policy exception may sound different from the rule it changes.

So keeping similar sentences together doesn’t guarantee that the chunk contains everything needed for an answer.

The ACL 2025 paper Document Segmentation Matters for Retrieval-Augmented Generation explores this problem through PIC, a method that uses document summaries to guide how sentences are grouped.

The paper reports improvements on its benchmarks. That gives us a reason to consider the approach, rather than a reason to assume it will win on every document.

What overlap helps with—and what it doesn’t

Chunk overlap repeats some text between neighboring chunks.

If useful context sits near a boundary, overlap can help keep it available on both sides.

But an exception several pages away won’t be recovered by repeating a few sentences.

Too much overlap creates another problem: repeated search results.

If the retriever returns five chunks and three contain almost the same text, the model may receive less additional evidence than the result count suggests.

Overlap is most useful when it protects nearby context without filling the results with duplicates.

Search a small passage, then bring back the surrounding section

Here’s where RAG design gets more interesting.

We don’t have to use the same amount of text for searching and answering.

With parent–child retrieval, the system can:

1. Search smaller chunks to find a precise match.
2. Identify the larger section each matching chunk belongs to.
3. Send the relevant section to the answering model.

In the leave-policy example, a small chunk could help locate the carry-forward allowance. Bringing back its surrounding section could also recover the eligibility condition.

This separates two jobs: finding the right location and providing enough context to explain it.

There still needs to be a limit. Returning an entire chapter for every match can add unnecessary text and use up the context budget.

Overlapping sections should also be combined so the model doesn’t receive the same passage repeatedly.

What if the chunk loses meaning before the search even starts?

Consider this sentence:

It applies only to employees based in London.

What does “it” refer to?

A person reading the previous paragraph might know immediately. An embedding model processing that sentence in isolation may have less information to work with.

This is the problem late chunking tries to address.

In the usual approach, text is split first and each chunk is embedded separately.

With late chunking, a compatible embedding model processes a longer passage first. The system then creates individual chunk vectors from token representations that have already seen the surrounding text.

That gives each chunk’s vector more context.

However, the answering model still needs the actual supporting text. A better embedding doesn’t automatically add the previous paragraph to the retrieved passage.

Late chunking also requires a compatible embedding pipeline. Simply sending a longer document to a standard embedding API doesn’t implement it.

How do we know whether chunking is helping?

Checking whether a search result contains the answer is a start.

But for our policy example, finding five unused annual leave days is only a partial success if the eligibility condition is missing.

A stronger evaluation asks:

• Did retrieval find all the evidence needed?
• Did the answer preserve the conditions and exceptions?
• How much unrelated text reached the model?
• How much of the retrieved text was repeated?
• Did the cited passages support the full answer?

Chroma’s report looks at retrieval at the token level, measuring relevant text alongside unnecessary or repeated content.

That helps explain why a large chunk containing the answer can still be an inefficient search result.

We also need to compare context sizes fairly. Five short chunks and five long chunks give the model different amounts of information.

Recording the actual amount of retrieved text makes the comparison more useful.

What matters when choosing a chunking strategy?

Start with the kind of information the document contains and the questions people will ask.

Clear headings make structure-aware splitting useful. Weak formatting makes recursive splitting a reasonable baseline. Unmarked topic changes may justify semantic chunking.

Overlap helps with nearby boundaries. Parent–child retrieval can recover surrounding explanations. Late chunking addresses context loss when creating embeddings.

These approaches solve different problems, and they can be combined.

The goal is to preserve enough meaning that a relevant search result also becomes useful evidence.

When your RAG app misses a condition or gives an incomplete answer, look at the retrieved passages.

Did the model receive the full explanation or only a sentence that looked like one?