RAG retrieval: hybrid search, reranking and measuring recall

When a RAG system gives a wrong answer, the model usually isn't the problem. It answered from the chunks it was given, and the right chunk wasn't among them. Fix retrieval and most of the bad answers go away.

Measure retrieval on its own

Before changing prompts or models, write down 50 to 100 real questions and, for each one, the chunk or chunks that actually answer it. Then measure retrieval without the model in the way.

• recall@k: The share of questions where at least one correct chunk appears in the top k results. If recall@5 is 60%, the model is missing the answer on four questions out of ten before it writes a word.
• MRR: Mean reciprocal rank: 1 divided by the position of the first correct chunk, averaged over questions. It tells you whether the right chunk is near the top or scraping in at rank 9.

Every change below should move one of these numbers. If it doesn't, it isn't helping.

Chunk by structure, not by character count

Fixed 500-character splits cut tables in half and separate step 4 from steps 1 to 3. Split on the document's own structure instead: headings, paragraphs, list items, table rows kept together. Prefix each chunk with its heading path, like "Billing > Refunds > Partial refunds", so a chunk that says "this takes 5 days" still knows what "this" is.

Size is a trade-off. Tiny chunks lose context. Huge chunks blur several topics into one embedding, and the one sentence that matters gets diluted.

Hybrid search: keywords and vectors together

Embeddings are good at meaning and bad at exact strings. Ask about error E1042, SKU 88-210 or a person's surname and vector search often returns things that are about the same topic but not that exact thing. Keyword search such as BM25 handles exactly those cases.

Run both and merge the result lists. Reciprocal rank fusion is the simple, robust way to do it, because it uses only the rank in each list, not the scores, which aren't comparable between the two systems:

def rrf(result_lists, k=60):
    scores = {}
    for results in result_lists:
        for rank, doc_id in enumerate(results, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

k=60 is the value from the original RRF paper and a reasonable default.

Rerank the candidates

Embedding search compares a query vector with chunk vectors that were computed separately. A cross-encoder reranker reads the query and each chunk together, so it judges relevance much better, but it's too slow to run over the whole corpus. Retrieve 50 candidates cheaply, rerank them, keep the top 5.

Filter before you rank

If the question is about one product, one year or one customer's documents, apply that as a metadata filter, not as something you hope similarity will sort out. Permissions especially: filter out documents the user isn't allowed to see before retrieval. A prompt that says "don't mention confidential documents" is not access control.

Plan for finding nothing

Some questions have no answer in your data. If the best reranked score is below a threshold you've checked against your test set, tell the model nothing relevant was found and let it say so. "I couldn't find this in the documentation" is a good answer. A confident answer stitched together from loosely related chunks is the worst one.

Originally published at anukumar.site.