Dev.to AI 🤖 Ai 👁 0 📖 11 min read

Deep Dive: Why RAG Retrieval Must Be "Hybrid" — BM25 + Dense + RRF + Rerank

Deep Dive: Why RAG Retrieval Must Be "Hybrid" — BM25 + Dense + RRF + Rerank What matters: why a single retriever is not enough; how BM25 (sparse) and Dense (vector) complement each other; the two fusion strategies RRF

Deep Dive: Why RAG Retrieval Must Be "Hybrid" — BM25 + Dense + RRF + Rerank

What matters: why a single retriever is not enough; how BM25 (sparse) and Dense (vector) complement each other; the two fusion strategies RRF and weighted sum; and the boundaries and cost analysis of Rerank (cross-encoder).

Industrial context: enterprise knowledge-base Q&A — hybrid retrieval is how you lift Recall@10 by ≥15% over vector-only retrieval.

1. First, the intuition: why one retriever is not enough

The two mainstream retrieval methods each have a fatal blind spot, and the blind spots point in opposite directions — that is precisely why "hybrid" exists.

1.1 Dense retrieval — understands semantics, but is blind to exact tokens

Dense retrieval embeds query and documents and scores by semantic similarity. It handles paraphrases, informal phrasing, and the case where the same meaning is expressed without the exact keyword.

Its weakness is precisely insensitivity to exact tokens:

  • Ask "How much RAM does the iPhone 15 Pro have?", the document says "A17 Pro chip with 8GB unified memory" — semantically fine.
  • But ask about "error code E4097" or "invoice INV-2024-1337" — proper nouns / IDs / abbreviations. A semantic vector carries no "entity relation", so it easily confuses similar IDs or folds the exact token into the semantics and loses it.

Deeper reason: Dense retrieval maps discrete symbols into a continuous vector space, and in that map exact symbol-level information is smoothed away. "E4097" and "E4098" have nearly identical tokens, so their vectors are close — the embedding model was never explicitly trained to distinguish that kind of fine-grained symbol difference.

1.2 BM25 — strong on exact match, blind to synonyms

BM25 is the classic keyword/frequency retrieval, essentially inverted index + term-frequency weighting:

  • Documents are tokenized into an inverted index: term → documents containing it;
  • A query looks up each term and scores by TF (term frequency) and IDF (inverse document frequency);
  • It is excellent at exact keywords, IDs, and entities — ask "E4097", the doc literally contains "E4097", 100% hit.

The weakness is the mirror image: it does not understand semantics. The user says "how much phone memory", the doc says "8GB RAM" — no surface-level term matches, so BM25 misses.

1.3 One table: why they complement each other

Retriever Strength Weakness
Dense (vector) Semantic similarity, paraphrases, informal phrasing Exact IDs / proper nouns get blurred; diluted in long docs
BM25 (sparse) Exact keywords, IDs, entity hits Paraphrases, informal phrasing, semantic relatedness

The core insight (the soul of this article):

The two retrievers' hit sets overlap heavily but not completely. Take their union, and recall — the probability of catching the correct answer — is higher than with either single retriever.

That is the source of the mainstream acceptance criterion "hybrid retrieval lifts Recall@10 by ≥15% over vector-only".

2. BM25 deep dive

2.1 The full formula

BM25(q, d) = Σ IDF(q_i) × TF(q_i, d) × (k1 + 1) / (TF(q_i, d) + k1 × (1 - b + b × |d|/avgdl))
  • IDF(q_i) = log((N − n(q_i) + 0.5) / (n(q_i) + 0.5) + 1): N is the document count, n(q_i) the number of documents containing the term. Common terms get IDF≈0 ("the", "is"); rare terms score high.
  • TF(q_i, d): how many times the term appears in the document.
  • k1: term-frequency saturation (default 1.2–2.0), controls the ceiling on frequency gain.
  • b: document-length normalization (default 0.75), controls how much length affects the score.
  • avgdl / |d|: average document length / current document length.

The three intuitions (BM25's atomic unit is one term: it scores each term against a doc, then sums per document):

1. IDF — how scarce is this term? Common terms are worthless, rare terms are valuable. Search "Java" vs "the" in 100 résumés: "the" appears in 99 → IDF≈0; "K8s" appears in 3 → high IDF. The +0.5 is smoothing to avoid log(0); essentially a "smoothed scarcity measure".

2. TF + k1 — frequency saturation. Appearing once beats zero strongly, twice beats once a little, but 100 times barely beats 10 — diminishing returns. k1=1.2 has a low ceiling (anti keyword-stuffing); k1=2.0 raises it, good for corpora where frequency itself matters (paper retrieval). Without k1 it would be linear: 100 repeats scores 100× a single mention — unreasonable.

3. b — length normalization. A 100k-word thesis and a 200-word abstract both mention "Transformer" three times — are they equal? Long docs naturally contain more keywords, so discount by |d|/avgdl. b=0.75 moderately penalizes long docs; b=0 ignores length (news titles); b=1 scales fully by length.

Combined: final score = scarcity (IDF) × saturated frequency (TF+k1) × length correction (b).

Parameter cheat sheet

Param Default Raise Lower When to touch it
k1 1.2 more credit for high frequency, higher ceiling saturate fast, anti stuffing long/repeated content ranks too high → lower
b 0.75 heavier penalty on long docs treat lengths equally length varies and short docs are higher quality → raise
relation — k1 and b usually move together — change one, re-check the other

ES / OpenSearch / Lucene's k1=1.2, b=0.75 is a decades-validated default. Don't touch it without a concrete pain point; run defaults, watch bad cases.

2.2 Engineering essentials

1. Tokenizer choice (BM25's "eyes"). BM25 only knows "terms". English is naturally space-separated; Chinese is not — without segmentation "南京市长江大桥" is stored as one term and searching "长江" returns zero. Use jieba / HanLP for Chinese. Tokenizer choice matters 10× more than tuning k1/b.

2. Stopword list. Filter "的/是/在/了": they have IDF≈0 yet are scored on every query; worse, in a small corpus "的" may happen to miss a few docs, getting a small positive IDF and polluting ranking.

3. IDF smoothing (mandatory safety net). The raw log(N/n) divides by zero for a brand-new term (n=0). The smoothed (N−n+0.5)/(n+0.5) never crashes; mainstream search engines use the smoothed form by default.

4. Vector-space view. BM25 is at bottom a dot product of two vectors (weight = IDF × saturated TF). That is why it can be accelerated by ANN indexes, and why Qdrant / Milvus support BM25 in hybrid retrieval.

2.3 Known weaknesses of BM25

Weakness Plain words Example How vectors fix it
1. Bag of words only whether terms appear, not order "cat bit dog" = "dog bit cat" (opposite meaning) understands semantic roles, agent vs patient
2. Term independence terms scored independently, no collocation "apple phone" vs "apple fruit" indistinguishable encodes "apple phone" as one meaning
3. OOV unseen / misspelled terms die "Kubernetess" (extra s) won't match "Kubernetes" robust to typos
4. Short-query disadvantage too little signal to score searching "OOM" — near-zero discrimination links "OOM" to "out of memory / heap space" neighbors

Concept upgrade: BM25 is the ceiling of exact matching; vectors are the foundation of semantic understanding — not substitutes, complements. BM25 owns proper nouns / models / error codes / names; vectors own paraphrases / concept association / fuzzy intent; Rerank does the final precision pass over both recalls.

3. Dense retrieval deep dive

3.1 The Bi-encoder architecture

Query: [CLS] user question [SEP] → Encoder → q_vec (768-dim)
Doc:   [CLS] doc content [SEP]   → Encoder → d_vec (768-dim)
Score: CosineSimilarity(q_vec, d_vec)

Key design decisions:

  1. Why not a cross-encoder for recall: a cross-encoder concatenates query+doc and runs attention jointly — complexity O(N×Q) (every pair goes through the model); a bi-encoder is O(N) (docs encoded once, batchable, indexable). At a million docs and one query: bi-encoder ~10–50 ms (GPU batch); cross-encoder ~500 ms–5 s. Use bi-encoders for recall and cross-encoders for rerank.

  2. Training objective: contrastive learning — pull positive pairs together, push negatives apart, with the InfoNCE loss L = −log(exp(sim(q,d+)/τ) / Σ exp(sim(q,d−)/τ)); τ is the temperature controlling hard-negative weight.

  3. Common models: BGE (among the best open-source for Chinese; 8192-token context), GTE (Alibaba, multilingual), E5 (Microsoft, English).

3.2 Engineering speedups

  • HNSW index: approximate nearest neighbor, O(log N) queries, 10–100× faster than brute force;
  • GPU batch inference: both query and doc encoding batch, so GPU utilization is far higher than for a cross-encoder.

4. The hybrid pipeline

          ┌── Dense vector retrieval (embedding + HNSW) ──→ own top-K
 query ───┤
          └── BM25 sparse retrieval (inverted index) ────→
                              │
                union (dedup) → fusion score/rank → hybrid top-N downstream

Plainly: each retriever produces its own top-K, merge into one candidate pool, then fuse the ranking. Recall grows from the union; precision grows from ranking the truly relevant documents first.

Choosing top-K: too small (5 each) → small union, limited recall gain; too large (200 each) → noise, higher rerank cost. Common compromise: top-50 each, ~50–100 after merge. This is the two-stage skeleton: recall wide, rerank narrow.

Engineering details: BM25 via Whoosh / Elasticsearch / RankBM25; Dense via FAISS / Milvus / Qdrant; union dedup by document ID; Dense and BM25 run fully in parallel (they are independent).

5. Fusion: RRF or weighted sum?

5.1 RRF (Reciprocal Rank Fusion)

Think of Dense and BM25 as two "interviewers" — one ranks by semantic similarity, one by keyword match. As the hiring director you ignore the raw scores (a 0–1 similarity and a 0–30 BM25 score are not comparable) and only look at ranks:

RRF_score(doc) = Σ 1 / (k + rank_i(doc))     i ∈ {dense, bm25}
  • rank_i is the document's rank in retriever i; k defaults to 60 (empirical optimum from the MRRR paper).
  • Each document's score is the sum of the reciprocal contributions of its ranks.

Example (k=60):

Doc Dense rank BM25 rank RRF score
A 1 not in topK 1/61 = 0.0164
B 3 8 1/63 + 1/68 = 0.0306
C not in topK 2 1/62 = 0.0161

Ranking B > A > C: B is first in neither list, but appears in both, so it wins — rewarding consensus.

Why k=60: k controls how fast rank contributions decay. At k=1 the #1 gets 0.5 and #2 drops to 0.333 — winner-take-all, too aggressive; at k=200 #1 is 0.005 and #10 is 0.0048 — rank barely matters. 60 balances "rewarding the head" against "tolerating rank jitter" (near-optimal across most retrieval settings; k in 20–100 behaves similarly).

Three properties:

  1. Invariance: as long as the rank order is unchanged, RRF results are identical no matter how the raw scores are rescaled — you never need to normalize Dense similarity and BM25 scores into the same scale.
  2. Relation to Borda count: RRF is a weighted Borda count (rank r gets 1/(k+r)), with more concentrated reward on Top-1.
  3. Relation to probabilistic models: 1/(k+r) can be read as an estimate of "relevance given rank r"; RRF just sums the two retrievers' estimates.

5.2 Weighted sum

score(doc) = α × norm(dense) + (1 − α) × norm(bm25)

Both scores must first be normalized into the same scale — this is its main difference from RRF and the source of its friction. α tunes "trust semantics vs keywords".

Normalization traps:

Method Operation Pitfall
Min-Max (x−min)/(max−min) one outlier flattens everyone; min/max shift if the serving distribution differs
Z-Score (x−μ)/σ assumes normality; retrieval scores are rarely normal
Log log(1+x) good for long tails, but changes relative gaps

Even worse: switching the normalizer invalidates the previously tuned α — normalization method and α are two coupled hyperparameters, doubling tuning cost. This is why it costs more to maintain in production than RRF.

5.3 The comparison

Dimension RRF Weighted sum
Depends on ranks normalized scores
Needs normalization ❌ ✅
Tuning barely (k=60) must tune α
Robustness high (score scale irrelevant) low (depends on normalization quality)
Recommendation production default optional when you have a clear preference

Bottom line: use RRF unless you have a strong reason — it removes the two most annoying chores, score normalization and parameter tuning.

6. Rerank (precise ranking): the last gate of the retrieval pipeline

6.1 What it is

Rerank is not an LLM, and not a recall-stage retriever. It is a dedicated cross-encoder (e.g., BGE-Reranker) that takes (query, doc) pairs and outputs a 0–1 relevance score. It generates no text — only ranks.

6.2 Where it sits

Strictly after RRF fusion, before LLM generation:

query → BM25 top50 ─┐
      → Dense top50 ─┴→ [1] RRF fusion → ~80 candidates
                          → [2] Rerank (cross-encoder) scores each pair → Top-5
                          → [3] Top-5 into the prompt → LLM answer

Key clarification: RRF feeds Rerank; Rerank filters for the LLM. The three are a strict serial pipeline — no parallel, no substitution.

6.3 Why it is necessary

Recall encodes query and doc independently, and "shape similarity" alone cannot handle: semantic alignment ("RAM" ↔ "working memory"), negation ("not recommended" ↔ "recommended"), and metaphor / irony / domain abbreviations. A cross-encoder concatenates the input and runs bidirectional cross-attention at every layer — real token-level interaction that fixes missed and false recalls.

6.4 Why not use it over the whole corpus

Stage Complexity Typical latency Full-corpus?
Dense recall O(N) 10–50 ms ✅ millions
BM25 recall O(log N) 5–20 ms ✅ millions
RRF fusion O(K) <1 ms ✅
Rerank O(K×L²×layers) 50–200 ms (50 pairs) ❌ 100k ≈ 83 minutes

Rerank is 10–20% more precise than recall but ~100,000× slower. The industry standard is always two-stage: recall wide (50–100), rerank narrow (Top 5–20).

6.5 Four engineering optimizations

  1. Batch inference: pack 50 (query, doc) pairs into one batch — 5–10× throughput.
  2. Token truncation: for overly long documents, truncate the tail, keep the head (the query is at the front and attention is more sensitive to the front).
  3. Quantization: INT8/INT4 cuts memory and latency, precision loss usually <3%, verify with A/B.
  4. Distillation forward: use Rerank scores as soft labels to train a stronger bi-encoder — push some of the precision down into recall, reducing the noise that even reaches Rerank.

7. Boundaries and common pitfalls

7.1 Rerank is not a cure-all

If recall pushed the right answer outside top-50, Rerank cannot rescue it. Rerank improves ranking precision, not recall — recall comes from hybrid retrieval + chunking + embeddings. Intuitively: it picks the biggest fish already caught; it cannot catch fish still in the water.

7.2 Budget the latency

Rerank is a non-negligible share of end-to-end latency. A standard budget: Dense ≤20 ms + BM25 ≤10 ms + RRF ≤1 ms + Rerank(50) ≤100 ms → total retrieval ≤150 ms, leaving headroom for the LLM. Don't double latency for 1% precision.

7.3 Do you even need it?

With small data and tidy queries, recall top-5 may already be enough. Rule-of-thumb checks:

  1. Compare Recall@5 vs Recall@10: if Recall@10 is markedly higher (>10%), the right answer often sits beyond top-5 — Rerank is worth it;
  2. Check business precision need: customer-service QA where Top-1 matters → worth it; document browsing where one correct hit in Top-10 suffices → skippable.

7.4 Other common pitfalls

  1. Chunking: too-large chunks → noise; too-small → broken semantics. Use 512–1024 tokens with 50–100 overlap.
  2. Embedding domain fit: generic embeddings can perform poorly on vertical corpora (medical, legal) — fine-tune or switch.
  3. Cold start: newly ingested docs are unseen; confirm the index update policy (full rebuild vs incremental).
  4. Metrics: don't only watch Recall@K; add MRR and nDCG — they measure how high the right answer ranks.

One line to take away: production RAG retrieval = BM25 for exactness + vectors for semantics + RRF for normalization-free fusion + a small cross-encoder for final ranking — recall wide, rerank narrow. Drop any link, and recall or precision collapses.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.