Hybrid retrieval is decided by your tokeniser, not your fusion weights

Semantic search looks like it works until someone pastes a part number into the box.

Ask a vector index for HS-2841-B and it returns "related" chunks about heating systems, because embeddings treat the string as vague semantics rather than an exact identifier. Ask BM25 for the same thing and it usually gets the row — unless the tokeniser mangled the code first, which is the part nobody checks.

Hybrid retrieval is the usual answer: run both, fuse the rankings. The interesting engineering is in the two edges of that sentence: how each side tokenises your actual corpus, and how you fuse without inventing a scale.

The lexical side is a dictionary choice

Most Postgres full-text setups call to_tsquery with a language configuration. The default English config stems words, drops stop words and splits on punctuation — all reasonable, all wrong for identifiers. HS-2841-B becomes tokens that no longer match the string a user typed.

One knowledge platform I maintain sidesteps the question by using the simple dictionary, which does no stemming:

WITH keyword_query AS (
  SELECT plainto_tsquery('simple', %s) AS tsq
)
SELECT {search_projection()},
       ts_rank(c.text_search, keyword_query.tsq) AS recall_score
FROM traditional_chunks c
WHERE {where_sql}
  AND c.text_search @@ keyword_query.tsq
ORDER BY recall_score DESC, c.created_at DESC
LIMIT %s

ts_rank (not ts_rank_cd) scores term frequency against the query, and the statement runs with a hard statement_timeout, because a retrieval query that hangs is worse than one that returns less.

The simple dictionary buys exactness and loses language handling: no stemming, so "retrieval" and "retrieving" stop matching each other. That trade is fine when the corpus is technical, and wrong when it is prose.

The case the dictionary cannot fix: no spaces

Chinese, Japanese and Korean text has no spaces between words, so plainto_tsquery sees one enormous token per sentence. Nothing errors. The lexical side simply returns nothing, forever, and the hybrid score quietly becomes pure vector similarity.

The workaround in the same codebase is a second, literal pass that strips whitespace before matching:

def compact_cjk_query(query: str) -> str:
    if re.search(r"[\u3400-\u9fff\uf900-\ufaff]", query):
        return re.sub(r"\s+", "", query)
    return query

The pattern is then used in an ILIKE scan alongside a BM25-style recall pass. It is a substring match, not a tokeniser, and it has the cost you would expect on a large table — which is why it runs as one of several candidate generators with its own limit rather than as the only path.

The lesson generalises past CJK: if you cannot say which tokeniser indexed the corpus, you cannot say what the lexical half of your hybrid search does on the queries that matter. Test it with a product code and a phrase in the second language before you trust the fusion.

Fusion without inventing a scale

BM25 scores and cosine similarities live on different scales. Adding them requires normalising both, and the normalisation is where most implementations quietly break: a fixed weight tuned on one corpus fails on the next.

Reciprocal rank fusion avoids the scale question by using only the rank each retriever assigned:

def rrf_score(candidate: SearchCandidate) -> float:
    return sum(1.0 / (RRF_K + rank) for rank in candidate.ranks.values())

Each retriever contributes 1/(K + rank), so being first in one list is worth roughly the same regardless of the raw score. K (60 in this codebase) controls how fast a first-place hit stops dominating: a small K makes the top of each list decisive, a large one flattens the curve.

Two details make it survivable in production. Ties break on the best raw score, so a candidate that was first somewhere outranks one that was third everywhere:

ordered = sorted(
    candidates.values(),
    key=lambda candidate: (rrf_score(candidate), max(candidate.raw_scores.values() or [0])),
    reverse=True,
)

And the cut-off is applied to an RRF score normalised against the best candidate of this query, not to a fixed number — otherwise a query whose best possible match is weak still returns ten results, which is how retrieval quietly becomes a random sample.

When you should not use hybrid retrieval

Three cases where the extra machinery is a net loss:

• One language, clean words, short documents. BM25 alone is fast, explainable and often better. Adding vectors doubles the index and adds an embedding call per query.
• The corpus is small enough to read. Under a few hundred chunks, a linear scan over a good tokeniser beats both and is easier to debug.
• You cannot evaluate it. Fusion parameters, chunk size and threshold all have defaults that are wrong for your data. Without a small labelled query set, you are tuning by vibes.

A reranker is the same argument one level up: it improves the top of the list and costs latency on every query. Measure before adding it, and measure with the queries your users actually type — including the ones with codes in them.

The reverse view

The opposite of hybrid is deliberate simplicity: one retriever, chosen for the corpus, with the tokenisation written down and tested. I have seen single-retriever systems outperform hybrid ones simply because someone maintained the keyword index and nobody maintained the fusion weights.

Fusion is not free accuracy. It is an average of two opinions, and an average is only better than either when both are individually defensible.

The implementation is public, including the SQL: Company-Brain, and the retrieval core specifically at search.py. The longer version with file and line citations is at aiengstore.com/blog/hybrid-retrieval. More systems are at aiengstore.com.