Dev.to WebDev 🛠 Dev 👁 0 📖 7 min read

My search box scored 1.00 on a query it understood nothing about

On 21 November someone typed 18米高8米宽 into the search box on my site. That means "18 m high, 8 m wide" — a question about scaffolding, arriving from a comment thread about a job site. The search returned a table of rebar

On 21 November someone typed 18米高8米宽 into the search box on my site. That means "18 m high, 8 m wide" — a question about scaffolding, arriving from a comment thread about a job site. The search returned a table of rebar hook lengths and a chart of bricks per cubic metre of masonry.

Both results were served at a relevance score of 1.00.

The query contains no noun. Not one. It is two dimensions and nothing else, and my matcher looked at it and said: perfect match, here you go.

This is about the arithmetic that made that inevitable, why the obvious fixes don't help, and the four-line gate that replaced a scoring problem with a refusal to guess.

The failure has two stages, and the first one behaved correctly

Search on the site is a plain substring AND over an index string per card. Each card carries a data-k attribute — title, category, a one-line description, the slug, and a handful of pinyin and synonym aliases. Stage one requires every whitespace-separated token of the query to appear somewhere in that string.

// stage 1
n = apply(function (key) {
  for (var i = 0; i < toks.length; i++) {
    if (key.indexOf(toks[i]) < 0) return false;
  }
  return true;
});

Chinese has no spaces, so 18米高8米宽 is one token, and no index string contains it. Zero results. That is correct for stage one, and it is why stage two exists: people type questions, the index stores nouns, and substring matching won't bridge that on its own.

Stage two is a 2-gram fallback. Split the query into CJK bigrams and alphanumeric runs, drop segments that appear nowhere in the index, then score every card:

var gs = raw.match(/[a-z0-9]+|[\u4e00-\u9fff]+/g);   // segment the query
// CJK segment → all overlapping 2-grams; ASCII/digits → kept whole
score = hits / valid.length;                          // fraction of grams found
if (max >= 0.5) showEverythingTiedAtMax();

That "drop segments that appear nowhere" step is the clever part, and it is normally what makes this kind of fallback work at all. A user types 一米多重怎么算; 怎么算 exists on no card, so counting it in the denominator would cap every real match at 20%. Filtering noise out of the denominator is what lets the signal win.

On this query, the filter did its job perfectly — and that is the problem. 米高 and 米宽 are not in the index. Neither is anything else. What survived was the digits.

The score is hits / tokens, which rewards queries it understands least

Two grams survived, so the denominator was 2, so one hit scored 0.5 and cleared the threshold. Required matches is always ceil(n/2): four grams need two, two grams need one, and a single-gram query needs a hit to score a flat 1.00.

Read that back: the less of the query the parser could keep, the lower the bar to answer it. Confidence and comprehension move in opposite directions — the exact inversion of what a relevance score should mean.

The index has 238 cards and 4,204 distinct CJK bigrams, and the frequency distribution is brutally top-heavy:

Bigram Cards containing it Share of index
算器 206 86.6%
计算 187 78.6%
用量 64 26.9%
面积 61 25.6%

Every card title ends in "calculator", so any fragment of that word is universal. A two-gram query where one gram is 计算 scores 0.5 everywhere before it has found out what the page is about.

What the two grams actually were

Here is the part I did not expect. The surviving grams were 18 and 8, and two cards contain both. Both contain them because both index strings contain the number 180:

  • the rebar hook page, whose index text includes 180度弯钩 — a 180-degree hook;
  • the brick table, whose index text includes 120 / 180 / 240 — wall thicknesses in millimetres.

18 and 8 are substrings of 180. One page means degrees, the other means millimetres, and the scoring function saw identical evidence in both. The 1.00 was not a measurement of relevance. It was a coincidence of string layout, and it happened to land on exactly the two cards whose numbers differ in unit from the question that was asked.

Digit fragments are rare in the index, which made the coincidence look like a signal:

Fragment Cards containing it Share
18 2 1%
8 5 2%
1 13 5%
5 14 6%

And because the fallback shows everything tied at the maximum in build order rather than by score, the answer was stable, reproducible, and arbitrary. 1.5米 produced seven cards at 1.00 for the same reason: grams 1 and 5, one hit each, seven winners.

Rarity doesn't rescue you

The reflex fix is IDF — weight rare grams up, common grams down. It makes this worse. 18 appears on 2 of 238 cards, so it is maximally rare, so IDF would push its weight to the ceiling and make the wrong answer more confident.

The confusion is between a rare term and a rare substring. 18 is rare because numbers are fragments of larger numbers, not because it carries meaning. A tokenizer that treats digits as units of evidence is measuring typography, not language. Same reason 8 and 800 and 1800 all look like unrelated, highly specific facts.

The gate: does the query still contain a subject?

I stopped trying to make the score smarter and put a check in front of it instead. Strip the units, the dimension words, the numerals and the question particles. If nothing is left, the query is not a document lookup — it is a parameter lookup, and a search box has no business answering it.

var MEASURE = /米|厘米|毫米|公分|吨|公斤|千克|kg|mm|cm|多高|多宽|多长|多重|多厚|多少|几个|几根|几块|几米|[0-9.\s]/g;

function hasSignal(raw) {
  var t = raw.replace(MEASURE, "").replace(/高|宽|长|厚|重|大|小|的/g, "");
  // a CJK noun survived, or an identifier like hrb400 / c30 / pavement
  return /[\u4e00-\u9fff]/.test(t) || /[a-z]{2,}/.test(t);
}

Two details matter. The strip list is deliberately greedy — 高宽长厚重 are the words people use to describe a dimension, and leaving them in would let 18米高 pass as a subject. The second clause keeps real identifiers alive: hrb400 and c30 are not measurements, and killing them would break every steel and concrete query on the site.

When the gate returns false, the fallback does not run, and the UI says nothing was found and offers the category list. Zero results beats a plausible wrong one, because the two failures have different costs. Someone who gets nothing retypes the query. Someone who gets a hook-length table concludes the site doesn't have a scaffolding calculator and leaves.

Measured, before and after

Twenty dimension-only queries — 18米高8米宽, 6米跨度, 4.5米层高, 300高, 0.5米厚, and so on — run against the same 238-card index, with the old fallback and the gated one:

Answered Relevant
Old fallback 19 / 20 0 / 19
Gated 3 / 20 0 / 3

Nineteen out of twenty got an answer, and not one of them was about the right thing. That is the shape of the bug: not a rare wrong answer, but a system that answered almost everything and was almost never right.

I'll be straight about the residual. Three still get through, because 跨度 and 层高 are genuine nouns even though the query is a dimension: 6米跨度 returns four cards, 4.5米层高 returns two (paint coverage and wallpaper, both of which mention ceiling height in passing). And 18米高8米宽间距1.5 returns fourteen, because 间距 survives the strip — it is a real word, and it is also the one word in that sentence that describes a parameter rather than a product. A vocabulary of parameter nouns would fix those. I haven't built it, and I'd rather ship the gate that removes 16 wrong answers than wait for the one that removes 19.

The second-order cost, in the analytics

Every search reports its result count. Zero-result queries are the useful ones: a demand list of things people asked for that the site doesn't have, which I read every morning.

For as long as the fallback answered these queries, they never appeared as misses. The 18 × 8 m scaffold, the 6 m span, the wall thickness — all recorded as successful searches. The bug didn't just serve the wrong page; it deleted the evidence that the page was missing. Two weeks of demand data, silently scored 1.00.

That is the part I'd flag to anyone with a hand-rolled matcher: a false positive in search is also a false negative in your product backlog, and only one of those is visible to the user.

Reproducing it

The harness reads the built index.html, pulls every card's data-k, and runs both versions of the fallback against the same 20 queries:

score = sum(1 for g in valid if g in key) / len(valid)   # identical to the JS
if max_score >= 0.5: winners = [c for c in cards if s(c) >= max_score - 1e-9]

Run it against your own index and look at two numbers: how many cards tie at the maximum, and how many of the query's surviving grams are digits or fragments of digits. If both are non-zero, your fallback is answering questions it cannot read.

The site this came from is jianzhujisuan.com, a set of construction-quantity calculators with 238 tools on the index. The scaffolding page is the one that visitor was actually looking for, and it took a comment-section bug report for the search box to stop hiding it.

The lesson I keep re-learning, in one line: a relevance score is a claim about the query, not about the page. Mine was computed entirely from the page, and so it was 1.00 on a query that had nothing in it to be relevant to.

📰 Read the original article on Dev.to WebDev

Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.