My RAG tool refused to answer a question my notes clearly answered

I built a retrieval-augmented generation tool over my own notes — no LangChain, no LlamaIndex, every stage by hand.

Then I asked it a question my notes definitely answer, and it said:

I can't find the answer to that in your notes.

The answer was there. I could point at the line.

Bug one: the passage arrived mid-word

The question was "why are input and output tokens priced differently?" My notes say:

Output tokens cost about 8x more than input tokens. $2.50 versus $0.30 per million.

Here is what retrieval handed the model:

y 8x more per token than
input**. The `/stats` command prints...

Read the first two characters. That is ...roughly 8x with its opening sliced into the previous chunk.

My 500-character cut had landed mid-word. Half the sentence in chunk N, half in chunk N+1, and neither half complete enough to match anything. I raised top-k to 10 assuming the good chunk ranked lower — it never appeared, because no chunk contained that sentence whole.

Nothing reported a problem. No exception, no warning. The fact was in my notes, unreachable, and the tool looked like it was working.

The fix is to stop treating chunk size as a hard boundary — scan backwards for somewhere sensible to cut:

def _soften(text, start, hard_end, overlap, lookback):
    # The cut must stay far enough from start that the next chunk still
    # advances past the overlap, or the loop never terminates.
    span = hard_end - start
    floor = min(start + max(overlap + 1, span - lookback), hard_end)
    if floor >= hard_end:
        return hard_end

    # Paragraph break, then sentence ending, then any whitespace.
    for marker in ("\n\n", ". ", "! ", "? ", " "):
        found = text.rfind(marker, floor, hard_end)
        if found != -1 and found + len(marker) <= hard_end:
            return found + len(marker)

    return hard_end

I found the infinite-loop risk in that floor line by feeding it a 2,000-character string with no whitespace at all.

Bug two: the same bug, backwards

So I tried bigger chunks. 900 characters. More context is better, surely.

A different question broke.

My notes answer "how did I keep my API key safe?" in four numbered steps under a heading of almost that name. It is eight lines long. At 900 characters those eight lines get absorbed into a chunk dominated by unrelated material, and the embedding becomes an average of all of it. It stopped matching.

350 works because my notes are short sections with frequent headings, so ~350 characters is about one section. That is a property of my documents, not a universal constant.

The lesson is not the number. It is that chunk-count statistics cannot tell you this. Only real questions can.

The assumption: a threshold can detect an unanswerable question

Early on, three sentences on one topic scored 0.617 against each other. Three unrelated ones scored 0.031. A twenty-fold gap — so reject anything below a cut-off and you handle unanswerable questions.

Then I tried it for real.

I asked "what time does the corner shop close on Sundays?" Nothing in my notes about shops or opening hours. Best match: 0.3386 — it matched a chunk containing Sun Oct 4, because Sundays and Sun genuinely are related.

My weakest genuinely useful answer scored 0.3655.

A gap of 0.027.

I nearly set the floor at 0.35. Then I swept it across all four real questions instead of just my best example:

At 0.35 the bad question is silenced and two real ones drop to two passages. At 0.40 they return nothing.

The scores are not calibrated across questions. A question worded like my notes scores high on everything; one phrased differently scores low on everything, even when its top hit is right. An absolute threshold compares numbers that were never on the same scale.

What works instead is the prompt — giving the model an acceptable way out:

3. If the passages do not contain the answer, say so plainly - for example:
   "I can't find the answer to that in your notes." Do not guess, do not fill
   gaps from general knowledge.

Told only "answer from the context", a model handed irrelevant passages still produces something, because refusing is not an option it has. Given explicit permission, refusal becomes the compliant response.

My relevance floor ships switched off.

Three takeaways

Chunk-count statistics are not quality measurements. I spent a day comparing chunk counts and mean lengths, then picked a value that failed on the second real question I asked.

Silent failures are the ones to design against. Every real problem here produced no error — the severed sentence, the diluted section, a mismatched embedding model returning confident nonsense.

Test the weakest case, not the strongest. My threshold looked safe until I swept it across questions that were not my best example.

Code and every measurement behind these decisions, including the ones I got wrong first: github.com/Iddrisusalima/ask-my-docs

If you have built something similar, how do you handle the unanswerable-question problem? The prompt fix works, but it feels like there is a retrieval-level signal I am missing.