Your Retriever Found the Paragraph and Lost the Heading Above It

The assistant answered
questions about the pricing policy.

Somebody asked
how long they had to ask for a refund.

Thirty days,
it said,
and quoted the policy.

The customer was on the basic plan.

The basic plan
allows fourteen.

The quote was real.

It came from the section
for business accounts.

The policy document
had a heading for each plan,
and under each heading
a paragraph that began
the same way.

Refunds are available within.

Before anything was indexed,
a script cut the document
into pieces of a few hundred tokens,
counted from the top,
with no idea where a section began.

The heading
landed at the end of one piece.

The paragraph
landed at the start of the next.

So the index held two paragraphs
about refunds,
nearly identical,
neither of which said
which plan it was about.

Asked about refunds,
the retriever found both.

It ranks by resemblance,
and they resembled the question equally.

The model received
two confident answers
with no labels,
and picked one.

Nothing was wrong
with the model.

It answered from
exactly what it was given.

Text means less
the further you cut it
from what surrounds it.

A row without its column names.
A step without its procedure.
A limit without the product it limits.

So stop cutting by count.

Split along the structure
the author already built,
headings,
sections,
list items,
rows that keep their header.

Then give every piece
its address.

Before you embed a chunk,
put the document title
and the path of headings above it
at the front.

Pricing policy,
basic plan,
refunds.

Store the plan
as a field you can filter on,
and filter
when you know who is asking.

Show the model the heading
along with the text,
and ask it to cite both.

And write the test
that catches this.

Questions whose answer
differs by section.

Same words,
different plan,
different number.

A paragraph is not a fact.

It is a fact
plus where it was written.

Keep the where.

– Serguey Asael Shinder