Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 2 min read

Checking every number an LLM writes against the source file

LabChure turns uploaded documents (PDF, PPTX, DOCX, HWP, images and more) into a web brochure with its own URL. Getting the model to write the page was the easy part. Numbers were the hard part. A model that rewrites a

LabChure turns uploaded documents (PDF, PPTX, DOCX, HWP, images and more) into a web brochure with its own URL.

Getting the model to write the page was the easy part. Numbers were the hard part. A model that rewrites a spec sheet will sometimes turn "12 kW" into "15 kW", and the page still reads fine.

So we added a gate. Before a page can be published, code checks its numbers against the uploaded files.

What the code checks

  • Numbers with a unit: mm, kg, kWh, Hz, rpm, Β°C, %, inch and similar
  • Prices: β‚© $ € Β₯ Β£, or amounts with KRW / USD

Both sides are normalized first: Unicode NFKC, lowercase, no spaces, no thousands separators. So "1,200 mm" and "1200mm" are the same token.

Matching has two steps:

  1. The passage that section cites
  2. If it's not there, every file you uploaded

What happens on a mismatch

The page goes to "needs review" and the publish API returns 409. The editor shows the token: Not found in your documents: 15 kW. You fix the text or press "Publish anyway". Nothing passes silently.

After an AI edit, the check runs again on its own. After a manual edit, the page goes back to "needs review" until you run the check again or press "Publish anyway". After a full generation, if the first review flags sections, the model gets one try to fix those sections, then they're reviewed again.

What the code doesn't check

  • Dates and years
  • Numbers without a unit
  • Short units like g, m, W, V and lb

For these, a second model pass reads each section next to its sources. It catches a lot, but it's still a model.

Bugs that shaped the regex

  • The source said "120 C", the page said "120 Β°C". The gate called it a mismatch. Now the degree sign is dropped before matching, and a bare C after a number counts as Celsius.
  • The source said "19-inch", the Korean copy said "19인치". In the source files, a hyphen between a number and a unit is now read as a space, and 인치 maps to inch.
  • Test certificates put the unit in the column header: Mass Fraction (%), with 17.803 Β± 0.099 in the cell. Our regex looks for a unit after the number, so it matched nothing, and the composition values were dropped from the page. We added a separate header pattern instead of loosening the main regex, because the gate shares that regex.

Scanned PDFs

Scanned pages and text inside images are read page by page. If a page can't be read, the file list says Read X of Y pages, and that page isn't used. We don't guess.

Retrieval

There's no separate vector database. Search runs inside our SQL database: vector search plus full-text keyword search, merged with RRF (k=60). Each section keeps the passages it cites, and the gate checks those first.

Known limits

  • You need an account to build a page.
  • Tables in PDFs can come out with rows misaligned.
  • The model pass on dates and plain numbers can miss things.

Question

If you've built something that checks LLM output against source documents, where did it break? For us it's documents without a clear structure.

A published example: https://labchure.site/itemwright-digital-sat

I used AI (Claude) to help write this post.

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.