Dev.to AI 🤖 Ai 👁 0 📖 3 min read

Cite only what you need: building an offline document RAG on AMD ROCm

Part of my work in the AMD AI League by lablab.ai and AMD. #AMD #lablab Most RAG demos answer a question and paste a list of "sources" underneath. That list is usually padding: the top five chunks, whether they helped o

Part of my work in the AMD AI League by lablab.ai and AMD. #AMD #lablab

Most RAG demos answer a question and paste a list of "sources" underneath. That list is usually padding: the top five chunks, whether they helped or not. For the AMD AI League's RAG challenge the grading was stricter. You return the value and exactly the files that value needs: no more, no less. On top of that the corpus is messy company data and the whole system has to run offline on an AMD GPU.

These are the design decisions that mattered.

The corpus is hostile

The input folder mixed PDFs (some scanned, some encrypted), DOCX, PPTX, XLSX, CSV, logs, source code and screenshots. Two rules kept the indexer alive:

  • Every file gets its own try. One corrupt PDF must never cost you the rest of the corpus. Unreadable, encrypted and unknown files are logged and skipped.
  • An encrypted file is never a source. If you weren't given the password, you didn't read it, so you can't cite it.

Office files are just zipped XML, so DOCX and PPTX are read straight from their XML with the standard library, tables included. Spreadsheets keep row and column labels, so "Q3 revenue" stays next to its number.

One model for text and images

Screenshots and scanned pages carry real answers, such as an error code on a dashboard or a value in a chart. Instead of a separate OCR stack, a single vision-language model (Qwen2.5-VL-7B on ROCm) transcribes each image into label/value pairs at index time. After that, images are searchable text like everything else.

Retrieval that respects identifiers

Company data is full of identifiers like ORR-1847 or THERM_ALERT#. A normal tokenizer splits them and they stop matching. Keeping them whole in BM25 fixed a whole class of misses.

Some questions need two hops: "the log shows an incident; which release fixed it?" The first hop finds the log line. A second hop follows the identifiers found next to the question's words to the files that define them, such as the bug database.

Superseded documents

Real folders contain policy_r1.pdf, policy_r2.pdf and a file titled "WITHDRAWN". The indexer labels older revisions and withdrawn files as SUPERSEDED, and the model sees that label. When two files disagree, the current one wins.

Citations by necessity

The model answers in JSON with the value, the files it needed and the evidence line. Then code checks the result:

  • a file is cited only if it actually contains the answer (or the identifier that leads to it);
  • current documents are preferred over superseded ones;
  • a second file is added only when it supplies a piece the first one lacks.

When nothing in the corpus answers the question, the system returns an empty answer with no citations. A confident wrong citation is the worst possible output.

Running it on ROCm

  • Base image: the official rocm/pytorch. transformers is installed with --no-deps so pip never swaps in a CUDA build of PyTorch.
  • The model loads once into a resident server behind a unix socket. Each query process only pays for retrieval plus one generation.
  • Indexing and answering run without network access.

Code

Open source (MIT): https://github.com/automatizacionesmorente-png/amd-ai-league-mini-challenges (the rag/ folder).

Thanks to AMD and lablab.ai for the GPU credits and a challenge that rewards honesty over volume. Questions and critiques are welcome in the comments.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.