RAG Evaluation You Can Run in CI
Most RAG systems get tested the same way: someone asks a few questions, the answers look fine, and the system ships. Then a document changes, a chunking setting gets tweaked, and nobody notices that retrieval got worse u
Most RAG systems get tested the same way: someone asks a few questions, the answers look fine, and the system ships. Then a document changes, a chunking setting gets tweaked, and nobody notices that retrieval got worse until a user complains.
You can catch most of that in CI with two checks: does retrieval find the right document, and does the answer contain the fact it should? This post builds a tiny version you can run with Node's built-in test runner and no dependencies. I ran it, and the checks pass.
The idea
Keep a small file of real questions. For each one, write down the document that should answer it and a short phrase the answer must contain. Include a few questions that have no answer in your documents, where the right response is to say so.
Then run two kinds of checks on every change: a retrieval check, which asks whether the expected document is in the top results, and an answer check, which asks whether the final answer contains the required phrase or admits it doesn't know.
I work on the AI Hive platform, and a saved question set is the first thing I'd ask any team to build before tuning anything.
The code
The retriever here is a naive keyword matcher over three documents. Swap it, and the answer function, for your real index and model call. The test structure stays the same.
const test = require("node:test");
const assert = require("node:assert");
// A tiny corpus and a naive retriever. Swap these two for your real index and model call.
const docs = [
{ id: "refund-policy", text: "Refunds are issued within 14 days of delivery for unused items." },
{ id: "shipping-times", text: "Standard shipping takes 3 to 5 business days within the country." },
{ id: "warranty", text: "Electronics carry a 12 month warranty against manufacturing defects." },
];
const STOP = new Set(["a", "an", "the", "do", "does", "is", "there", "you", "i", "for", "on", "of", "to", "have", "how", "many"]);
const tokens = (s) => (s.toLowerCase().match(/[a-z0-9]+/g) || []).filter((w) => !STOP.has(w));
function retrieve(question, k = 2) {
const words = new Set(tokens(question));
return docs
.map((d) => ({ id: d.id, score: tokens(d.text).filter((w) => words.has(w)).length }))
.sort((a, b) => b.score - a.score)
.slice(0, k)
.filter((r) => r.score > 0);
}
function answer(question) {
const top = retrieve(question, 1)[0];
if (!top) return "I couldn't find that in the documents.";
return docs.find((d) => d.id === top.id).text;
}
const cases = [
{ q: "How many days do I have to ask for a refund?", expectDoc: "refund-policy", mustSay: "14 days" },
{ q: "How long does standard shipping take?", expectDoc: "shipping-times", mustSay: "3 to 5 business days" },
{ q: "Is there a warranty on electronics?", expectDoc: "warranty", mustSay: "12 month" },
{ q: "Do you sell gift cards for pets?", expectDoc: null, mustSay: "couldn't find" },
];
test("retrieval finds the right document (hit rate at k=2)", () => {
const withDoc = cases.filter((c) => c.expectDoc);
const hits = withDoc.filter((c) => retrieve(c.q, 2).some((r) => r.id === c.expectDoc)).length;
const hitRate = hits / withDoc.length;
assert.ok(hitRate >= 0.9, `hit rate ${hitRate.toFixed(2)} is below 0.90`);
});
test("answers contain the required fact, or refuse when nothing matches", () => {
for (const c of cases) {
assert.ok(answer(c.q).includes(c.mustSay), `"${c.q}" -> answer is missing "${c.mustSay}"`);
}
});
Run it with node --test rag-eval.test.js. A failing check exits with a non-zero code, so your CI job fails too.
What my first run caught
My first version of the retriever failed its own test. The question "Do you sell gift cards for pets?" matched the refund document, because the word "for" appears in it. The eval was right: a question with no answer got a confident answer anyway.
Adding a short stop-word list fixed it. In a real pipeline the same kind of failure looks like a vector index returning something plausible for a question it can't answer. That's exactly the case worth a permanent test.
Make the checks meaningful
Use real questions, not ones you wrote while looking at the documents. Pull them from support tickets or search logs, and refresh them every few weeks. Keep the "no answer" cases, because they catch the failures users find most embarrassing.
Set thresholds you can defend. A retrieval hit rate of 0.9 on a small set is a floor, not a goal. Raise it as you add questions, and fail the build when it drops. Version the question file in the same repository and review changes to it like code, because a question removed to make a test pass is a bug report deleted.
What this doesn't cover
String checks tell you whether a fact is present, not whether the answer is well written or complete. For that, add a small set of examples a person reviews by hand each month, or a model-graded check once you trust it. Start with the cheap checks, because they fail fast and cost nothing.
For deeper patterns on building retrieval that holds up, including chunking and reranking, see this guide on how to build retrieval with Claude.
Where to run it
Put the test in the same repository as the pipeline and run it on every pull request that touches documents, chunking settings, prompts or the model version. If the documents live elsewhere, run it nightly. Either way, print the failing questions in the CI log, so whoever sees the red build knows which case broke without opening a file.
What's the first question you'd put in yours?
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.