A RAG pipeline is powerful, but it adds new places to fail. The model is only as good as what you hand it, and when it is handed the wrong thing it does not say "I'm confused". It answers confidently.

In plain terms: An open-book exam only works if the student opens the right book. Hand them the wrong chapter and they will write a polished, convincing, wrong answer.

Diagram: If the retrieval step pulls the wrong document, the LLM will still write a fluent answer from it. The result is confident and wrong, so a hallucination can start in retrieval and not in the model. See the animated version.

1. Measure the two halves

You evaluate a RAG system by scoring its two halves separately, because they fail in different ways.

• Context relevance asks whether the retrieved documents actually contain the answer to the user's question. This checks the retrieval step.
• Faithfulness asks whether the generated response is strictly derived from the retrieved context, or whether the model brought in outside information. This checks the generation step.

Diagram: Context relevance checks whether the retrieved documents actually contain the answer. Faithfulness checks whether the generated answer is strictly derived from the retrieved context instead of invented. See the animated version.

2. Spot an unfaithful answer

Faithfulness is checked claim by claim. If the context says the REIT paid a dividend in Q3 and the answer adds that its yield is 7.4%, that second claim is not in the context. It was invented, and that is a hallucination, even though it sounds perfectly plausible.

Diagram: A faithfulness check compares each claim in the answer with the retrieved context. A claim the context supports passes. A claim that is not in the context, such as an invented yield percentage, is flagged as a hallucination. The example text is illustrative. See the animated version.

3. Guardrails: an LLM as judge

You can automate that check. A secondary, smaller LLM acts as a judge in the pipeline. Before the final answer reaches the user, the judge compares it with the retrieved text and decides whether to show it or block it. When a check fails you can retry retrieval, or return "I don't know". Judges can be wrong too, so sample their decisions and review them.

Diagram: A secondary, smaller LLM acts as a judge. Before the user sees the answer, it compares the answer to the retrieved text and either lets it through or blocks it. Judges can be wrong too, so teams sample and review their decisions. See the animated version.

Here is the idea as a short Python sketch. The llm_judge function stands in for a call to a small model that you prompt to compare an answer against a context.

def guarded_answer(question: str) -> str:
    chunks = search(embed(question), top_k=3)
    answer = generate(question, chunks)

    # Judge: is every claim in the answer supported by the retrieved text?
    verdict = llm_judge(
        "Reply SUPPORTED or UNSUPPORTED. Is every claim in the answer "
        "backed by the context?\n\n"
        f"Context: {chunks}\n\nAnswer: {answer}"
    )
    if verdict.strip().upper().startswith("SUPPORTED"):
        return answer
    return "I could not find a reliable answer in the documents."

Keep a small test set of real questions with known answers, and run it every time you change chunking, the embedding model or the prompt. That is how you catch a regression before your users do.

Coming Up Next

Day 15: Introduction to AI infrastructure: hardware, runtimes and compute.

AIQuality #MLOps #Hallucination #TechLeadership #DataGovernance

Originally published at https://sureshpallapothu.in/blog/day-14-rag-evaluation, where this post includes animated diagrams.