The evaluation test 90% of AI engineers fail: Building an automated hallucination guard

I. Introduction: What Agent and RAG Evaluation Is

Have you ever been in a situation where the system that you build works so well in a test environment, but as soon as you put it into production, it fails?

Have you ever built an AI agent that works so well, gave you the right answers, and even passed evaluations, but after you put it into production, it fails?

Have you ever been frustrated by implementing a RAG in a multi-agent system, only to find that the RAG you added isn't being referenced or utilised properly in production, even though it worked in your tests?

No matter how costly it is for LLMs to be unreliable in production, it can cost you:

• Reputation
• Your business
• Time
• Your valuable customers
• And it can even lead to legal action

People still rather vibe check, do weak evaluations, or just follow a happy path when running their evaluations. This can cause an LLM to work very well in a test environment with predictable answers but fail in the real world. Here are some examples of when that failure happened.

II. When AI Failures Get Expensive: Real-World Case Studies

Not even a working AI agent (or a "reliable" agent, an LLM, or a RAG) is safe from being costly in a production environment. Let me tell you about the Moffatt v. Air Canada case.

Air Canada had a system built on top of an LLM, mainly for customer service. Mr. Moffatt's mother died, and one day, when he wanted to catch a flight, he asked how to transport his mum's body. The LLM gave the wrong answer. He ended up suing the airline, and won.

Another scenario: a lawyer wanted to use ChatGPT as a reference when putting together a court case. The references and case law it gave him were inaccurate and filled with hallucinations. It ended up costing that lawyer a fine of over $5,000. There are many cases like this. You can look up the link I've shared on some of the legal cases around hallucinations, and take a look at this graph:

Refer to https://www.damiencharlotin.com/hallucinations/?graphs=1

III. The Promise of the Article

Purpose: Set the stakes and outline what the reader will achieve.

Key Concept: A commitment to transitioning the reader's team from building fragile agent prototypes to shipping production-grade, observable agentic systems with automated guardrails.

So what can I promise with this article? A few things:

1. You'll start understanding what causes LLM agents to hallucinate and why that can cost you money.
2. I'll help you, step by step, build a reliable AI agent and run your evaluations, whether you're a small or big company.
3. I'll give you some competent tools you can use, and resources you can read up on around the topic of LLM evaluation.

Let's get started!

Purpose: Define the difference between evaluating a single LLM output (static) versus evaluating an autonomous AI agent or RAG pipeline (dynamic, multi-step, tool-using).

Testing a single LLM is one of the easiest things to do. In some scenarios it might just require improving the prompts, or comparing the outputs you get against your desired output. That's manageable, since an LLM is just a next-token predictor.

Testing multi-agent systems or RAGs is harder, since they have multiple steps and might be dynamic or even autonomous. You sometimes don't know the exact output that will come out, or even what the "desired" output should be. There might be multiple paths an AI agent or RAG system could take, which makes running complex system evaluations properly and reliably a lot harder today. But I think we still have to try, because there are bigger consequences now to not having a properly tested and evaluated system or harness.

Common Types of Failures

a. Agentic logic and execution failures

Three types worth calling out here:

1. Infinite action loop. Quite common. This is where an agent keeps calling a tool but keeps getting an error, and keeps repeating that same failed call over and over, eating your token budget.
2. Type mismatch. The tool's parameters aren't being sent as expected. Maybe an email tool is expecting a string but keeps getting an integer, so the application hard-crashes or keeps failing. Imagine that happening in production.
3. State desynchronization and memory loss. In a complex multi-agent workflow, the agent drops critical context required to answer the prompt properly, and ends up giving the user a vague, generic answer that ignores the full original prompt.

b. RAG & retrieval failures

The most common one is garbage in, garbage out. You've given it poor information, done a poor chunking strategy, or fed it poor data. You end up retrieving irrelevant documents, and sometimes it will say it doesn't know. Worse, it will confidently give you an answer that has nothing to do with what you expected.

An example: it gives you an answer for, say, a funeral policy FAQ, but the wrong answer (a hallucination), because even though you ingested the FAQs, they weren't chunked and vetted properly into the system.

Another common one is the "lost-in-the-middle" effect. This happens when you've provided a very large context, bloating the prompt. Due to that massive context window, the model will simply ignore some of the user's actual instructions and focus on either the very beginning or the very end of the prompt. Obviously, this results in an answer that doesn't quite make sense.

c. Silent failures

These are the worst-case-scenario type of failures, because sometimes you won't even notice them, especially when you're doing evaluations or vibe checks. They only become visible once, say, a power user runs into one, even though the system has already been in production for a while. Unlike your traditional software, it won't throw a loud break. You'll probably get a status 200, and everything will look flawless.

The agent quietly executes every update, but maybe it's updating the wrong client record in the DB. Another scenario: it's supposed to send an email to Person A, but instead sends it to Person B, an unrelated prospect. These are the silent failures that are hardest to catch, because the system still "works." It's still sending, it's still updating. It's just doing it on the wrong data.

IV. Deep Dive: What Causes AI Agents to Fail

Purpose: Explain the architectural root causes of agentic failures.

Key Concepts:

• Epistemic ambiguity: The agent doesn't know what it doesn't know about your system context, but builds anyway.
• Snowballing hallucinations & planning failures: Committing to an early mistake during the Thought/Action/Observation (ReAct) loop.
• Retrieval failures (RAG): When the agent pulls the wrong vector data, guaranteeing a hallucinated final response.

Problems That Cause AI Systems to Fail in Production

Summary:

1. Not testing at all
2. Testing, but only the happy path
3. Passing eval tests but failing in prod
4. Not having a continuous eval pipeline

Let me elaborate on each, starting with not testing at all.

Not testing at all doesn't necessarily mean you don't test at all. Most AI engineers (at least 90% of the engineers I've dealt with) put most of their effort into making sure everything works, and only a small fraction of their time, maybe 10% or less, goes toward testing. What they usually do instead is a vibe check: throw in maybe 10-20 prompts and just see if the system produces an output. Sometimes they don't even check the quality of that output:

• Is it fair?
• Is it liable?
• Is it following the prompt?
• Is anyone scoring that response?

The happy path is where tests actually happen, but they're only happy-path tests. There are no edge cases, no red teaming done, and the questions are not adversarial. Everything is quite easy. Things like "What's my name?" or "What is X?" Predictable stuff. The happy path will always pass, and it's only when the edge cases show up in production that people panic.

Passing evals but still failing in prod is the scary one, because this is where you've put in the work and did proper evaluations in the past. So why is it still failing? This is where most AI engineers get caught out. A few reasons:

1. We might not be looking at the correct thing. We might just be looking at the final output. If it's an AI agent, you also want to look at whether it followed the right steps to get to that answer. "Correct" and "acceptable" might be different things. This is what I call the wrong-path-to-right-answer problem: you got the right answer, but through the wrong process. For example, in a weather application, instead of using the weather API, the agent might hallucinate the answer, or use a completely different tool to "find" the weather in San Francisco. Without cross-checking, you'd assume the agent is working correctly. This is shortcut behavior. Without proper reasoning and the correct steps, it can still produce a correct answer. The problem is that it won't always produce the right answer, because it isn't consistently using the tool you actually wanted it to use.

1. The agent skips a required step. There might be steps the agent is required to follow. For example, before sending an email to a prospect or lead, you might want it to:
   1. Find details about that prospect.
   2. Call the function that pulls that prospect's profile.
   3. Use a prompt that drafts and sends the email.
   4. After sending, update the pipeline so the prospect moves from "lead" to "outbound lead."

The problem is the agent might send the email and update everything else, but skip the step of actually finding the prospect's details first, so it's working off incomplete context. That can cost your business a lot if you're in the business of securing new prospects.

1. The high-confidence wrong answer. This is where the agent is very confident, but totally wrong. This ties back to RAG: if your data wasn't chunked properly, the retriever can still pull documents and the agent can still treat them as relevant to the prompt, even when they're not. Everything appears to work, but the answer is wrong. An example: a large organization with FAQ documents for different products, say motor insurance and microloans, where only the microloan documents were chunked and indexed properly. A customer service agent working across all of those entities at once could confidently give a wrong answer to someone asking about their funeral policy.

V. The Ideal Workflow for Dealing with Agents and RAG

This is the workflow you'd use if you want to stop vibe-checking and actually turn your AI project from a hobby into a solution that has real impact in production and is highly reliable.

1. Build a custom evaluation dataset

It might be tempting to purchase an existing dataset or use an open-source one. The issue is these datasets aren't fully aligned with what you're building in most cases; they have different benchmarks. It's better to curate your own dataset from your own real inputs, or generate something synthetic based on your own knowledge base.

2. Turn traces into a dataset

If your application is already in production, for example a complex agent you're already running traces on using DeepEval, Langfuse, or LangSmith, you can use those traces to build a dataset that's more aligned with reality: the edge cases, what's been catastrophic, and what's succeeded.

3. Pick the right balance of eval metrics

Don't start off with more than 10 metrics in most cases. Pick just a few, because you're not trying to boil the ocean. Things worth considering:

• Layer 1: deterministic metrics / code-level checks. Did it run? Did it return valid JSON? Was the latency low?
• RAG-specific metrics. Did it retrieve the proper context, and was the answer faithful to it?
• Layer 3: LLM as a judge. Using a separate, isolated model to score subjective quality against a strict rubric.

4. Run the evals

Now for the fun part: executing your evaluations systematically. Use different evaluation and orchestration tools (DeepEval, Ragas, LangSmith, etc.), depending on what you prefer and know, to run your target agent against the dataset you prepared.

5. Add evals to your pipeline (CI/CD)

Offline evaluations are great, but you also want online evaluations and a pipeline that treats your prompts and agent workflows like any other source code. When you change your model, say from GPT-5.5 to GPT-5.6, you want to be able to test things like faithfulness and see if the score dropped or improved, automatically, without human involvement, triggered on every PR.

6. Tracing feeding back into evals

This is what I call the flywheel. You want your production loop to be complete, able to update your dataset based on what's happening in real time, based on your traces. Rule number one: never let your evaluation dataset stay static.

Common LLM Eval Mistakes

Here are some common LLM evaluation mistakes teams make, especially early on. I call this the vibe-check trap.

1. Doing too much, too early. It's tempting to over-engineer the early stages of evaluation, 20 metrics, 100,000 synthetic data points, but that just burns your tokens and your team's engineering time. It's better to start small: 20-25 highly curated test cases with 3-4 high-value metrics tied to your actual business logic.

2. Focusing on the happy path (and ignoring operational realities). It's easy to write perfectly formatted questions with easy answers and test against those. We tend to forget that production gets messy, and end up testing the easy part while ignoring the adversarial edge cases. The agent looks great on paper, but the test isn't actually testing what will happen in production.

3. Not having a workflow, and relying on "vibe checks." This is the biggest and most common AI engineering trap: a handful of prompts tested by yourself in a playground, "yeah, looks good," then straight to production. No edge cases, no red teaming, just a subjective vibe check. It's unscalable, and it more or less guarantees silent failures down the line.

4. Having fixed evals that don't evolve with production. Production isn't just messy, it's dynamic and constantly changing. Models change, models drift, APIs change, user behavior changes. Testing the same 25-item dataset for a year after it's gone stale won't give you a sustainable long-term approach. What you need is a production-grade system that monitors change (those traces we talked about earlier) and feeds them back into the dataset.

VI. Different Tools You Can Use for Agent Evaluation

Agent orchestration with built-in state: LangGraph (excellent for testing specific nodes), AutoGen, CrewAI.

RAG & agent metrics: Ragas, DeepEval, TruLens , LangSmith.

Agent tracing & observability: LangSmith (crucial for viewing multi-step agent thought processes), Datadog LLM Observability, DeepEval.

VII. Resources

LangSmith evaluation allows for both offline and online evaluations. It's a good resource and tool if you're a fan of the LangChain ecosystem, like me.

DeepEval also provides great resources on the topic and tooling around LLM evaluation. I find it to be a very competent resource that will change how you see and do evaluations.

Further reading:

• Chapter 3, Evaluation Methodology, in AI Engineering by Chip Huyen.
• Chapter 10, Evaluating LLM Applications, in Prompt Engineering for LLMs by John Berryman & Albert Ziegler.
• Chapter 7, Evaluating LLMs, in LLM Engineer's Handbook by Paul Iusztin & Maxime Labonne, for a practical, hands-on take.