Before You Trust an AI Agent With Customers, Give It a Test You Can Rerun
An AI agent that answers customer questions can look great on launch day. The harder question comes later: is it still giving good answers after someone changes something? Vendors release new models, a team member edits
An AI agent that answers customer questions can look great on launch day. The harder question comes later: is it still giving good answers after someone changes something? Vendors release new models, a team member edits the agent's instructions, someone adds a fresh batch of documents. Each change can make answers better, or quietly make them worse. Too often, the first sign of a problem is a customer complaint, a wrong refund or a bad review.
Full disclosure: I built Agent Ready Rankings (https://rankings.leffke.com), a site that scores AI agent tools out of 10. The scores below are overall scores from my latest data run, not a same-day refresh. Browsing the site is free; the full score breakdown is part of Pro, $10/month.
Why "it worked last month" isn't enough
Traditional software does the same thing every time you give it the same input. AI agents don't work that way. Their answers depend on the model underneath, the instructions they were given and the documents they can see. Change any of those and the answers can shift, sometimes in ways nobody notices for weeks.
That doesn't mean you can't rely on an agent. It means you need a cheap, repeatable way to check it.
The habit: a test set you rerun
The simplest version looks like this:
- Collect real questions. Pull 50 to 100 questions customers have actually asked, including the awkward ones: refunds, cancellations, pricing, anything with legal or money consequences.
- Write down good answers. Have someone who knows the business mark what a correct, acceptable answer looks like for each one.
- Rerun it after every change. New model, new instructions, new documents: run the whole set again before customers see the change.
- Compare before and after. If scores drop, or a sensitive question gets a worse answer, hold the change until someone looks at it.
Evaluation tools exist to make steps 3 and 4 fast. They run the questions, score the answers and keep a history so you can see whether a change helped or hurt.
The eval and safety tools I track (overall scores)
LangSmith: 7.1 out of 10. An observability and evaluation platform from the LangChain team, built around tracing agent runs and running evaluations on them. Free tier with paid plans. Latest tracked update: its cloud changelog for the week of November 2 to 9, 2026.
Arize Phoenix: 6.45 out of 10. Open-source observability and evaluation for AI apps, including tracing and evals, and it can be self-hosted if you'd rather keep data in-house. Latest tracked release 21.0.0 on October 29, 2026.
Ragas: 5.2 out of 10. An open-source evaluation framework focused on question-answering over documents, with metrics you can use as quality gates. Free. Last tracked release January 13, 2026.
Questions to ask whoever runs your agent
- Do we have a test set of real customer questions, and who owns it?
- When the model, instructions or documents change, do we rerun it before going live?
- Which questions are "must never get wrong," and how are they checked?
- Who sees the results, and who can stop a change that makes answers worse?
None of this requires you to understand how the model works. It's the same discipline as testing a new checkout page before you switch it on.
About the rankings
Agent Ready Rankings scores AI agent tools out of 10 and is free to browse. The eval and safety category is here: https://rankings.leffke.com/categories/eval-safety. I built it, so treat the scores as one informed opinion, and tell me where you think they're off.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.