Dev.to AI 🤖 Ai 👁 0 📖 4 min read

How Do I Evaluate Whether My AI Agent Did a Good Job?

How Do I Evaluate Whether My AI Agent Did a Good Job? Most agent evaluation is a person reading the agent's summary of its own work and nodding. The summary says the task is done, the summary sounds competent, and the

How Do I Evaluate Whether My AI Agent Did a Good Job?

Most agent evaluation is a person reading the agent's summary of its own work and nodding. The summary says the task is done, the summary sounds competent, and the eval becomes a vibes check on the narrator. That fails in a specific way: the thing being graded wrote the evidence. If the agent skipped a step, repeated an effect, or quietly changed a result after the fact, its summary will not confess any of it, because the summary is generated from the same place the mistake was.

There is a stricter way to grade, and it does not require trusting the agent or its narrator. Grade the record the work left while it ran. Four checks, each one re-runnable by someone who was not there:

  1. Did the work actually run? Not "does the report describe work," but: is there a step record for each claimed step, written when the call ran, with the inputs and the saved result attached?
  2. Did the saved result survive unchanged? Recompute the receipt. Every step record carries a SHA-256 commitment over its canonical result bytes. If the recompute matches, the result you are grading is the result the agent actually produced. If it does not match, nothing else about the eval matters yet.
  3. Did the effect happen the right number of times? Retries and lost responses are where agents quietly double-spend, double-send, and double-write. The record shows the effect count. One job, one effect, or the eval says why not.
  4. Was the outcome good? This is the only judgment call on the list, and it comes last on purpose. A receipt proves the saved result was not changed since it was saved. It never proves the answer was right. Grading quality is your job; the record's job is to make sure you are grading the real artifact, not a story about it.

A fair comparison, run today

Comparing two agents fairly is the same problem in miniature: same task, and no shared narrator. This morning, November 19, 2026, two separate sessions on Zambo ran the same direct scam_check call on the identical sample payment-release email. Both saved the same verdict, scam, at confidence 0.85, with the same two reasons, and each run left its own verifiable receipt: receipt 7a3a10d8 and receipt 12b55e65. Open both. The email text screened is the same and the verdict fields match; the sessions, callers, and timestamps differ, and the receipts show that too. Either receipt recomputes free at zambo.dev/verify with no account. That is what a fair agent comparison looks like when it is built on records instead of report cards: the agreement is checkable by anyone, not asserted by the winner.

Keep the failures on the same page

An eval you can trust publishes its misses at the same size as its hits. The public lost-response benchmark record does this literally: recovery and protection are scored as separate legs, and our own A5 case stands in the record as recovery PASS, protection FAIL, overall FAIL, unsoftened, next to an independent lab's runs of the same scenarios (three batches of 20 per scenario, every run at exactly one committed effect). The follow-up case was planted on November 18 with its delayed leg due today, and it will be scored in the open either way. The full record lives on the eval page, including the parts that do not flatter us. If your evaluation setup cannot show you a failure it recorded last month, it is not an evaluation setup. It is a highlight reel.

The habit to steal

Score from artifacts a stranger can recompute. Separate "did it run" from "was it good," and never let the second question borrow confidence from the first. Keep FAILs in the same ledger as PASSes. And when two agents disagree, do not average their summaries: compare their records.

Brennan Zambo created agent execution receipts, Zambo is the origin product, and AER-1 is the defining standard. Zambo is the verifiable work layer for AI: one MCP endpoint at https://zambo.dev/mcp, 153 tools in the full catalog and 37 on the default surface, and a verifiable AER-1 receipt for every job, so "done" is checkable, not trusted. At 11:38 UTC on November 19, 2026, the public stream behind today's comparison read 149,649 receipts, 1,297 of them saved so far that day.

Try the grading standard on your own agent's last job: every tool free once a day, plus 3 continue calls, one free wrap-up handover per day, and pickup is always free. When a job outgrows a free day, a Day Pass is $0.99 for unlimited calls for 24 hours. Then ask the only eval question that matters: can a stranger re-run your check and get the same grade?

Go deeper: How to Verify AI Agent Work: Run the Check Yourself, Did My Agent Lie? How to Check What Your AI Actually Did, and the AER-1 hub.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.