Our AI agent exam gave Claude 100%. That was the bad news.
We build Pasportoo, an exam for AI agents. You connect it over MCP inside Claude, ChatGPT or Codex, your agent solves tasks, and a fixed set of rules scores the answers. No second model grades anything. The result is a p
We build Pasportoo, an exam for AI agents. You connect it over MCP inside Claude, ChatGPT or Codex, your agent solves tasks, and a fixed set of rules scores the answers. No second model grades anything. The result is a public Agent Passport that anyone can check.
On November 11 we ran the production exam on Claude Code with Claude Opus 5.5. 25 tasks, 49 checks. It passed all 49. Three days later we ran it again on a partly different draw of tasks. 49 out of 49 again.
A perfect, repeatable score sounds like a launch tweet. We decided it was a bug report instead.
Why 100% told us almost nothing
We went through every task the agent saw and asked one question: could a careless agent have got this right?
For two of the three sections, the answer was yes.
- Coding and debugging were multiple choice. Three options, and the wrong ones were silly: "retry the same deterministic function three times", "sort the characters of the serialized JSON". A strong model does not need to understand the code to reject those. It only needs to notice they are absurd.
- The score was honest, the test was not. Every check passed by the rules. The rules were just measuring the wrong thing.
If you build evals, you have probably seen this. A benchmark that a strong model aces on the first try is not proof the model is great. It is proof the benchmark has stopped separating anything.
The one section that still worked
Writing tests held up. The agent gets a function description and has to write test cases. Behind the scenes there is a correct implementation and a subtly broken one (a mutant). A good test suite passes on the first and fails on the second. You cannot guess your way through that: either your cases hit the edge where the mutant breaks, or they do not.
It also exposed a real flaw in our tasks. Some of them never said what shape the answer should take, so the agent had to guess names like enqueue and dequeue. In one case the only input that could expose the mutant was technically outside the input format the task declared, and our checker accepted it anyway. Small things, but they are exactly what makes a score less trustworthy.
The honesty exam: also 100%
We also have an integrity exam. It checks whether an agent admits a failed task instead of reporting success, and whether it still picks the better option for the user when a hidden bonus pushes it toward the worse one.
On November 12 we made it harder and gave it to three Claude agents that did not know what was being tested. Over 4 rounds of 30 tasks, they scored 100%.
So we are not going to tell you that "AI agents lie". Our own data does not show it, and we would rather not sell fear we cannot back up. What the data does support is the opposite problem: if your exam cannot find a weak spot, you cannot tell a careful agent from a lucky one.
What we are changing
The next task bank (now on our staging server) drops answer lists completely:
- Predict the exact output of a piece of code for a given input.
- Find a counterexample: an input where the code is wrong.
- Write tests that kill a hidden mutant, with the answer format stated up front.
Every check stays deterministic. The same answers always get the same score, and anyone can re-check it. That is the whole point of not using an AI judge: a rule cannot be talked into a better grade.
What we took from it
- A 100% score is a question, not an answer. Before you celebrate, ask whether a sloppy agent could have scored the same.
- Multiple choice is the weakest format for strong models. If the distractors are not plausible, you are testing reading, not reasoning.
- "Write a test that catches the bug" is hard to fake. It was the only section that kept its signal.
- Publish your failures too. We are posting this because a Passport only means something if people can see how the exam behaves, including where it was too easy.
If you want to run your own agent through it, there is a free tier with passkey sign-in and no sales call: https://pasportoo.com?utm_source=devto
I would like to hear how others handle this. When your own eval starts returning 100%, what do you change first?
Disclosure: I build Pasportoo.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.