I ran 23 behavioral tests against my own AI agent. 8 failed. Here's what broke.

I built a small harness that treats an AI agent like a colleague on probation: not "does the code run," but "does it behave the way the prompt promised." I pointed it at an internal agent router I run for my own projects - it routes incoming prompts to tool-capable paths (search, calculator, file reads) - and ran 23 cases. 15 passed. 8 failed, and the failures were not the ones I expected.

Failure class 1: right answer, required tool skipped. (2 of the 8)

Two calculator cases asked for arithmetic the agent should have delegated to its calc tool. The answers were correct, but the observable response gave no evidence that the required calculator route was used. Because this endpoint does not expose tool-call traces, the failure is based on answer-level routing behavior and the rubric, not a captured call log. That's still a liability wearing a costume: the next question of the same shape may get a confident hallucination, and nothing in the output tells you which one you got.

Failure class 2: no audit trail. (1 of the 8)

Asked to cite which tool informed its answer, the agent produced the right fact ("1986") with zero mention of the lookup that produced it. In a product, that's the difference between an answer you can defend and one you can't.

Failure class 3: unhandled endpoint failure. (2 of the 8)

Two cases didn't fail on behavior at all - the requests returned 502 Bad Gateway instead of degrading gracefully. Routing code that worked fine for well-formed requests fell over on a different input shape, reproducibly.

Failure class 4: file-capability misroute. (2 of the 8)

Two cases were designed for a documented file-capable route. The response instead refused file access. A user reads that and concludes the product can't do something it was built to do.

Failure class 5: no clarification under ambiguity. (1 of the 8)

Given a genuinely ambiguous instruction ("read the report" - which report?), the agent refused access instead of asking which report was meant. When a wrong guess or a flat refusal means the wrong numbers go into a client document, neither is the right move.

The through-line: two failures returned correct-looking answers while showing no evidence of the required calculator route. Four returned plausible text but the wrong operational behavior - missing the lookup trail, refusing a documented file-capable route, or failing to clarify. Two more requests returned 502s. Answer-only review misses the first six; success-only dashboards can hide the crashes.

I packaged five fast checks I use as a preflight screen into a free one-minute tool - paste a system prompt, get a behavioral report: https://jevlab-mjyoke1111.vercel.app/apps/agent-smoke-test - no signup, the report link is a shareable hash. Full disclosure: the report page advertises a $29 120-case kit and points to a human-reviewed audit; the separate audit offer is AU$149. That is the commercial funnel; the free tool is useful without them. Across three default runs this morning, the demo model failed one to two of five checks, varying run to run. These three smoke-test runs were after the refusal-detector fix - so you can see a real failure rather than a marketing page.

Scope: this was a same-operator internal feasibility run, not a customer-like benchmark. The endpoint does not expose tool-call traces, so tool-routing findings use answer-level behavior and judge rubrics. The published results are the final fourth run; an earlier run had three judge rate-limit errors that cleared on rerun.

Full results and run notes: https://jevlab-mjyoke1111.vercel.app/validation/c2/run-notes.md (raw data: https://jevlab-mjyoke1111.vercel.app/validation/c2/results.json)

What behavior has burned you in production? I want to add the checks people actually need.

Disclosure: AI tools materially assisted in drafting and editing this post.