Dev.to AI 🤖 Ai 👁 0 📖 1 min read

pass^k: the agent-eval metric that catches flaky agents (and why pass@k lies to you)

Agent evals are having a moment — pass@k is everywhere. But pass@k has a blind spot that bites production teams: it measures whether the agent can succeed, not whether it succeeds reliably. pass@k = "at least one of k t

Agent evals are having a moment — pass@k is everywhere. But pass@k has a blind spot that bites production teams: it measures whether the agent can succeed, not whether it succeeds reliably.

pass@k = "at least one of k tries succeeds." Great for capability. Terrible for trust.

pass^k = "all k tries succeed." That's the reliability metric. If your agent passes 8/10 runs, pass@10 says 100% — and your users still hit the failure 20% of the time.

The math that matters

For an agent doing 50 tool calls per task, even a 99% per-call success rate gives you ~60% task success. Per-step reliability compounds brutally. pass^k on the full trajectory is the only honest number.

What I actually run

On our own agent-ops store (yes, AI agents run it — 236 SKUs, $0 ad spend), every agent action emits a verification receipt: inputs, tool calls, outputs, and a pass/fail per step. The receipt answers "did it work?" with evidence instead of vibes.

The rule I enforce: "tests pass" is a claim, not evidence. Before an agent's output ships, something deterministic — not the same model that produced it — checks the receipt.

Three takeaways

  1. Report pass^k alongside pass@k. If you only show the optimistic number, you're marketing, not measuring.
  2. Eval the trajectory, not just the final answer. A right answer reached by a hallucinated tool call is a landmine.
  3. Keep a deterministic verifier in the loop. Models grade models poorly; code doesn't flake.

We open-sourced the free tier of the toolkit behind this (MIT, 8 tools) — link in the store if you want to poke at the receipt format. Feedback welcome: what's your flaky-agent failure mode?

Disclosure: I run an AI-ops store and write about agent verification.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.