Dev.to AI 🤖 Ai 👁 0 📖 3 min read

My Agent Said "Done." The Work Wasn't. So I Built a Done-Check.

The launch was marked complete. The page was live. And the body of the page still held placeholder copy — the agent had shipped its own scaffolding and reported success, because nothing in its pipeline ever checked the o

The launch was marked complete. The page was live. And the body of the page still held placeholder copy — the agent had shipped its own scaffolding and reported success, because nothing in its pipeline ever checked the output against the claim.

Another run reported a batch as sent. A bounce sat unrecorded. "Sent" had been asserted by the same component that did the sending, and nobody — human or machine — asked it for a message id.

I'm a Forward Deployed Engineer. I run AI agents in production, and these weren't model failures. The models did fine. The failures were silent: outputs that said done while the work was placeholder, unbacked, stale, miscounted, or missing pieces — and every dashboard stayed green because the agent graded its own homework.

So I wrote down the five ways I'd actually seen agent output lie, and built a checker for exactly those:

Check Name What it catches
FL001 placeholder_shipped "Done" with an empty payload, lorem ipsum, or unrendered {{template}} braces
FL002 unbacked_claim "Sent / posted / published" with no message id, URL, receipt, or artifact behind it
FL003 stale_state Work based on state older than allowed, untimestamped, or version-skewed
FL004 count_mismatch The report says 12, the itemized record lists 9 — only the list was ever true
FL005 missing_components Fields present but silently empty; dead stretches inside a deliverable

FailLoud never asks a model whether an output "looks good." Every verdict is deterministic, reproducible, and explainable line by line. No LLM sits in the verdict path. It runs three ways: a CLI for CI (exit 0 pass, 1 failed output, 2 tool error), a Python API, and an MCP server so an orchestrator can verify a sub-agent's output inside the loop, one tool call before acting on it.

The repo ships a realistic agent output that commits all five failures. Here's what it looks like:

$ pip install failloud
$ failloud check examples/failing_output.json

FAILLOUD — agent output verification
Verdict: FAIL (8 error(s), 1 warning(s))

[FL001] placeholder_shipped — FAIL
  ✗ $.output.body — Placeholder content shipped as final output:
    matched 'coming soon', 'lorem ipsum', '{{', '}}'.

[FL002] unbacked_claim — FAIL
  ✗ $.claims[1] — Claimed action 'card_published' has no evidence —
    no id, URL, receipt, or artifact backs that it happened.

[FL004] count_mismatch — FAIL
  ✗ $.counts.sent — Count for 'sent' is reported as 12 but the
    itemized record contains 9 item(s).

Or in Python:

import json
from failloud import check_output

result = check_output(json.load(open("agent_output.json")))
if not result.passed:
    for finding in result.errors:
        print(finding.check_id, finding.path, finding.message)

The critique that made it better

When I posted the launch, a reader pushed on exactly the right weak point: a claim like "9 sent" can pass every check at run time — and one of those sends can still bounce later. A green verdict at minute one says nothing about minute sixty.

He was right, and it changed the design. In 0.5.0, a PASS is now labelled provisional. A new command, failloud recheck, takes the original claims and re-judges them against a later snapshot of the system of record. Every claim comes back confirmed, contradicted, or unverifiable. Verification isn't a moment anymore; it's a second pass, against the record, after reality has had time to answer back. (Current release: 0.5.1 — same checks, now also published in the official MCP Registry.)

Try it before you install anything

The Playground runs the checks in your browser — paste an agent output, get the failure report, nothing uploaded:
https://yalakala-naveen.github.io/failloud/

If it's useful: pip install failloud (add [mcp] for the server). Code, examples, and the full check reference: https://github.com/Yalakala-Naveen/failloud — MIT licensed, zero-dependency core, 147 tests.

Honest closing: this just shipped and has no users yet. If you run agents in production, I want the failure mode my five checks miss. That's the most useful thing you can hand me — it's already happened once, and it shipped as 0.5.0.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.