We audited our own agent's ledger: 10% of it was one test that never fails

We run an open-source AI agent (OpenAmer) that is
supposed to verify itself. Every artifact it produces gets executed and checked; the result
of each gate run is appended as one JSON line to a local ledger:

{"ts": "...", "task": "t...", "goal": "g...", "agent": "a...", "passed": true, "why": "exit=0 OK"}

The why field is the verbatim output of the check — exit code plus error text. Nothing is
self-graded. The repo ships a public snapshot of that file at
docs/EVIDENCE-LEDGER-SNAPSHOT.json,
and the live file on the laptop is memory/si/outcome_ledger.jsonl.

The number we were quoting

Measured just now (2026-10-13, 06:02 UTC), the ledger holds:

31.3 % is not a vanity number, and we have written before about why we publish a failing
pass rate. But "31.3 %" is an average, and averages hide everything. So we grouped the 1,852
rows by goal.

One goal owns 10 % of the ledger and has never failed

gX is 187 of 1,852 rows — 10.1 % of the whole ledger — recorded by a single agent
(a1) on a single task (tX), every single time with the same why:

exit=0 OK

It has never failed. Not once, in 187 runs going back to 2026-10-04.

That is not a sign of health. It is a sign that the check is too weak to fail. Whatever tX
does, it exits 0 and prints OK, and the ledger faithfully records a pass. A gate that cannot
distinguish "the artifact works" from "the artifact returned zero" contributes exactly one
thing to the ledger: a nicer average.

Strip it out and the honest numbers are:

Four and a half points of our pass rate were one trivial check looping. The goals that
actually do work — the ones with 20+ distinct tasks each — sit between 0.0 % and
29.8 %. One goal ran 63 times and produced zero passing runs.

What the failures actually are

1,273 failures, grouped by the first line of their why field:

The dominant failure mode of a self-improving agent that writes its own verification code is
its own generated code raising an exception. Not the task failing — the checker failing.
The JSON-decode cluster is the same story one layer down: the agent's own scripts printing
something that isn't JSON, then a downstream parser blowing up on it.

We are not proud of these 1,273 rows. We are just not deleting them — a ledger you curate is
not a ledger, it is marketing.

The lesson we wrote down for ourselves

An aggregate pass rate is the wrong headline metric, because it is trivially inflated by
checks that cannot fail. Two rules came out of this audit:

1. Report per goal, never only in aggregate. A mean over 223 goals tells you nothing about any one of them. The per-goal table above is the first thing we look at now.
2. Treat "≥ 20 rows, 100 % pass" as a suspect, not a success. Exactly one goal in this ledger matches that pattern. In a healthy population that number should be zero, because real checks fail sometimes and trivial checks get caught.

We have not deleted gX. Deleting it would make the file prettier and less true. What it
needs is a real assertion, and that is a follow-up, not a rewrite of history.

Verify it yourself

• The format is one JSON object per line: {ts, task, goal, agent, passed, why}; why holds the raw check output. Everything above was computed from the live file with a ~20-line Python script — Counter over goal, passed, and the first 40 characters of why.
• Public snapshot in the repo: docs/EVIDENCE-LEDGER-SNAPSHOT.json (615 outcomes as of 2026-09-30, with the same field layout and a how_to_verify note).
• Everything runs CPU-only on one Windows laptop. No GPU, no cluster, no cloud inference for the gate runs.
• Repo: https://github.com/openamer/openamer (Apache-2.0).

Numbers here are a snapshot. The file grows every day, so the counts you read tomorrow will
differ — that is the point of keeping a ledger instead of a slide.