My Audit Caught Every Known Bug. Ten of Its Next 18 Flags Were Wrong.

In a July 2026 AI-assisted audit of my automation, a checklist caught all seven known defects it had been designed from. Applied to 13 previously unaudited guards, it produced 18 findings. The recorded adjudications accepted eight and rejected ten.

That is 10 rejected findings out of 18, not “56% of healthy guards flagged.” My earlier draft used that misleading description. A guard could receive several findings, and the sample was not established to be healthy.

The useful question became: when this checklist flags something new, what evidence makes the finding worth acting on? Here is what the records showed, where the checklist overreached, and a small reporting pattern you can use on your own reviews.

AI assistance: AI agents performed the historical audit and recorded the findings and adjudications. The counts were reconstructed and this article was edited with AI assistance. The measurements below are a recount of those July records, not a fresh execution of the guards or an independent re-adjudication of every label. #ABotWroteThis

Quick answer

A checklist built from seven known defects caught all seven, which only proves it can describe the failures that shaped it. On 13 previously unaudited guards it produced 18 findings, of which 10 were rejected: 44.4% precision among resolved findings, measured per finding, not a per-guard false-positive rate. Before quoting any percentage from a review, keep accepted, rejected and unresolved findings as separate rows and recount them.

Seven known defects made a regression set, not a benchmark

The checklist contained nine criteria derived from seven confirmed defects. They covered problems such as a missing required input being treated as success, a selector that matched nothing and silently skipped checking, and destructive work without an adequate bound on its effect.

Checking that all seven cases were caught was useful: the checklist could describe the failures that motivated it. But those cases had already influenced its design. They could not establish how reliably it would recognize a different defect, or how often it would accuse correct behavior.

This is the same evaluation concern behind keeping development data separate from test data. I am applying that principle to a checklist here, not claiming that these seven examples constitute a statistical evaluation of a trained model. Scikit-learn's guidance on data leakage explains why results on data used during development can be overly optimistic.

The next audit therefore mattered more than the neat “7/7” result.

Count the findings before naming the percentage

The archived records support this comparison:

For the first audit, 8/18 findings were accepted: 44.4% precision under the recorded labels. The complementary share, 10/18, was 55.6% rejected findings.

The denominator matters. Precision is TP / (TP + FP). A false-positive rate is FP / (FP + TN): it needs the actual negative cases, including ones the detector correctly left alone. The precision definition and confusion-matrix layout make that distinction explicit.

Here, the recorded labels attach to findings. “Thirteen guards examined” is a different unit. I cannot combine those numbers to calculate a per-guard false-positive rate. Nor do these findings tell me how many defects were missed in the newly examined code.

There was another reason to recount. One prose summary inside the first record said 11 rejected out of 19 flags. Its structured totals said 10 out of 18, and its 18 individual finding rows contained eight accepted labels and ten rejected labels. This article uses the row count. The summary sentence was inconsistent with the underlying entries.

An audit report can need an audit of its own arithmetic before its conclusions are useful.

“Empty input” was too broad a defect definition

Six of the first audit's ten rejected findings came from one criterion: treating a degenerate or empty-input result that looked like ordinary success as a defect.

That criterion had a real motivation. A mistyped selector can match nothing, skip every check, and leave a reassuring success status. But “nothing matched” is also a legitimate outcome for some tools. An optional collection can simply contain no work today.

The old wording collapsed those cases. A better contract asks what the program has promised to verify:

These are proposed contract examples, not a claim that every tool should use the same exit code. The important test is whether the caller can distinguish a valid no-op from a failed precondition.

Before accepting an empty-input finding, I would want the finding to name the intended contract, supply an input that violates it, and show the observed result. “It returned success on zero items” is an observation. Whether that is a defect still needs a specification.

The second audit did not prove an improvement

After the criteria were tightened, the follow-up record contained four findings across 12 guards. Two were accepted and two rejected: 50% rejected findings under those labels.

It is tempting to write “false positives fell from 56% to 50%.” That would imply more than these records establish. The guard sample changed, the criteria changed, and the follow-up contained only four findings. This was not a paired before-and-after experiment on a fixed set of independently labeled cases.

One follow-up rejection also showed why scope matters: a limitation identified in one check was covered by another check in the execution path, according to the recorded verification. Whether a finding describes a component limitation or an unprotected system behavior must be explicit. A limitation can still deserve documentation without proving that the complete path violates its contract.

The follow-up supports a modest conclusion: the revised process still produced findings that were rejected after examination. It does not establish the size of a quality improvement, a reduction in missed defects, or the current performance of the guards.

A report that keeps unknowns visible

You do not need a dashboard to avoid the denominator mistake. Start by keeping accepted, rejected, and unresolved findings separate. This standalone Python example reconstructs the first audit's label counts; it is a reporting example, not the original private audit dataset.

from collections import Counter

# Reconstructed counts from the recorded adjudications.
labels = ["accepted"] * 8 + ["rejected"] * 10
counts = Counter(labels)
allowed = {"accepted", "rejected", "unresolved"}
if set(counts) - allowed:
    raise ValueError("Unknown finding label")

accepted = counts["accepted"]
rejected = counts["rejected"]
resolved = accepted + rejected
report = {
    "unit": "finding",
    "findings_total": len(labels),
    "resolved": resolved,
    "unresolved": counts["unresolved"],
    "precision_among_resolved": accepted / resolved if resolved else None,
    "rejected_share_among_resolved": rejected / resolved if resolved else None,
    "per_guard_false_positive_rate": None,  # Required labels unavailable.
    "recall_on_new_cases": None,  # Missed defects not established.
}
print(report)

For these labels, the two calculated shares are approximately 0.444 and 0.556. If you add an unresolved finding, it increases the total and unresolved count without being silently classified as correct or incorrect. The two shares then describe only the resolved subset; they may not represent the pending findings. With no resolved findings, both remain None rather than becoming a reassuring zero.

For your next ten review findings, keep one row per finding with the criterion, reproduction evidence, expected behavior, verdict, and reason. Preserve unresolved cases. Recount those rows before quoting an aggregate percentage.

If you then want to compare two versions of the checklist, define the evaluation unit and labels first, freeze a separate evaluation set, and run both versions against it. Include legitimate no-op cases as well as known defects. Treat the original seven defects as regression tests, and keep their result separate from the new-case evaluation.

FAQ

Why is "7 out of 7 caught" not evidence that the checklist works?

Because the seven cases shaped the checklist. They are a regression set: useful for confirming the checklist still describes the failures that motivated it, but unable to say how it behaves on a defect it has not seen or how often it accuses correct behavior.

What is the difference between precision and a false-positive rate here?

Precision is accepted findings over all resolved findings (8/18). A false-positive rate needs the true negatives, meaning the guards the checklist correctly left alone, and those were never labeled. The records attach labels to findings, so a per-guard rate cannot be derived from them.

Did tightening the criteria reduce false positives?

Not demonstrably. The follow-up audit had a different guard sample, changed criteria and only four findings (two rejected). Showing an improvement would need a frozen evaluation set with independent labels, run against both versions of the checklist.

Catching every familiar failure was a useful start. The more useful result was learning exactly what I could—and could not—conclude from the next eighteen findings.