From AI Finding to Deterministic Guard

An AI reviewer says a parser accepts a malformed document. The comment is plausible. It is not yet a regression test, and it is not yet a policy.

Between a model finding and a reliable guard sits a small but essential engineering process: validate the behavior, identify an owner, reproduce the failure, encode the invariant, and prove the new check catches the case without breaking the supported ones.

Treat the model output as a lead

A useful finding should describe more than a suspicious line. It should offer a concrete path from input to consequence:

• Which input or state triggers the behavior?
• What does the system do?
• What invariant is violated?
• What would a correct result look like?
• Which evidence supports the claim?

If those questions have no answer, ask for clarification or reproduce the behavior before editing code. A model can be right about the shape of a risk while wrong about reachability, ownership, or intended behavior.

Do not let the model write a test that merely enshrines its own interpretation. A test is executable policy: a maintainer must agree that the assertion is the rule the product should keep.

The finding-to-guard path

1. Assign a human owner

Choose someone who understands the affected boundary or can route it to the right owner. The model may help locate code; it does not take accountability for deciding priority or semantics.

2. Reproduce or reject

Build the smallest reliable example. It might be a failing unit test, an integration fixture, a browser path, or a packaged-runtime reproduction. Record the environment and version. If the reproduction fails, preserve that result and explain why the original hypothesis was rejected.

“Could not reproduce” is not the same as “impossible.” It may mean the environment or evidence was incomplete. Keep uncertainty visible.

3. State the invariant

Write the expected rule in language a reviewer can assess. For example: “An import that fails validation must not replace the currently selected project.” That is more useful than “handle the error safely.”

4. Add the smallest durable check

Encode the invariant at the lowest test layer that can reliably detect the behavior. A unit test may be enough for a pure parser rule. A persistence boundary may need an integration test. A first-launch claim may require a packaged application. A test at one evidence plane does not prove another.

5. Prove the guard has signal

Run it on the known failing case and on representative supported cases. Where appropriate, temporarily remove or invert the behavior to show the test fails for the defect it is meant to catch. Avoid asserting that a test is strong simply because it passed once.

6. Connect the check to the policy

If this behavior must block a merge, bind the result to an owned workflow and the exact revision through repository rules. A local green run and a required merge check are different facts.

A finding is not automatically a rule

Some model findings are contextual, subjective, or too noisy to encode. A one-off design trade-off may need a decision record, not a static analyzer. A threat hypothesis may need a security review. A code-style suggestion may belong in a formatter—or may not deserve a comment at all.

The right outcome can be:

• accept and add a regression guard;
• accept the design change but use a different test;
• reject the finding with a short reason;
• defer it to a named issue because required evidence is unavailable;
• escalate if the potential consequence needs specialist review.

Forcing every observation into a CI rule creates a brittle test suite. Ignoring every observation because the model is imperfect throws away useful attention. Human triage is the conversion boundary.

Preserve evidence provenance

For a durable guard, record what it tested and what it cannot show:

• fixture identity and expected behavior;
• test command and relevant environment;
• source revision;
• whether the check ran locally, in CI, in a browser, or on a packaged artifact;
• known blind spots and any deferred platform path.

This prevents later readers from turning “the test passed” into a stronger claim than the test supports.

From correction to institutional memory

A review comment disappears into a pull-request timeline. A regression test can keep the lesson available to the next contributor. A static policy can make a repeated architectural boundary explicit. A runbook can teach a human how to investigate a failure that cannot yet be automated.

The goal is not to automate every judgment. It is to avoid paying again for a failure mode that can be stated precisely and checked cheaply.

In the next post, we will look at a harder version of the same problem: how to know that the evaluator itself—the tests, tools, and evidence pipeline—has not drifted away from the claim it is supposed to support.

References

• SWE-bench (2023) — a benchmark for issue-resolution agents, not a production assurance claim.
• SWE-Proof (2026 preprint) — formal checking of test-passing patches in a purpose-built corpus; results are corpus- and specification-dependent.
• GitHub ruleset rules
• SLSA v1.2 provenance

AI assistance was used to prepare this draft. The human editor is responsible for choosing the invariant, validating the example, and reviewing the publication claims.