Model-Drafted Property Checks Need an Admission Test: Mutants, Fixture Hashes, and a Flaky Floor

A model can draft ten property checks in a minute. Most of them will be green forever, and one of those green checks may be the reason your gate stays quiet while behavior drifts. Drafting is cheap now; admission is the expensive part.

This post describes one admission procedure for checks that arrive from a model rather than a human: lock the fixtures, screen the source, prove the check can fail, prove it fails the same way twice, then cap its runtime. Every gate is deterministic and none of them requires a model to run.

The code below is an illustrative sketch, not a benchmark. I have not executed it for this article, and the numbers in the decision table are hypothetical.

Three ways a drafted check lies without ever failing

Tautology. assert result is not None is a real assertion in some codebases and noise in most. It cannot distinguish a working implementation from a broken one, so it passes on both.

Fixture loosening. The candidate passes because a fixture was widened in the same batch — a tolerance bumped from exact to approximate, a mock returning a superset, a golden file regenerated. The check is fine; the world it measures changed.

Flakiness. A candidate that passes on retry teaches the team to rerun, and rerun-until-green is selection bias with extra steps. This one is worse than a missing check, because it looks like coverage.

Each gate below targets exactly one of these.

Step 0 — Hash the fixtures before drafting starts

Take a manifest before the model touches anything, and recompute it after. If the fixture tree moved while candidates were being drafted, every candidate from that batch is suspect — you no longer know what they were written against.

# admit.py — illustrative sketch, not executed for this article.
import hashlib, pathlib, sys

ROOT = pathlib.Path(".")

def fixture_manifest(root: pathlib.Path, pattern: str = "tests/fixtures/**/*") -> dict[str, str]:
    return {
        str(p.relative_to(root)): hashlib.sha256(p.read_bytes()).hexdigest()
        for p in sorted(root.glob(pattern))
        if p.is_file()
    }

before = fixture_manifest(ROOT)
# ... draft candidates into tests/props/candidate_*.py ...
after = fixture_manifest(ROOT)
changed = sorted(k for k in before | after if before.get(k) != after.get(k))
if changed:
    sys.exit(f"fixture drift during drafting: {changed}")

The operational rule matters more than the script: generated checks run against frozen fixtures, and any fixture edit is a separate commit with a human name on it. Otherwise a batch of candidates can quietly redefine "correct" and then agree with itself.

Step 1 — Screen the source for smells, not for verdicts

A static pass is cheap and catches the laziest failures. It is a smell detector. Treat it as a reject only for unambiguous cases, and as a warning everywhere else.

import ast

HARD_REJECT = {"assert_const"}

def static_screen(path: pathlib.Path) -> set[str]:
    tree = ast.parse(path.read_text())
    flags: set[str] = set()
    for node in ast.walk(tree):
        if isinstance(node, ast.Assert) and isinstance(node.test, ast.Constant):
            flags.add("assert_const")              # assert True / assert 1
        if isinstance(node, ast.Compare) and len(node.ops) == 1:
            op, right = node.ops[0], node.comparators[0]
            if isinstance(op, ast.IsNot) and isinstance(right, ast.Constant) and right.value is None:
                flags.add("assert_only_not_none")  # weak, needs kill evidence
        if isinstance(node, ast.Call) and getattr(node.func, "attr", "") in {"skip", "xfail"}:
            flags.add("skip_or_xfail")
    return flags

assert_only_not_none must not be auto-rejected. Some invariants really are "the object exists and is well-formed downstream". Let Step 2 decide.

Step 2 — Mutant adequacy: prove the check can fail

Keep a small, reviewed mutant set per module under test. A mutant is a deliberate one-line defect — a flipped comparison, an off-by-one, an early return — applied at import time through an environment variable read by conftest.py.

def mutant_kills(candidate: str, mutants: list[str]) -> int:
    kills = 0
    for name in mutants:
        env = {**os.environ, "MUTANT": name, "PYTHONHASHSEED": "0"}
        r = subprocess.run(
            [sys.executable, "-m", "pytest", candidate, "-q", "-x"],
            env=env, capture_output=True, text=True,
        )
        kills += int(r.returncode != 0)
    return kills

Two details decide whether this gate is real or theatrical:

1. The conftest.py hook must raise on an unknown MUTANT value. A typo that silently applies nothing returns a survivor, and a survivor looks like a weak check rather than a broken harness.
2. The mutant set must be reviewed like test code. It bounds every conclusion you draw from it.

Zero kills is "unproven", not "useless". The mutant set may simply not cover that invariant. Route zero-kill candidates to human review with the mutant list attached instead of deleting them — otherwise the gate optimizes for the sins it already knows how to catch.

Step 3 — Determinism, and a flaky floor for candidates

Run each survivor three times on the unmutated tree with a fixed hash seed and a fixed application seed, then compare exit codes.

def outcomes(candidate: str, runs: int = 3) -> set[int]:
    seen = set()
    for _ in range(runs):
        r = subprocess.run(
            [sys.executable, "-m", "pytest", candidate, "-q"],
            env={**os.environ, "PYTHONHASHSEED": "0", "PROP_SEED": "1234"},
            capture_output=True,
        )
        seen.add(r.returncode)
    return seen

if len(outcomes("tests/props/candidate_a.py")) > 1:
    print("candidate is flaky -> freeze the candidate, not the suite")

A flaky candidate gets frozen at candidate scope, not suite scope. That distinction is the whole point: freezing the suite hides which check is unreliable, and the freeze then outlives its cause. A frozen candidate may keep running informationally, but it cannot be cited as evidence that a patch is safe.

Step 4 — Cap wall clock per candidate

Measure the candidate in isolation and park anything over budget in a nightly pool. A gate that takes minutes per patch gets bypassed within a week, regardless of how good its assertions are.

Where free model access actually helps

The drafting step is the one part of this pipeline that benefits from a model. Candidate checks are text, they are disposable, and volume helps: the more candidates you generate, the more likely one survives all four gates.

MonkeyCode's free model access is relevant here — it lowers the cost of generating candidate properties you expect to throw away. The free server option is relevant at Step 2, because a mutant sweep is a batch of subprocess runs rather than an interactive loop, which is exactly the shape of work you want off your laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the harness model-agnostic on purpose. Availability and limits for free tiers change, so verify current terms in the product before depending on them, and make sure swapping providers changes nothing except the drafting prompt. Validation must stay local and deterministic — the gates should produce the same verdict whether the candidate came from a large model, a small one, or a bored colleague.

Decision table

Counts below are hypothetical, shown to make the routing explicit.

Verify the pipeline, not just the candidates

Before trusting the harness, feed it two synthetic candidates:

1. A file containing assert True. Expected: Step 1 rejects it.
2. A check known to fail against a specific mutant in the set. Expected: Step 2 counts exactly one kill, and Step 3 returns a single exit code across three runs.

If the first is accepted, your static screen is not wired to the reject path. If the second reports zero kills, your MUTANT hook is not applying anything — check that before you conclude the check is weak.

Limitations and who should not use this

• Adequacy is bounded by the mutant set. A check that kills nothing proves nothing about the invariant you care about if no mutant touches it.
• Overfit checks pass and lie later. A drafted check can encode the current implementation closely enough to fail on a correct refactor. Mutation adequacy does not detect this; a refactor dry run does.
• The sketches assume one Python repo with pytest. The gates translate to other stacks, the code does not.
• Skip this if patches already get per-patch human review. Four gates to save a reviewer five minutes is a bad trade.

Admission is boring work: hash, screen, mutate, repeat, time it. That is the point. The interesting part is the check someone cared enough to prove could fail.

If you run something like this, the MUTANT hook is the part worth comparing notes on — a harness that silently applies no mutation is worse than no harness at all.