The Fixture Moved With the Assertion: A Co-Movement Gate for Agent Patches

An agent patch can turn a red suite green without fixing anything, and the cheapest way it does that is not a skipped test or a reordered run. It edits the input fixture and the expected value in the same commit. The assertion still runs, the run still passes, and the only thing that changed is what the test is willing to accept.

This post is about catching that specific move mechanically, before merge. It assumes you already check properties and you already freeze flaky tests; the missing piece is a rule that treats the fixture and the oracle as one unit of evidence.

The failure mode: evidence and oracle moving together

A test is a claim of the form "given this input, I expect this output." An agent under pressure to make it green has two knobs: the code, or the claim. Turning the second knob is often locally correct — the spec really did change — and that is exactly why it cannot be waved through or banned outright.

What you can ban is silence. If the input fixture and the expected literal both move, someone must state which behavior changed and produce a witness for it. If only the fixture moves, the test has been re-pointed at an easier input. If only the expected literal moves, the oracle has been loosened. All three deserve a verdict, not a vibe.

Step 1 — Classify every changed file by role

Start with the diff, not with the agent's description of the diff. Each path gets exactly one role, and order matters: the first classifier that matches wins.

#!/usr/bin/env python3
# comove.py - verdicts for agent patches where evidence and oracle co-move
import re, subprocess, sys, json

CLASSIFIERS = [
    ("snapshot", re.compile(r"(^|/)(__snapshots__|golden)/|\.snap$")),
    ("test",     re.compile(r"(^|/)tests?/|_test\.py$|\.test\.[jt]sx?$|_spec\.rb$")),
    ("fixture",  re.compile(r"(^|/)(fixtures?|testdata)/")),
    ("source",   re.compile(r"^(src|lib|app|internal|pkg)/")),
]

def role(path: str) -> str:
    for name, rx in CLASSIFIERS:
        if rx.search(path):
            return name
    return "other"

Narrow CLASSIFIERS to your real layout on day one. A tests/fixtures/ directory that also matches the test rule will silently collapse two roles into one, and the gate will then miss the case it exists for.

Step 2 — Pull the oracle literals out of the test diff

You do not need to parse the language. You need the constants that a human would call "the expected value," taken only from removed lines, because a removed constant that was not re-added is what "the oracle changed" means.

ORACLE = re.compile(r"assert\s+\S.*?==\s*(.+)$|\.to(Be|Equal)\((.+?)\)\s*;?\s*$")

def oracle_literals(lines):
    seen = set()
    for raw in lines:
        line = raw[1:].strip()          # drop the diff marker
        m = ORACLE.search(line)
        if m:
            seen.add((m.group(1) or m.group(3)).strip())
    return seen

def parse(patch: str):
    files, added, removed = {}, [], []
    for line in patch.splitlines():
        if line.startswith("+++ b/"):
            files.setdefault(line[6:], None)
        elif line.startswith("+") and not line.startswith("+++"):
            added.append(line)
        elif line.startswith("-") and not line.startswith("---"):
            removed.append(line)
    return {role(f) for f in files}, oracle_literals(added), oracle_literals(removed)

Using before - after rather than a symmetric difference is deliberate. A patch that only adds a new test introduces new constants and should not be flagged; a patch that replaces 200 with 500 should be.

Step 3 — Score the patch with a five-row table

The verdict is a function of three inputs: did source move, did evidence (test/fixture/snapshot) move, and did a removed oracle literal disappear. Write the table down before you argue about any individual patch.

def verdict(roles, before, after):
    oracle_changed = bool(before - after)
    if "source" not in roles and roles <= {"test", "fixture", "snapshot"}:
        return "BLOCK", "evidence-only patch: no product code changed"
    if "fixture" in roles or "snapshot" in roles:
        return "REVIEW", "fixture/snapshot moved in the same patch"
    if oracle_changed:
        return "REVIEW", "expected value changed: attach an oracle witness"
    return "PASS", "run the suite; record the hash of this diff"

$ git diff -U0 HEAD~1 HEAD > /tmp/p.diff
$ python comove.py HEAD~1 HEAD
{
  "verdict": "REVIEW",
  "reason": "fixture/snapshot moved in the same patch",
  "roles": ["fixture", "source", "test"]
}

A "witness" is a concrete artifact a reviewer can check without trusting the agent: a linked spec change, a failing-before/passing-after run against the old fixture, or a property test that still holds on the widened input. If none of those can be produced, the patch is a claim, not evidence.

Step 4 — Make every freeze declare what it withdraws

A flaky-test freeze is a loan against your coverage, and most teams record the loan without recording the collateral. Keep the ledger explicit, with a due date and the command that re-earns the freeze.

# freezes.yaml - a freeze is a loan against a property, with a due date
- test: tests/test_retry.py::test_backoff_caps_at_max
  property: PROP-RETRY-CAP
  why: "upstream clock skew makes the 12s case nondeterministic"
  witness: "python -m pytest tests/test_retry.py -k cap -q"
  owner: "@finley"
  expires: 2026-11-02

Then join the two artifacts: if a patch touches source mapped to PROP-RETRY-CAP and that property's only witness is frozen, the gate must emit BLOCK — not PASS with a footnote. This is the check that stops co-movement from being laundered through a freeze: an agent cannot loosen the oracle of a test that is already skipped, so the frozen witness has to count against the patch instead.

python - <<'PY'
import yaml, json, subprocess
freezes = yaml.safe_load(open("freezes.yaml"))
frozen = {f["property"] for f in freezes}
touched = set(json.load(open("properties-touched.json")))
print(json.dumps({"blocked": sorted(frozen & touched)}, indent=2))
PY

Step 5 — Run the gate where the patch is produced

The gate needs no model at runtime: it reads a diff and prints a verdict, so it can sit in the same job as the suite. Because it is CPU-only and short-lived, it fits a small always-on runner, which is where the operator-supplied free server option on MonkeyCode is actually relevant — the audit job can run next to the agent instead of consuming a second paid CI slot. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Free model access is useful one step later, for the REVIEW bucket: ask a model to draft the missing witness description for a co-movement patch, then treat that draft as a claim to verify rather than as evidence. The verdict comes from the diff; the model only helps a human fill in the blank faster. Keep the gate's output as the gate's output.

Self-test: six synthetic diffs

A gate you have not tried to fool is a gate you do not know. These six cases are table-driven and run offline, no repository required.

import comove, pytest

CASES = [
    ({"source"},                    set(),   set(),   "PASS"),
    ({"source", "test"},            set(),   {"1"},  "PASS"),
    ({"source", "test"},            {"200"}, {"500"},"REVIEW"),
    ({"source", "test", "fixture"}, set(),   set(),   "REVIEW"),
    ({"test", "snapshot"},          set(),   set(),   "BLOCK"),
    ({"source", "docs"},            set(),   set(),   "PASS"),
]

@pytest.mark.parametrize("roles,before,after,want", CASES)
def test_verdict(roles, before, after, want):
    got, _ = comove.verdict(roles, before, after)
    assert got == want

$ python -m pytest test_comove.py -q
6 passed

Add a seventh case the first time a real patch slips past you. That case is worth more than any rule you copy from this post.

What this gate cannot see

• Generated fixtures. If a script rebuilds testdata/ on every run, fixture diffs are noise and the REVIEW bucket will flood. Pin generated fixtures by hash or exclude the directory explicitly.
• Refactors. Moving a constant into a named variable makes a removed literal vanish without any semantic change. Expect false positives on renames; the REVIEW bucket is where they belong.
• Languages without literal oracles. Golden-file snapshot tools, schema validators, and property tests express expectations structurally. The regex sees nothing there, so those suites need their own role rule.
• Intent. A correct spec change and a laundered assertion look identical in a diff. Only the witness separates them, and the witness is a human artifact.

Who should skip this

Do not adopt the gate if nobody owns the test suite, because every REVIEW will become a rubber stamp. Skip it in repositories where fixtures are generated artifacts and oracles live in external schemas — the co-movement signal is empty there. And skip it on a solo prototype where a red test is a note to yourself; the ledger costs more than the bug it catches.

For everyone else, the sequence is short: classify the diff, extract the oracle, score the table, join the freezes, run the six cases. If you wire it up, tell me which verdict your repository disagrees with first — that disagreement is the rule you actually needed.