Fixtures Are Part of the Claim: Gating Agent Patches on Test Support Diffs

A green suite after an agent patch is a claim about the fixture version that just ran, not about the fixture you reviewed last month. If the patch moved the fixture, the passing test is a different test. This article describes a gate for that one surface: the support files that decide what an assertion actually sees.

Prerequisites: a git history where the patch is a diffable commit or range, a test layout you can enumerate by path, and the willingness to treat fixtures as reviewed code. Nothing here depends on a specific test runner. The code below is illustrative and was not executed — treat it as a shape to adapt, not a tool to install.

The failure mode, stated precisely

A test result is a conjunction of three things: assertion logic, fixture state, and wiring. Agent-produced patches can edit all three in one diff, and reviewers tend to read the first and skim the other two.

There are two vacuity routes. Assertion tautology makes a check unable to fail. Fixture vacuity makes a check unable to observe anything — the data it inspects became empty, degenerate, or was replaced by a value the assertion expects.

A property like "the sum of line totals equals the order total" holds trivially on an empty order. If a patch changes a factory default from a seeded two-line order to an empty one, the property still passes and now proves nothing. The assertion did not change. The meaning did.

Step 1: Enumerate the fixture surface by path

Do not split on directory names like tests/. Split on intent: does this file change what an assertion observes? Keep the glob list in the repository so the gate itself is reviewable.

tests/fixtures/**
tests/factories/**
tests/baselines/**
**/__snapshots__/**
tests/helpers/**
tests/conftest.py

Anything outside that list is application code and follows the normal review path. Anything inside it is a claim about test meaning and needs an explicit justification in the patch description.

Step 2: Classify the diff, not the file

The useful unit is the (category, change kind) pair. A new factory method is not the same risk as a changed factory default. This table is the decision rule to put in a PR checklist.

The right-hand column is the part most gates skip. "Tests pass" is not evidence about a fixture change; it is the outcome the fixture change was supposed to make true.

Step 3: Pin properties to a fixture digest

Compute a digest over the fixture surface at the patched tree and record it next to every property result. This makes the coupling explicit and cheap to re-check later.

# Illustrative, unexecuted. Stdlib only. Adapt globs to your layout.
import hashlib
from pathlib import Path

FIXTURE_GLOBS = [
    'tests/fixtures/**/*', 'tests/factories/**/*', 'tests/baselines/**/*',
    '**/__snapshots__/**/*', 'tests/helpers/**/*', 'tests/conftest.py',
]

def fixture_files(root: Path) -> list[Path]:
    seen = {}
    for pattern in FIXTURE_GLOBS:
        for p in root.glob(pattern):
            if p.is_file():
                seen[p.relative_to(root)] = p
    return [seen[k] for k in sorted(seen)]

def fixture_digest(root: Path) -> str:
    h = hashlib.sha256()
    for p in fixture_files(root):
        h.update(str(p.relative_to(root)).encode())
        h.update(b'\0')
        h.update(p.read_bytes())
        h.update(b'\0')
    return h.hexdigest()[:16]

Two rules follow from the digest. First, a property result is only comparable to another result recorded at the same digest. Second, a flaky-test freeze measured at digest A does not carry over to digest B — it expires on fixture change, not on a calendar date.

That second rule is the narrow delta to a general quarantine policy. Calendar expiry assumes time is what makes a freeze stale. On agent patch queues, it is usually the fixture that moved.

Step 4: Order the gate

Order matters, because each step can invalidate the previous one.

1. Compute the fixture surface for the patch range.
2. Classify each entry as (category, change kind); reject unclassified support-file edits.
3. Compute the fixture digest for the patched tree.
4. Look up existing freezes for the touched tests and drop any recorded at a different digest.
5. Run property checks, labelling each result with the digest.
6. Only then run test selection — selection keyed on a stale digest selects the wrong suite.

Skipping step 4 is the common mistake. A freeze recorded before the fixture moved will silently suppress exactly the test the fixture change broke.

Step 5: An illustrative walkthrough

Hypothetical; no real repository, patch, or run is implied.

A patch adds a discount path for line items and also edits tests/factories/order.py. Step 2 classifies that edit as (factory, changed default). The reviewer asks the Step 2 question — which properties read this factory — and finds the totals property from earlier. On the patched tree the order is empty, so the property passes without touching the discount code at all. Step 3 shows a new digest, so the previously frozen totals test no longer carries its freeze and runs. It fails. The patch is not rejected for being wrong; it is rejected for arriving with a fixture that hid the failure.

That is the whole value of the gate: it converts "the suite is green" into "the suite was green against these fixtures, and here they are."

Where a shared workbench changes the picture

If your team runs agent-produced patches through a shared workbench, the fixture surface is also a permissions surface. MonkeyCode (open-source repository: github.com/chaitin/MonkeyCode) is described in its current README as a team AI workbench covering office, coding and design, with shared team resources, configurable models, MCP integrations, access management, and self-hosting.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two of those documented properties interact with this gate honestly. Shared team resources mean a fixture, template, or baseline may be reachable by more than one person's agent run, so a fixture diff can originate outside the patch author's own review. Access management is the control that decides who may modify that shared surface — which is the same question Step 2 asks in prose. If you automate fixture edits at all, the permission boundary and the review boundary should name the same files.

One naming caution: the repository also contains README.monkeycode.md, which documents the historical MonkeyCode AI development platform. That is a different document from the current workbench README; do not assume the two describe an identical feature set. For self-hosted deployments, note as well that external model and tool providers may still receive data through configured integrations — self-hosting is not by itself a data-locality guarantee.

Limitations and who should not use this

• Monorepos with fixtures inside application packages break the glob split. You need an explicit manifest, and the manifest becomes the thing to review.
• Generated fixtures defeat byte-level digests; digest the generator inputs instead.
• Snapshot-heavy frontends produce large regenerated diffs, and the Step 2 evidence requirement becomes real review work rather than a checkbox.
• No stable base commit means digests are not comparable; fix the baseline before adding the gate.
• Small, human-authored, fixture-light patches do not justify the tooling. Use the Step 2 table as a checklist and skip the rest.

How to check it works on your repository

Do not trust the classifier because it looks right. Plant three fixture-only changes on a throwaway branch — a shrunk data case, a changed factory default, a regenerated baseline — and confirm each is classified and each produces a different digest. Then confirm a freeze recorded on the base commit is dropped for the patched tree. If any of the three slips through, the globs are wrong, and no amount of downstream assertion review will catch it.