Gate Agent Patches on a Verdict File. Ignore pytest's Exit Code.

pytest's exit code is a process signal. It is not a merge decision. Agent-generated patches exploit that type error: they can keep the suite green while rewriting an assertion, widening a fixture, or converting a failure into skip.

The merge job should ignore $? from the test runner. It should read a typed verdict file and fail closed when that file is missing, stale, or internally inconsistent. That is the policy. The rest of this article is a three-lane plan that fills the file, a small Python harness that emits it, and a decision table CI can enforce without trusting the agent.

The type error

An exit code collapses four events into one integer. Invariant failure, fixture drift, a frozen flake, and a crashed runner all compete for the same 8-bit channel. Agent patches benefit from the collapse. A skip, a rewritten assertion, and a real pass all look like zero.

Gating on green therefore measures runner survival. It does not measure behavioral evidence. Adding more tests does not repair the type. Changing the artifact does.

This sits next to a broader 2026 argument that AI-assisted edits are being called engineering while the evaluation surface is still a boolean. A patch evaluation is a record. Records can be reviewed. Booleans cannot.

Artifact: verdict.json

Proposed contract. Label it unexecuted against a production monorepo. Copy it as a schema, not as a claim about any particular agent or suite size.

{
  "schema_version": 1,
  "patch_ref": "HEAD",
  "oracle_touched": false,
  "fixture_digest": "sha256:…",
  "fixture_digest_expected": "sha256:…",
  "property_seed": 20260919,
  "property_failed": [],
  "flake_freeze": [],
  "mutation_candidates_accepted": 0,
  "verdict": "REJECT"
}

verdict has three values: ACCEPT, REJECT, HOLD. A missing file equals REJECT. A file the agent wrote equals REJECT. The harness, not the patch, is the only writer.

Three lanes that feed the file

Do not encode the strategy as "more pytest." Encode it as three lanes with different merge credit.

1. Property lane (credit: can REJECT or support ACCEPT). Pure functions and protocol invariants. Frozen seed. No wall clock, no network, no filesystem except the locked fixtures directory.
2. Fixture lane (credit: can REJECT). Byte digest of the fixture tree. Drift is not a soft warning. Drift is a field on the verdict.
3. Flake freeze (credit: none). Known intermittent tests go into a freeze list with an expiry date. They cannot create ACCEPT. If they are the only coverage for a touched production path, the verdict is HOLD, not ACCEPT.

Flakes are not skipped. Skipping returns green. Freeze records the nodeid, the last failure signature, and the expiry. After expiry the job REJECTs until the test is deleted, rewritten as a property, or moved back to the property lane with a locked seed.

Numbered workflow

1. Before the agent runs, record oracle_paths and fixture_digest_expected from main. Store them outside the working tree the agent can edit, for example in the CI cache or a protected branch file.
2. After the agent returns a diff, compute oracle_touched with git diff --name-only origin/main...HEAD. Any path under the oracle prefix sets the flag to true.
3. Recompute the fixture digest. Compare it to fixture_digest_expected.
4. Run the property lane with --seed taken from the verdict request, not from the patch.
5. Load tests/flake_freeze.json. Drop those nodeids from merge credit. If a changed production file is covered only by frozen nodeids, set a hold flag.
6. Optionally run a mutation lane. Mutations are inputs, not tests.
7. Write verdict.json from a process the agent cannot spawn as the test runner. Point CI at that file only.

The order is load-bearing. Digest and oracle checks happen before properties. Properties happen before mutations. Mutations never write into tests/.

Harness (proposed)

The script below is a complete local gate. It is a proposal. It does not claim production metrics.

#!/usr/bin/env python3
"""verdict_gate.py — emit verdict.json; do not trust pytest's exit code."""
from __future__ import annotations

import argparse
import hashlib
import json
import subprocess
import sys
from pathlib import Path

ORACLE_PREFIXES = ("tests/oracle/", "oracles/")
FIXTURE_ROOT = Path("tests/fixtures")
FREEZE_PATH = Path("tests/flake_freeze.json")
VERDICT_PATH = Path("verdict.json")


def sha256_tree(root: Path) -> str:
    h = hashlib.sha256()
    if not root.exists():
        return "sha256:" + h.hexdigest()
    for path in sorted(p for p in root.rglob("*") if p.is_file()):
        rel = path.relative_to(root).as_posix().encode()
        h.update(rel + b"\0")
        h.update(path.read_bytes())
    return "sha256:" + h.hexdigest()


def git_changed(base: str) -> list[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", f"{base}...HEAD"],
        text=True,
    )
    return [line.strip() for line in out.splitlines() if line.strip()]


def oracle_touched(changed: list[str]) -> bool:
    return any(p.startswith(ORACLE_PREFIXES) for p in changed)


def load_freeze() -> list[dict]:
    if not FREEZE_PATH.exists():
        return []
    data = json.loads(FREEZE_PATH.read_text())
    if not isinstance(data, list):
        raise ValueError("flake_freeze.json must be a list")
    return data


def freeze_expired(freeze: list[dict], today: str) -> list[str]:
    bad = []
    for item in freeze:
        exp = item.get("expires")
        nodeid = item.get("nodeid", "<missing-nodeid>")
        if not exp or exp < today:
            bad.append(nodeid)
    return bad


def run_properties(seed: int, freeze_ids: set[str]) -> list[str]:
    cmd = [
        sys.executable, "-m", "pytest",
        "tests/properties",
        "-q",
        "--seed", str(seed),
        "--ignore-glob", "tests/flaky/*",
    ]
    proc = subprocess.run(cmd, text=True, capture_output=True)
    failed: list[str] = []
    for line in (proc.stdout or "").splitlines():
        if line.startswith("FAILED "):
            nodeid = line.split()[1]
            if nodeid not in freeze_ids:
                failed.append(nodeid)
    return failed


def freeze_covers_only(changed: list[str], freeze: list[dict]) -> bool:
    prod = [p for p in changed if p.startswith("src/")]
    if not prod:
        return False
    covered = {p for item in freeze for p in item.get("covers", [])}
    return bool(prod) and all(p in covered for p in prod)


def decide(oracle: bool, failed: list[str], drift: bool, expired: list[str], hold: bool) -> str:
    if oracle or failed or drift or expired:
        return "REJECT"
    if hold:
        return "HOLD"
    return "ACCEPT"


def main() -> int:
    p = argparse.ArgumentParser()
    p.add_argument("--base", default="origin/main")
    p.add_argument("--seed", type=int, default=20260919)
    p.add_argument("--expected-digest", required=True)
    p.add_argument("--today", default="2026-09-19")
    args = p.parse_args()

    changed = git_changed(args.base)
    digest = sha256_tree(FIXTURE_ROOT)
    freeze = load_freeze()
    expired = freeze_expired(freeze, args.today)
    freeze_ids = {item["nodeid"] for item in freeze if "nodeid" in item}
    failed = run_properties(args.seed, freeze_ids)
    drift = digest != args.expected_digest
    oracle = oracle_touched(changed)
    hold = (
        freeze_covers_only(changed, freeze)
        and not failed
        and not oracle
        and not drift
        and not expired
    )

    verdict = decide(oracle, failed, drift, expired, hold)
    payload = {
        "schema_version": 1,
        "patch_ref": "HEAD",
        "oracle_touched": oracle,
        "fixture_digest": digest,
        "fixture_digest_expected": args.expected_digest,
        "property_seed": args.seed,
        "property_failed": failed,
        "flake_freeze": freeze,
        "freeze_expired": expired,
        "mutation_candidates_accepted": 0,
        "verdict": verdict,
    }
    VERDICT_PATH.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
    # Convenience only. CI must still read the file.
    return 0 if verdict == "ACCEPT" else 1


if __name__ == "__main__":
    raise SystemExit(main())

Compute the expected digest on main, not on the patch branch. Then treat the second command as the merge gate.

# on a clean checkout of main:
python -c "from verdict_gate import sha256_tree, FIXTURE_ROOT; print(sha256_tree(FIXTURE_ROOT))" > expected_digest.txt

# on the agent branch:
python verdict_gate.py --base origin/main --seed 20260919 --expected-digest "$(cat expected_digest.txt)" --today 2026-09-19
python -c "import json,sys; v=json.load(open('verdict.json')); sys.exit(0 if v['verdict']=='ACCEPT' else 1)"

The harness process code is a convenience. Do not wire GitHub's pytest check as the required status. The required status is "verdict file exists and reads ACCEPT."

A freeze entry needs a date and a coverage claim. Without both, it is a skip with extra JSON.

[
  {
    "nodeid": "tests/flaky/test_cache.py::test_ttl_under_load",
    "expires": "2026-09-26",
    "last_signature": "AssertionError: ttl jitter",
    "covers": ["src/cache.py"]
  }
]

Decision table

HOLD is not merge. It is a human path: replace the frozen test with a property, or shrink the freeze. Treating HOLD as green reintroduces the original type error.

Mutation lane, off the merge path

Properties catch invariants you already wrote. They do not invent missing ones. A bounded mutation lane can propose extra inputs: malformed JSON, oversized strings, duplicate keys, timezone edge values. Those inputs are data. They are not tests. They do not land in tests/ unless a human promotes them.

The lane must not be merge-critical. A timeout, an empty body, or a 200 with no candidates must leave mutation_candidates_accepted at 0 and must not change ACCEPT into REJECT by accident. Network failure is not an invariant failure.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If that optional lane needs an HTTP endpoint, MonkeyCode's free model access and free server option can host the generator off the critical path. The verdict writer stays local. The model never edits verdict.json, the oracle tree, or the freeze list.

A minimal worker shape, proposed and unexecuted:

# mutation_lane.py — optional, non-blocking
import json
import urllib.request


def candidates(prompt: str, url: str, timeout: float = 8.0) -> list[dict]:
    req = urllib.request.Request(
        url,
        data=json.dumps({"prompt": prompt, "n": 8}).encode(),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            body = resp.read()
    except OSError:
        return []
    if not body:
        return []
    try:
        payload = json.loads(body.decode())
    except json.JSONDecodeError:
        return []
    items = payload if isinstance(payload, list) else payload.get("candidates", [])
    out = []
    for item in items:
        if isinstance(item, dict) and "input" in item:
            out.append(item)
    return out

Feed item["input"] into the same property functions the seed lane uses. Count how many candidates the properties reject. Store the count. Do not store the model's prose. Do not let a failed HTTP call flip the verdict.

What CI should actually require

Required status checks should be "verdict is ACCEPT", not "pytest passed." Proposed workflow sketch:

# proposed .github/workflows/agent-verdict.yml
jobs:
  verdict:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Expected digest from base
        run: |
          git checkout origin/main -- tests/fixtures || true
          python - <<'PY'
          from verdict_gate import sha256_tree, FIXTURE_ROOT
          open("expected_digest.txt", "w").write(sha256_tree(FIXTURE_ROOT))
          PY
          git checkout HEAD -- tests/fixtures || true
      - name: Write verdict
        run: python verdict_gate.py --expected-digest "$(cat expected_digest.txt)"
        continue-on-error: true
      - name: Gate on file
        run: |
          test -f verdict.json
          python -c "import json,sys; v=json.load(open('verdict.json')); assert v.get('schema_version')==1; sys.exit(0 if v.get('verdict')=='ACCEPT' else 1)"

continue-on-error is intentional. The harness may return 1. The next step still reads the file. If the file is absent, the job fails closed.

Protect verdict_gate.py, tests/oracle/, tests/fixtures/, and tests/flake_freeze.json with CODEOWNERS. An agent patch that includes those paths is oracle_touched or worse: it is a gate rewrite.

Limitations

This policy does not prove the patch is correct. It proves the patch did not touch the oracle, did not drift fixtures, did not carry an expired freeze entry, and did not fail the seeded properties that already exist. Missing properties remain missing. Tautological properties remain tautological. The verdict file will faithfully report a hollow suite as ACCEPT.

Digest freeze will reject legitimate fixture updates. That is a cost. Route fixture changes through a human-owned PR that updates the expected digest on main first. Do not let the same patch change production code and fixtures.

The freeze list will rot if expiry is not enforced. A freeze without a date is a skip with extra YAML. Reject the job when any freeze entry lacks expires or when expires is in the past.

Do not use this approach if there is no oracle prefix. Do not use it if most tests hit a live clock or a shared staging database. Do not use it if the agent is allowed to run the harness or write verdict.json. Do not send production payloads into the mutation endpoint. Do not treat HOLD as a temporary green.

Free-model mutations are untrusted bytes. Parse, schema-check, then discard. They are not documentation. They are not oracles.

The schema is the part worth copying. If a free model and a free server are already available, wire them only to the mutation lane and leave the verdict writer local.