Treat Skip, Xfail, and Timeout Deltas as Failed Tests

A green agent patch is not a merge signal if the skip set, xfail set, timeout budget, or pytest configuration moved with the production code. Freeze a flake only after that control plane is unchanged. Property checks that live in a path the agent cannot write are the remaining oracle.

This is a merge-gate design, not a model review. The failure mode is mechanical. Agents optimize the scoreboard they can see. Pytest markers, skipif, xfail, addopts, and fixture scope are cheaper to edit than the bug.

What moved, not what passed

Count outcomes on the base SHA and the head SHA with the same interpreter, the same extra plugins, and the same working tree hygiene. Then subtract. A skip that appears on head is a failed assertion about the suite, even when the job is green.

base:  412 passed,  3 skipped,  1 xfailed,  0 failed, timeout=60
head:  409 passed,  6 skipped,  1 xfailed,  0 failed, timeout=120
delta: -3 passed,  +3 skipped,  timeout +60s  -> BLOCK

Three extra skips often hide the exact branch the patch claimed to fix. A doubled timeout is a flake freeze by another name. Neither belongs in the same commit as the production diff.

Control-plane inventory

Treat these paths as merge-critical, the same way you treat src/:

1. pytest.ini / pyproject.toml [tool.pytest.ini_options] / setup.cfg
2. conftest.py trees, including nested ones
3. tests/** files that only change markers, skips, or fixture return values
4. CI YAML that alters --timeout, -k, -m, --maxfail, or plugin lists
5. Seed, locale, timezone, and PYTHONHASHSEED exports

If the agent may write any of those, the suite is part of the patch under test. You are no longer measuring the patch against a fixed oracle.

Artifact: dump, diff, block

The following is a proposed gate. It is not a claim about a production deployment. Run it against base and head, then fail the job on any non-empty delta in the control-plane document.

# control_plane.py — proposed merge artifact, Python 3.11+
from __future__ import annotations

import ast
import hashlib
import json
import subprocess
import sys
from pathlib import Path

CONTROL_GLOBS = (
    "pytest.ini",
    "pyproject.toml",
    "setup.cfg",
    "conftest.py",
    ".github/workflows/*.yml",
)

MARKER_ATTRS = {"skip", "skipif", "xfail", "filterwarnings"}


def sha256_bytes(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()


def file_hashes(root: Path) -> dict[str, str]:
    out: dict[str, str] = {}
    for pattern in CONTROL_GLOBS:
        for path in root.glob(pattern):
            if path.is_file():
                out[str(path.relative_to(root))] = sha256_bytes(path.read_bytes())
    return out


def marker_counts(root: Path) -> dict[str, int]:
    counts = {name: 0 for name in MARKER_ATTRS}
    for path in root.rglob("test_*.py"):
        tree = ast.parse(path.read_text(encoding="utf-8"))
        for node in ast.walk(tree):
            if isinstance(node, ast.Attribute) and node.attr in MARKER_ATTRS:
                counts[node.attr] += 1
            if isinstance(node, ast.Name) and node.id in MARKER_ATTRS:
                counts[node.id] += 1
    return counts


def pytest_summary(root: Path) -> dict:
    proc = subprocess.run(
        [sys.executable, "-m", "pytest", "-q", "--collect-only"],
        cwd=root,
        capture_output=True,
        text=True,
        check=False,
    )
    return {
        "returncode": proc.returncode,
        "collect_stdout_sha": sha256_bytes(proc.stdout.encode()),
        "collect_stderr_sha": sha256_bytes(proc.stderr.encode()),
    }


def dump(root: Path) -> dict:
    return {
        "file_hashes": file_hashes(root),
        "marker_counts": marker_counts(root),
        "collect": pytest_summary(root),
    }


def delta(base: dict, head: dict) -> dict:
    keys = sorted(set(base["file_hashes"]) | set(head["file_hashes"]))
    files = {
        k: {"base": base["file_hashes"].get(k), "head": head["file_hashes"].get(k)}
        for k in keys
        if base["file_hashes"].get(k) != head["file_hashes"].get(k)
    }
    markers = {
        k: {"base": base["marker_counts"][k], "head": head["marker_counts"][k]}
        for k in MARKER_ATTRS
        if base["marker_counts"][k] != head["marker_counts"][k]
    }
    collect = base["collect"] != head["collect"]
    return {"files": files, "markers": markers, "collect_changed": collect}


def main() -> int:
    base = json.loads(Path(sys.argv[1]).read_text())
    head = dump(Path(sys.argv[2]))
    diff = delta(base, head)
    Path(sys.argv[3]).write_text(json.dumps({"head": head, "delta": diff}, indent=2))
    blocked = bool(diff["files"] or diff["markers"] or diff["collect_changed"])
    return 2 if blocked else 0


if __name__ == "__main__":
    raise SystemExit(main())

Collect the base document on main before the agent runs. Store it as a build artifact. On the patch SHA, run the same script and fail on exit code 2. Do not parse pytest's human summary line as the source of truth; agents can rewrite addopts to hide tests from collection.

git checkout "$BASE_SHA"
python control_plane.py /dev/null . base_plane.json || true
python - <<'PY'
import json, pathlib, sys
# Re-dump base without comparing.
import control_plane
pathlib.Path("base_plane.json").write_text(
    json.dumps(control_plane.dump(pathlib.Path(".")), indent=2)
)
PY

git checkout "$HEAD_SHA"
python control_plane.py base_plane.json . plane_delta.json

Label the collect-only step. It is a fingerprint of the suite, not a substitute for running tests.

Property lane the agent cannot own

Keep behavioral oracles in tests/properties/. CODEOWNERS should require a human reviewer on that directory. CI should refuse the job if the agent commit list includes those files.

# .github/CODEOWNERS
/tests/properties/  @maintainers-of-the-oracle

# fail if the patch touches the property lane or the control plane
git diff --name-only "$BASE_SHA" "$HEAD_SHA" > changed.txt
if grep -E '^(tests/properties/|pytest.ini|pyproject.toml|conftest.py)' changed.txt; then
  echo "oracle or control plane edited in the same patch" >&2
  exit 2
fi

A minimal property is a relation, not a literal. The example below checks that a parser either returns a document whose round-trip bytes are stable or raises a typed error. It does not compare against a golden string the agent can rewrite.

# tests/properties/test_parse_roundtrip.py — proposed
import pytest
from app.parse import ParseError, parse, render

CASES = [
    b"",
    b"{}",
    b"[]",
    b"{" * 32,
    b"\x00\xff",
    b"{" + b"a" * 4096 + b"}",
]

@pytest.mark.property
def test_parse_is_total_and_render_stable():
    for raw in CASES:
        try:
            doc = parse(raw)
        except ParseError:
            continue
        again = parse(render(doc))
        assert again == doc

If you need more cases, generate them outside the merge oracle. A free model endpoint is enough to propose extra inputs from the production diff. A human pastes survivors into CASES. The model does not open a PR against tests/properties/.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access is useful for that proposal step. The free server option is useful for running control_plane.py plus the property lane on a machine that does not carry the author's local pytest plugins. Neither replaces the lock on who may edit the oracle.

Decision table

The table is the policy. Green JUnit XML is not.

Flake freeze that cannot ride with the patch

A freeze is a dated witness, not a skip. Put it in tests/freezes/ as its own commit. Refuse combination diffs.

{
  "test_id": "tests/test_cache.py::test_evicts_under_contention",
  "first_seen_sha": "a1b2c3d",
  "property_lane_sha": "e4f5a6b",
  "control_plane_sha": "c7d8e9f",
  "replays": 12,
  "fail_count": 3,
  "seed": "PYTHONHASHSEED=0",
  "owner": "oncall-cache",
  "expires_on": "2026-10-02",
  "reason": "ordering under contention; not a skip"
}

Load witnesses in a plugin that xfails by node id. Do not let the plugin read witnesses added in the current SHA if src/ also changed. That check is one git diff-tree call.

SRC_CHANGED=$(git diff --name-only "$BASE_SHA" "$HEAD_SHA" -- src/)
FREEZE_CHANGED=$(git diff --name-only "$BASE_SHA" "$HEAD_SHA" -- tests/freezes/)
if [ -n "$SRC_CHANGED" ] && [ -n "$FREEZE_CHANGED" ]; then
  echo "freeze witness cannot land with production code" >&2
  exit 2
fi

Twelve replays in the witness is a local convention, not a measured reliability of the suite. Record the seed and the property-lane SHA so a later change to the oracle cannot silently inherit the freeze.

Numbered merge workflow

1. Snapshot control_plane.dump on main. Store the JSON next to the CI logs.
2. Apply the agent patch on a clean branch. Forbid writes under tests/properties/, tests/freezes/, and the control-plane globs via a pre-receive hook or a CI name-status check.
3. Recompute the control-plane document on head. Block on file, marker, or collect deltas.
4. Run the property lane with plugins disabled except those pinned in the lockfile.
5. Run the rest of the suite. Compare skip, xfail, fail, and timeout totals to base. Any rise is a failed test.
6. If a single node id flickers and steps 3–5 are clean, open a freeze-only PR with the witness JSON. Do not bundle it.
7. Expire the witness on the date in the file. A freeze without an owner or expiry is a skip.

Step 4 is the reason a remote runner helps. Local conftest plugins, IDE pytest wrappers, and leftover PYTEST_ADDOPTS are control-plane leaks. The free server option is one way to get a tree that only has what CI has. A self-hosted runner with a stripped image is equivalent.

Limits

The AST walk under-counts dynamic markers (pytest.skip inside a helper, import pytest as pt). Pair it with the collect fingerprint. Collect-only still misses tests skipped at runtime inside fixtures.

The property lane does not prove functional completeness. It proves a few relations still hold. If the domain has no typed error and no round-trip, write a different relation. Do not replace it with a literal snapshot the agent can update.

Timeouts that fall are not automatically good. A patch that deletes the slow test also drops the timeout. The name-status check on tests/ plus the collect fingerprint is what catches that, not the timeout number alone.

This workflow assumes pytest. Translate the inventory if the suite is Go testing, JUnit, or a custom harness. The policy is the same: configuration, skips, and oracles are not owned by the patch author.

Who should not use this

Do not install this gate on a repository whose humans routinely land skips and production code together without review. The gate will fail those commits too. That is intended, but it is not a migration plan.

Do not use it as evidence that an agent is safe for unreviewed main. The gate removes one cheat. It does not inspect patch quality, licensing, or secret leakage.

Do not freeze tests that fail the property lane. A red property is a product defect. A freeze is only for a node id whose control plane is clean and whose oracle still holds.

If the team cannot pin a Python version and a plugin set, stop before step 1. Control-plane diffs against a moving runtime are noise.

The merge question is narrow. Did skip, xfail, timeout, or config move. Did the locked properties hold. If both answers are acceptable, review the production diff as code. If either is not, the color of the job is not data.