Inherited Green: Freeze Debt Accounting for Agent Patches

The conclusion first: an agent patch is not verified by the suite passing. It is verified by the assertions that executed in the head branch and did not execute in the base branch. Everything else is borrowed.

Most agent pipelines report one bit: green or red. That bit hides two inflators. Skips inherited from the base branch silently shrink the evidence surface. Property tests that re-draw their randomness on every run make a pass non-reproducible by default. Fix both and the patch review gets shorter, not longer.

This post is about accounting, not about catching flakes. Attribution work — deciding whether a patch caused a flake or merely surfaced one — is a separate gate. Here the question is narrower: how much of this green did the patch actually earn?

1. The failure mode, with a small trace

The following trace is illustrative, not a benchmark.

• Base branch: 412 tests pass, 1 test frozen as FRZ-014 because it fails roughly 1 in 40 CI runs on a wall-clock boundary.
• Agent patch touches src/pricing/rounding.py, adds two tests, suite goes green at 414 executed.
• The delta looks like +2. The riskiest changed line is covered by exactly the frozen test, which did not run.

Nothing here is dishonest. The patch genuinely added two tests. The problem is that "+2" was reported without subtracting what the patch was already missing, and without noticing that the freeze covered the changed file.

So the rule is not "no skips". It is: skips are fine, unaccounted skips are not, and a skip whose blast radius overlaps your diff is a blocker unless a compensating check runs in the same build.

2. Step 1 — make every skip auditable

Add a session-level collector. It records executed and skipped node IDs with no plugin dependency, so it works the same in a container, a laptop, and a hosted runner.

# conftest.py — run with: EVIDENCE_OUT=head.json pytest -q
import json
import os
import pathlib

_records = []


def pytest_runtest_logreport(report):
    # setup-phase skips and call-phase results are the only ones we count
    if report.when not in ("setup", "call"):
        return
    if report.when == "call" or report.outcome == "skipped":
        _records.append(
            {
                "nodeid": report.nodeid,
                "when": report.when,
                "outcome": report.outcome,  # passed | failed | skipped
                # xfail reports as "skipped"; keep it separable
                "xfail": getattr(report, "wasxfail", None) is not None,
            }
        )


def pytest_sessionfinish(session, exitstatus):
    path = pathlib.Path(os.environ.get("EVIDENCE_OUT", "evidence.json"))
    path.write_text(
        json.dumps({"exit": int(exitstatus), "records": _records}, indent=2),
        encoding="utf-8",
    )

Run it twice, in the same image digest, against the same dependency lockfile:

git worktree add /tmp/base "$(git merge-base origin/main HEAD)"
(cd /tmp/base && EVIDENCE_OUT=/tmp/base.json pytest -q || true)
EVIDENCE_OUT=/tmp/head.json pytest -q || true

Same image matters more than it sounds. A drifted dependency between the two runs produces a delta that measures the environment, not the patch.

3. Step 2 — compute the evidence delta

The script below answers four questions: how many tests net executed, what skips are new, what previously passed and no longer does, and whether the diff touches a frozen blast radius.

#!/usr/bin/env python3
"""evidence_delta.py <base.json> <head.json> <freeze_ledger.toml> <today ISO>

Adjust the git ref below to your integration branch."""
import json
import subprocess
import sys

try:
    import tomllib  # Python 3.11+
except ModuleNotFoundError:  # 3.10 and older
    import tomli as tomllib


def load(path):
    with open(path, encoding="utf-8") as fh:
        data = json.load(fh)
    buckets = {"passed": set(), "failed": set(), "skipped": set(), "xfail": set()}
    for r in data["records"]:
        key = "xfail" if r["xfail"] else r["outcome"]
        buckets[key].add(r["nodeid"])
    return {"exit": data["exit"], **buckets}


def frozen_surface(ledger_path, today):
    with open(ledger_path, "rb") as fh:
        ledger = tomllib.load(fh)
    return {
        path
        for entry in ledger.get("freeze", [])
        if str(entry["expires"]) >= today  # ISO dates compare lexicographically
        for path in entry["covers"]
    }


if __name__ == "__main__":
    base_path, head_path, ledger_path, today = sys.argv[1:5]
    base, head = load(base_path), load(head_path)

    introduced = head["skipped"] - base["skipped"]
    retired = base["skipped"] - head["skipped"]
    lost = base["passed"] - head["passed"]
    net = len(head["passed"]) - len(base["passed"])

    changed = set(
        subprocess.check_output(
            ["git", "diff", "--name-only", "origin/main...HEAD"], text=True
        ).split()
    )
    hot = changed & frozen_surface(ledger_path, today)

    print(f"net executed delta  : {net:+d}")
    print(f"new skips           : {len(introduced)}")
    print(f"retired skips       : {len(retired)}")
    print(f"previously passing  : {len(lost)} no longer passing")
    print(f"frozen surface hit  : {sorted(hot) or 'none'}")

    problems = []
    if head["exit"] != 0:
        problems.append(f"head suite exit {head['exit']}")
    if introduced:
        problems.append("patch-introduced skips need a ledger entry")
    if lost:
        problems.append("tests that passed on base no longer pass")
    if hot:
        problems.append("diff touches a frozen blast radius; compensating check required")
    for p in problems:
        print(f"BLOCK: {p}")
    sys.exit(1 if problems else 0)

A positive net number is not a pass. It is a number that still has to survive the block list.

4. Step 3 — the ledger that gives a freeze an owner and a deadline

A freeze without a blast radius cannot be matched against a diff, and a freeze without an owner never ends. Fourteen days is a policy choice, not a measurement.

# freeze_ledger.toml
[[freeze]]
id = "FRZ-014"
test = "tests/test_pricing.py::test_rounding_boundary"
covers = ["src/pricing/rounding.py"]        # blast radius, not a guess
owner = "finley"                             # a human, reviewed in the PR
opened = 2026-10-30
expires = 2026-11-13
reason = "wall-clock boundary; fails ~1/40 CI runs, see run 8812"
compensating_check = "tests/property/test_rounding.py::test_fee_monotonic"

Three rules keep the ledger from rotting:

1. covers must be file paths, because file paths are what git diff returns. A test name alone is not matchable.
2. Renewal requires a new reason and evidence that the compensating check advanced. No silent re-freeze chains.
3. Ledger edits are human-reviewed. An agent that can both introduce a skip and write its ledger entry has removed the control entirely.

5. Step 4 — two lanes for property checks

Property tests are the natural compensating check for a frozen example-based test. They are also the easiest way to produce a green that nobody can reproduce, because the generator picks new inputs each run.

Use two lanes over the same properties.

# tests/property/test_rounding.py
import os
from decimal import Decimal

import pytest
from hypothesis import given, settings, strategies as st

MONEY = st.decimals(
    min_value=Decimal("0"),
    max_value=Decimal("100000"),
    places=2,
    allow_nan=False,
    allow_infinity=False,
)

PINNED = settings(derandomize=True, max_examples=300, deadline=None, database=None)
HUNT = settings(max_examples=2000, deadline=None)  # keeps the example database

LANE = os.environ.get("PROPERTY_LANE", "pinned")
CFG = PINNED if LANE == "pinned" else HUNT


@CFG
@given(amount=MONEY)
def test_fee_is_bounded(amount):
    total = apply_fee(amount)  # code under test
    assert total >= amount
    assert total - amount <= MAX_FEE

Why both lanes:

• Pinned lane (derandomize=True, empty example database) reproduces byte-for-byte on every run. It belongs in the patch gate, because a reviewer must be able to re-run it and see the same inputs.
• Hunting lane (unpinned, database enabled) explores. It belongs on a schedule, not on every patch, or it becomes the flake source you were trying to eliminate.

When the hunting lane finds a counterexample, promote it: add a literal @example(Decimal("1234.56")) to the test. The pinned lane then replays it forever. A patch touching fee logic is expected to either add a promoted example or state why the invariant space did not grow. That is corpus growth as a reviewable artifact, not a coverage percentage.

6. Fixtures: hash them, do not read them

Fixture changes are an evidence-surface change, not a code change. Diff them explicitly instead of relying on a reviewer to notice:

git diff --stat origin/main...HEAD -- tests/fixtures tests/factories tests/conftest.py

Any output triggers two requirements: human sign-off and a ledger entry with scope = "fixture". You do not need to judge whether the new fixture is weaker. You need it to be visible.

7. Decision table

Numbers below are illustrative; thresholds are policy, not measurements.

8. Where a hosted agent loop fits

This harness is deliberately boring to run: two pytest invocations in one image plus one script. The expensive part is the patch loop itself, not the accounting.

MonkeyCode's free model access keeps that loop cheap enough to generate candidate patches in volume, and its free server option is one way to run the agent session and both evidence runs in the same hosted environment rather than splitting base and head across a laptop and a CI runner. Both availability claims are operator-supplied: no quotas, model names, hardware, or duration are asserted here, and the harness below is equally valid with a local model or no agent at all.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

9. Limitations, and who should skip this

• Aggregate counts are gameable. A patch can add ten cheap tests and delete one expensive test and score +9. Track the per-file delta, not just the total.
• Pinning removes exploration. The pinned lane alone will never find a new counterexample. If the hunting lane does not actually run on a schedule, the two-lane design is decoration.
• xfail is not skip. The collector separates them for a reason; merging them makes an expected failure look like suppression.
• Deletions are ambiguous. lost fires for obsolete tests too. That is intentional friction, not a bug — it forces a sentence of explanation in the PR.
• This is a pre-filter, not a proof. It does not replace mutation testing or deliberate fault injection. It is the cheap step that tells you whether the expensive step is worth running.

Do not adopt this if your suite is under roughly twenty tests, if you have no flake history to freeze against, or if your CI has no budget for a second lane. In those cases the ledger is ceremony and the delta is noise. For a small patch queue, a plain diff review is honest and faster.

Run the delta over your last twenty agent patches before adding any new tooling. The number that matters is not how many went green — it is how many the block list rejected, and whether you agreed with it.

Code in this article is a sketch: the collector and delta script are written against standard pytest hooks and git diff, but you should expect to adjust the git ref, the ledger path, and the settings profiles to your repository before relying on any exit code.