A Skipped Test Is a Withdrawal: Make Agent Patches Deposit a Property Before You Freeze a Flake

The cheapest way to make an agent patch go green is to skip a test. Most gates allow that skip with no compensating deposit, and then let the skip follow the patch into the next patch in the merge queue. Two weeks later nobody can tell whether the suite shrank because the code got safer or because eleven freezes quietly accumulated.

This post proposes a freeze ledger with four blocking rules: a bounded expiry window, a named owner, a compensating property that must actually run against the patch, and a hard ban on inheriting skips across patch boundaries. All four are enforced by a script short enough to read in one pass. The artifact is a policy file plus roughly sixty lines of Python, not a testing framework.

1. The two ways a freeze lies

A freeze is a claim that the skipped test was not carrying unique coverage. That claim fails silently in two specific ways.

1. Inheritance. In a merge queue, patch B's gate reuses the artifact from patch A's run. A's skip set rides along, so B is validated against a suite that neither B's author nor B's reviewer chose. The green badge is real; the denominator behind it is not.
2. Withdrawal without deposit. A skipped test stops paying rent. If nothing else asserts its invariant, the patch covers strictly less than the previous commit while reporting the same status. Nothing in a standard CI log distinguishes this from a clean run.

Neither failure is loud. Both are visible in one artifact you are probably not writing: the exact set of tests the gate skipped, tied to the commit that skipped them.

2. Step 1: make the freeze a reviewed record, not an inline decorator

An inline @pytest.mark.skip lives in the test file, far from the diff a reviewer is reading. Move the decision into one file that changes as its own diff.

{
  "schema": 1,
  "max_window_days": 14,
  "freezes": [
    {
      "test_id": "tests/test_billing.py::test_proration_rounding",
      "reason": "order-dependent failure in full suite; passes in isolation",
      "owner": "finley",
      "opened": "2026-10-12",
      "expires": "2026-10-26",
      "invariant": "proration amount never decreases when the period lengthens",
      "compensating_property": "tests/props/test_proration_props.py::test_monotonic_in_period",
      "reused_property": false
    }
  ]
}

Six of those fields are boring on purpose. The two that matter are invariant, written in words, and compensating_property, which must be a real node id. An agent patch that wants to skip a test now shows up in review as a ledger diff plus a property file. That is a reviewable claim; a decorator buried in line 214 is not.

The 14-day window is a policy choice, not a discovered constant. Pick a number your team can staff, then let expiry force re-attestation.

3. Step 2: validate shape, owner, and window

The first gate is pure bookkeeping. It rejects freezes that cannot be reviewed because nobody owns them, or that have already outlived their window.

# audit_freezes.py (reference implementation, not benchmarked)
import datetime, json, pathlib, sys

FIELDS = ('test_id', 'reason', 'owner', 'opened', 'expires',
          'invariant', 'compensating_property')

def load(path):
    return json.loads(pathlib.Path(path).read_text())['freezes']

def shape_problems(entry, today, max_window):
    problems = [f'missing {f}' for f in FIELDS if not entry.get(f)]
    opened = datetime.date.fromisoformat(entry['opened'])
    expires = datetime.date.fromisoformat(entry['expires'])
    if expires < today:
        problems.append('expired')
    if (expires - opened).days > max_window:
        problems.append(f'window {(expires - opened).days}d exceeds {max_window}d')
    return problems

Run it with an injected date so the check is deterministic in CI:

python audit_freezes.py --ledger gate/freeze_ledger.json --today 2026-10-18

An expired freeze should fail the build, not emit a warning. A warning on a skipped test is the same as no gate at all, because the merge button stays green.

4. Step 3: require a deposit that actually runs

Here is where most freeze policies stop being useful. A compensating_property string proves nothing unless the property ran in this patch's gate and consumed more than one input. Two automated checks handle that.

First, the property must be collected and it must appear in the patch diff:

import subprocess, sys

def collected_ids(root='.'):
    out = subprocess.run(
        [sys.executable, '-m', 'pytest', '--collect-only', '-q', '--no-header'],
        capture_output=True, text=True, cwd=root,
    )
    return {ln.strip() for ln in out.stdout.splitlines() if '::' in ln}

def changed_files(base='origin/main'):
    out = subprocess.run(
        ['git', 'diff', '--name-only', f'{base}...HEAD'],
        capture_output=True, text=True, check=True,
    )
    return {ln.strip() for ln in out.stdout.splitlines() if ln.strip()}

def deposit_problems(entry, ids, changed):
    pid = entry['compensating_property']
    problems = []
    if pid not in ids:
        problems.append(f'property not collected: {pid}')
    prop_file = pid.split('::')[0]
    if prop_file not in changed and not entry.get('reused_property'):
        problems.append('deposit does not appear in the patch diff')
    return problems

Second, the property must exercise a non-degenerate input space. Count distinct inputs per session and fail below a threshold you pick explicitly:

# tests/props/conftest.py
import json, pathlib

_seen = set()

def record(value):
    _seen.add(repr(value))

def pytest_sessionfinish(session, exitstatus):
    out = pathlib.Path('gate/prop_samples.json')
    out.parent.mkdir(exist_ok=True)
    out.write_text(json.dumps({'distinct': len(_seen)}))

python audit_freezes.py --check-deposits \
  --ledger gate/freeze_ledger.json \
  --samples gate/prop_samples.json \
  --min-distinct 8

A property that ran once, on one hardcoded tuple, is a renamed example test. Set --min-distinct low, but set it.

5. Step 4: refuse inherited skips in the merge queue

Each gate run writes the skips it actually applied. The queue script then compares parent and child sets and refuses anything the child did not re-derive from the ledger at its own commit.

import json, pathlib, sys

def freeze_ids(ledger_path):
    return {e['test_id'] for e in json.loads(pathlib.Path(ledger_path).read_text())['freezes']}

def refuse_inherited(parent_skips, child_derived, allow=()):
    inherited = set(parent_skips) - set(child_derived) - set(allow)
    if inherited:
        sys.exit(f'child gate skipped without re-attestation: {sorted(inherited)}')

The allow set exists for the one legitimate case: a freeze pinned to a release branch with a named approver. Everything else must be re-derived, which costs one pytest --collect-only per queued patch.

The ledger is applied by one hook, and the hook records what it did:

# conftest.py
def pytest_collection_modifyitems(config, items):
    if not config.getoption('--apply-freezes'):
        return
    frozen = freeze_ids('gate/freeze_ledger.json')
    applied = []
    for item in items:
        if item.nodeid in frozen:
            item.add_marker(pytest.mark.skip(reason='frozen: see gate/freeze_ledger.json'))
            applied.append(item.nodeid)
    pathlib.Path('gate/skipped.json').write_text(json.dumps(sorted(applied)))

6. Decision table

7. Where the agent, and the hosted run, fit

The drafting step is where a model helps. Give it the invariant text and the frozen test, and ask for three candidate properties plus the one input class it would not generate. It will produce plausible candidates; it will not tell you which are weak, so the distinct-input counter and the diff check stay yours.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator reports that MonkeyCode offers free model access and a free server option. In this workflow the model side is the candidate-property drafter, and the server side is where you would run the repeated property checks so the gate does not depend on one laptop. Measure the actual limits yourself before making either a dependency; nothing here assumes a quota, hardware spec, or duration, and I have not benchmarked them.

8. Limitations and who should not adopt this

• The ledger needs a merge queue or at least a shared branch to be meaningful. A solo repo with direct pushes gets the bookkeeping cost without the inheritance check.
• prop_samples.json counts distinct repr values. That catches the single-input property, not a weak generator that produces near-duplicates.
• The property-file-in-diff rule assumes one property per invariant. Splitting an invariant across two files needs a manual note in the ledger.
• If your flakiness comes from shared staging infrastructure, freezing just relocates the flake. Fix the lane or move the test to a nightly run.
• Teams that cannot staff a two-week expiry review should delete the flaky test outright and write the property. A freeze nobody revisits is worse than a deletion, because deletion is honest.

9. What to measure after a week

Three numbers, all cheap: how many existing freezes fail the deposit check, the median age of open freezes, and how many inherited skips the queue script refused. I am not going to invent those numbers for you — they depend entirely on how much debt your suite is already carrying.

Start with one repository and one rule: no freeze without a property that ran. Everything else in this post is enforcement around that sentence.