Patches Are Cheap Now. Verification Isn't: A Tiered Evidence Budget for Agent Patches

Generating a candidate patch now costs less than checking it. That inversion is the real change in agent-assisted repositories: the bottleneck moved from authorship to admissible evidence. Below is a concrete budget for that evidence — four tiers, one repeat loop, and a harness you can run today.

Why the ratio flipped

A model that drafts a plausible diff in seconds turns an expensive activity into a cheap one. When free model access is part of the loop, candidates arrive faster than anyone can review them. Two failure modes follow. Teams either rubber-stamp green CI, or they run the full matrix on every candidate and burn hours of wall-clock time.

Neither is a strategy. The first trusts a signal nobody calibrated. The second spends identical effort on a docstring typo and on a change to the payment retry path.

This account's focus is verifying patches written by coding agents, so cheap generation is the default here rather than the exception. MonkeyCode's free model access and free server option are what made a disposable verification runner practical instead of queueing drafts behind a laptop. Both are operator-supplied availability claims whose terms change; check the project page before wiring a process to them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The budget in one sentence

Spend verification effort in proportion to the blast radius of the patch, and let repeated runs — not single runs — decide whether a test result counts as evidence.

Four tiers, ordered by cost. Each tier can only produce three answers: reject, hold, or continue. Pick the total budget from your CI timeout, not from ambition — a budget you cannot finish inside the review window will be abandoned by week two.

Step 1 — Tier 0: static checks, always on

python -m ruff check . && python -m mypy src costs seconds and has no ambiguity. A patch that fails here never reaches a test run. This tier is cheap enough that skipping it is never the right trade.

Step 2 — Tier 1: touched tests

Map changed source files to the tests that cover them, and always include test files the patch touched. Most rejections should land here, at median cost. The mapping below is intentionally dumb — src/billing/retry.py maps to tests/test_retry.py — because a mapping you can read beats an accurate one you cannot debug at 2 a.m.

Step 3 — Tier 2: repeat runs, k of n

Run each touched test n times and require k passes. A test that passes 4 of 5 is not evidence of a fix; it is evidence of flakiness. Quarantine it in a ledger file and record the ledger's hash in the report, so a patch cannot quietly edit its own quarantine list. This is the tier that separates a green run from a trustworthy one.

Step 4 — Tier 3: full suite, budget permitting

Run the whole suite only for candidates that cleared tiers 0-2, and only while the time budget lasts. If the budget runs out, the verdict is hold, never merge. Silence is not a pass.

The harness

Stdlib only, no framework, no service. Drop it at the repo root next to your test directory.

#!/usr/bin/env python3
'''evidence_budget.py - spend verification effort in proportion to patch risk.

Run from the repo root:
    python3 evidence_budget.py --base origin/main --runs 5 --need 5
'''
import argparse, hashlib, json, pathlib, subprocess, time

ROOT = pathlib.Path.cwd()
QUARANTINE = ROOT / '.quarantine.json'


def sh(cmd, timeout=900):
    t0 = time.monotonic()
    try:
        p = subprocess.run(cmd, shell=True, cwd=ROOT, capture_output=True,
                           text=True, timeout=timeout)
        code, tail = p.returncode, (p.stdout + p.stderr)[-600:]
    except subprocess.TimeoutExpired:
        code, tail = 124, 'timeout after %ss' % timeout
    return {'cmd': cmd, 'code': code, 'secs': round(time.monotonic() - t0, 2),
            'tail': tail}


def git(*args):
    p = subprocess.run(['git', *args], cwd=ROOT, capture_output=True, text=True)
    return [line for line in p.stdout.splitlines() if line.strip()]


def tests_for(paths):
    '''src/billing/retry.py -> tests/test_retry.py; test files map to themselves.'''
    picked = set()
    for raw in paths:
        p = pathlib.Path(raw)
        if p.suffix != '.py':
            continue
        if p.name.startswith('test_') and p.exists():
            picked.add(raw)
            continue
        candidate = ROOT / 'tests' / ('test_%s.py' % p.stem)
        if candidate.exists():
            picked.add(str(candidate.relative_to(ROOT)))
    return sorted(picked)


def repeat(tests, runs, need):
    passes = {t: 0 for t in tests}
    for _ in range(runs):
        for t in tests:
            if passes[t] >= need:            # already proven, stop paying
                continue
            if sh('python -m pytest -q %s' % t, timeout=300)['code'] == 0:
                passes[t] += 1
    return {'runs': runs, 'need': need, 'passes': passes,
            'quarantine': sorted(t for t, n in passes.items() if n < need)}


def ledger_digest():
    if not QUARANTINE.exists():
        return 'absent'
    return hashlib.sha256(QUARANTINE.read_bytes()).hexdigest()[:12]


def verdict(report):
    t = report['tiers']
    if t['static']['code'] != 0:
        return 'reject'
    if 'touched' in t and t['touched']['code'] != 0:
        return 'reject'
    if report.get('repeat', {}).get('quarantine'):
        return 'hold-flaky'
    if 'full' not in t:
        return 'hold-budget'
    if t['full']['code'] != 0:
        return 'hold-baseline'
    return 'merge-candidate'


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument('--base', default='origin/main')
    ap.add_argument('--runs', type=int, default=5)
    ap.add_argument('--need', type=int, default=5)
    ap.add_argument('--max-seconds', type=int, default=1800)
    args = ap.parse_args()

    changed = git('diff', '--name-only', '%s...HEAD' % args.base)
    touched = tests_for(changed)
    report = {'base': args.base, 'changed': changed, 'touched': touched,
              'ledger': ledger_digest(), 'tiers': {}}
    spent = 0.0

    report['tiers']['static'] = sh('python -m ruff check . && python -m mypy src')
    spent += report['tiers']['static']['secs']

    if touched and spent < args.max_seconds:
        report['tiers']['touched'] = sh('python -m pytest -q ' + ' '.join(touched))
        spent += report['tiers']['touched']['secs']

    if report['tiers'].get('touched', {}).get('code') == 0:
        t0 = time.monotonic()
        report['repeat'] = repeat(touched, args.runs, args.need)
        spent += time.monotonic() - t0

    clean = not report.get('repeat', {}).get('quarantine')
    if clean and spent < args.max_seconds:
        report['tiers']['full'] = sh('python -m pytest -q', timeout=args.max_seconds)
        spent += report['tiers']['full']['secs']

    report['spent_secs'] = round(spent, 2)
    report['verdict'] = verdict(report)
    print(json.dumps(report, indent=2))
    return 0 if report['verdict'] == 'merge-candidate' else 1


if __name__ == '__main__':
    raise SystemExit(main())

Reading the report

{
  "base": "origin/main",
  "changed": ["src/billing/retry.py", "tests/test_retry.py"],
  "touched": ["tests/test_retry.py"],
  "ledger": "9f1c0a3b7d21",
  "tiers": {
    "static": {"code": 0, "secs": 6.4},
    "touched": {"code": 0, "secs": 11.2},
    "full": {"code": 0, "secs": 214.9}
  },
  "repeat": {"runs": 5, "need": 5, "passes": {"tests/test_retry.py": 5}, "quarantine": []},
  "spent_secs": 232.5,
  "verdict": "merge-candidate"
}

The numbers above are illustrative; your seconds will differ. The field names are the contract your CI reads, so keep them stable.

Two rules that keep the table honest

1. Baseline before blame. A failing full suite is not proof the patch broke anything. Run it on origin/main first; if it fails there, the verdict is hold-baseline and the pre-existing failure becomes its own ticket.
2. Quarantine is a diff, not a deletion. Flaky tests stay in the repository, listed in .quarantine.json, and the report carries that file's hash. A patch that grows the list is visible in review instead of invisible in a retry.

What this does not do

It does not detect a weakened oracle. If a patch edits both the assertion and the code, every tier here can go green while the test stops meaning anything. Pair it with mutation or property checks on the touched modules; that concern needs its own gate.

Repeated runs sample nondeterminism, they do not explain it. A test that depends on wall-clock time or an external socket will still pass now and fail in CI tomorrow. The repeat tier buys you a probability, not a diagnosis.

The file-to-test convention misses indirect coverage. In a repository where one test file exercises six modules, selection belongs on coverage data, not on filename stems.

And cost is linear in k: five runs is five times the test time for the touched subset. That arithmetic only works while the subset stays small.

Who should not use this

• Repositories whose entire suite finishes in under a minute: run everything, always, and skip the tiers entirely.
• Teams without a stable base branch or an agreed quarantine policy: the ledger will drift into a graveyard nobody reviews.
• Patches that change test infrastructure itself: the harness assumes the tests are trusted input.
• Anyone who needs a merge decision in under a minute: drop tiers 2-3 and accept hold as the normal outcome.

If you want to try the pattern without provisioning infrastructure, MonkeyCode's free model access and free server option are what this workflow was exercised against here — read the project page for current terms, then point the harness at a throwaway checkout and see which tier actually catches your bugs.