Green Tests, Slow Code: Budget the Clock in Your AI C++ Patch Eval Harness

An eval suite can be green and still ship a regression. The patch returns the right values, compiles clean, and clears every golden case — it just does the work about three times slower than the code it replaced. The numbers in this section are synthetic, chosen so the failure mode is easy to see; the gate further down is the real artifact.

The cause is boringly common. A std::map swapped in where a sorted std::vector was enough. A defensive copy added inside a hot callback. A std::string rebuilt per element because the AI model completed a loop that reads better than it runs. Every assertion stays true. Every reviewer nods.

Behavioral golden cases answer one question: does the patch produce the specified output? They say nothing about what that output cost. That is the third silent-regression class in an eval harness, after state leaks that ordering hides and grader drift that rubric canaries catch. This post adds a budget for it.

Why a slower patch survives review

A patch that is correct but slower passes every gate that only inspects results. Code review catches some of it, but review attention is finite and the interesting cases are subtle: a container change buried in a 400-line diff, an extra allocation behind an if that is only true on the hot path.

Benchmarks do not catch it either, in most repos, because they are run manually, weekly, or never. The regression lands between two benchmark runs, and by the time someone profiles, the commit that caused it is three sprints back in the log.

The fix is not a better benchmark. It is a budget that runs on every candidate patch and fails the patch, not the build. Treat wall-clock cost as another graded property, in the same loop that already grades correctness.

The artifact: a ratio budget gate

Two design decisions make this workable on ordinary hardware, including shared servers.

First, budgets are ratios against a baseline build, not absolute milliseconds. A patch that is 15% slower than the code it replaced is a regression regardless of whether the machine is fast or loaded.

Second, baseline and candidate runs are interleaved, not batched. Thermal drift, noisy neighbors, and page-cache state then affect both builds roughly equally.

Step 1 — Express budgets as ratios, not milliseconds

Keep the budgets in a versioned JSON file next to the golden cases. Absolute ceilings stay as a backstop; they catch a patch that is slow in absolute terms even when the baseline was slow too.

{
  "noise_floor_ms": 20,
  "default_k": 5,
  "cases": [
    { "id": "ring_push_pop", "arg": 2000000, "baseline_ratio_max": 1.15, "absolute_ms_max": 400 },
    { "id": "index_build",  "arg": 500000,  "baseline_ratio_max": 1.25 }
  ]
}

A 15% band on a microbenchmark is generous on purpose. Tighten it case by case only after you have measured the spread of repeated runs on the machine that will host the gate.

Step 2 — Build one bench binary, used twice

Baseline and candidate must be compiled from the same source except for the patch, with the same compiler and the same flags. A different -O level turns a performance gate into a toolchain diff.

The bench skeleton below takes a case id and a size, runs the workload, and returns a value so the optimizer cannot delete the work. Internal timing is printed for debugging; the gate re-times from outside, because it needs end-to-end numbers that include process startup consistently on both sides.

// bench.cpp -- one binary, one case per argv; baseline and candidate share it
#include <chrono>
#include <cstdio>
#include <cstdlib>
#include <string_view>

int run_ring_push_pop(long n);
int run_index_build(long n);

int main(int argc, char** argv) {
  if (argc < 3) return 2;
  std::string_view id{argv[1]};
  long n = std::strtol(argv[2], nullptr, 10);

  auto t0 = std::chrono::steady_clock::now();
  int sink = (id == "ring_push_pop") ? run_ring_push_pop(n) : run_index_build(n);
  auto ms = std::chrono::duration<double, std::milli>(
                std::chrono::steady_clock::now() - t0).count();

  std::printf("%s %.3f %d\n", argv[1], ms, sink); // debug output; gate re-times
  return sink == 12345 ? 0 : 0;
}

Build it twice: once at the merge base (bench_base), once with the candidate patch applied (bench_cand).

Step 3 — Interleave runs and compare medians

The driver runs k rounds. Each round measures baseline and candidate back to back, so machine-level drift is shared.

#!/usr/bin/env python3
"""perf_gate.py -- ratio-based performance gate for AI-written C++ patches.

exit 0 = all budgets met, 1 = regression, 2 = inconclusive (below noise floor)
"""
import argparse, json, statistics, subprocess, sys, time
from pathlib import Path


def one_run(binary, case_id, arg):
    t0 = time.perf_counter()
    subprocess.run([binary, case_id, str(arg)], check=True,
                   stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
    return (time.perf_counter() - t0) * 1e3


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--baseline", required=True)
    ap.add_argument("--candidate", required=True)
    ap.add_argument("--budgets", default="perf_budgets.json")
    args = ap.parse_args()

    cfg = json.loads(Path(args.budgets).read_text())
    floor = cfg.get("noise_floor_ms", 20)
    rows, regressed, skipped = [], 0, 0

    for case in cfg["cases"]:
        cid, arg = case["id"], case["arg"]
        k = case.get("k", cfg.get("default_k", 5))
        base, cand = [], []
        for _ in range(k):                      # interleave: drift hits both
            base.append(one_run(args.baseline, cid, arg))
            cand.append(one_run(args.candidate, cid, arg))

        b, c = statistics.median(base), statistics.median(cand)
        if b < floor:
            skipped += 1
            rows.append((cid, b, c, c / b, "SKIP", "below noise floor"))
            continue

        ratio, limit = c / b, case["baseline_ratio_max"]
        abs_fail = "absolute_ms_max" in case and c > case["absolute_ms_max"]
        status = "REGRESS" if (ratio > limit or abs_fail) else "pass"
        regressed += status == "REGRESS"
        rows.append((cid, b, c, ratio, status, f"limit {limit:.2f}"))

    print("| case | base ms | cand ms | ratio | status | note |")
    print("|---|---|---|---|---|---|")
    for cid, b, c, r, s, note in rows:
        print(f"| {cid} | {b:.1f} | {c:.1f} | {r:.2f} | {s} | {note} |")

    if regressed:
        return 1
    return 2 if skipped else 0


if __name__ == "__main__":
    sys.exit(main())

Run it with ./perf_gate.py --baseline ./bench_base --candidate ./bench_cand. Output is a Markdown table that pastes straight into a pull request comment.

Step 4 — Respect the noise floor and the inconclusive state

A case whose baseline median lands under noise_floor_ms is dominated by process startup and scheduling, not by the code under test. Marking it SKIP is more honest than letting a 0.4 ms case produce a confident 2.5× verdict.

The distinct exit code matters. 2 means the gate could not decide, which needs a human or a bigger input size — not a silent pass. If most cases return 2, the workload sizes are too small: raise arg until the baseline clears the floor.

Step 5 — Quarantine regressions, do not delete cases

When a case fails, the patch is held for review; the case stays. Deleting the case because the slow patch was accepted anyway trains the harness to approve whatever it last saw.

Keep a small history file with the case id, the accepted ratio, and the commit that set the new baseline. The next time the same case fails, the reviewer can see whether the budget was renegotiated deliberately or drifted upward one patch at a time.

A representative gate output

The table below is synthetic, from the illustrative slowdown described at the top.

| case          | base ms | cand ms | ratio | status  | note       |
|---------------|---------|---------|-------|---------|------------|
| ring_push_pop | 41.9    | 117.6   | 2.81  | REGRESS | limit 1.15 |
| index_build   | 96.2    | 101.0   | 1.05  | pass    | limit 1.25 |

Two rows, two different messages. One patch would have merged on correctness alone; the gate caught the container substitution before review.

Where free model access and a free server fit

Candidate patches have to come from somewhere, and a budget gate needs a place to run every night. MonkeyCode's free model access supplies the first: generate several candidate patches for the same issue and let the gate decide which ones are merely different and which ones are worse. Its free server option supplies the second: a host for the recurring perf_gate.py run instead of a laptop that sleeps.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Availability terms are the operator's claim, not a benchmark: free model access with a stated token allowance (10M tokens as of this writing), plus a free server option. Quotas, the model roster, and server specifications change, so read them from the live dashboard rather than from any post, including this one. If the free tier's constraints do not fit, check whether self-hosting or a paid runner does before building the workflow around it.

One practical caveat matters more than the rest. A shared or modest server is a noisy measurement host. That is exactly why this gate uses ratios, interleaved rounds, and an absolute ceiling as a backstop instead of trusting a single millisecond reading. The design does not assume quiet hardware; it assumes the noise is shared.

Limitations and who should skip this

Microbenchmarks are not production workloads. A ratio budget proves that a specific loop got slower, not that users will notice. Service-level latency needs service-level measurement.

The gate also misses scaling regressions that only appear at larger inputs. A patch that is 1.05× slower at n = 500000 may be 4× slower at n = 5000000. Add a second size per case and compare growth, not just the ratio, if that class matters to the codebase.

Caching can game the gate as easily as it games tests: a patch that memoizes results looks fast in a repeat-k run. Behavior cases still have to cover the uncached path. Budgets extend the grading surface; they do not replace it.

Skip this approach if the project already has hardware-isolated performance CI — use that instead of rebuilding it. Skip it too if the baseline build is not reproducible, or if the workload takes under a millisecond, because the gate will spend its life returning 2.

A short checklist before wiring it in

1. One bench binary, compiled twice from the same flags and compiler.
2. Ratio budgets in version control, next to the golden cases.
3. Interleaved rounds, median of at least five.
4. An honest SKIP for cases under the noise floor, and a non-zero exit code for it.
5. A budget history file, so a renegotiated limit is a decision rather than an accident.

Correctness gates protect the output. A clock budget protects the cost of producing it — and it is the cheapest place to find out that a green patch made the hot loop three times slower. If the workflow fits, the free tier is a reasonable place to generate the candidate patches this gate will judge.