The Test Oracle Stays Pinned: A When-Not-To Field Guide

The payment service sat on a green pipeline at noon. Forty new tests had landed from an overnight agent. Every assertion matched a fixture the agent also wrote. Staging accepted the card charge while production rejected it. The mock had never seen the real processor's 409 body. A free lane had authored both the patch and the proof.

This guide treats that pattern as a refuse case. Generated tests may explore a change under review. They must not certify that change for merge.

The shared-bias failure

An agent on a cheap lane writes product code. The same agent then writes the blessing tests. Continuous integration reports only a clean green run. Reviewers often trust that badge without reading asserts. The oracle and the implementation share one bias.

Free-tier endpoints drop context without a warning. They retry later with shorter and weaker prompts. They skip edge cases that look expensive to sample. None of that drift shows up in a passing badge.

The merge queue then ships a self-checked change. On-call inherits the miss during the next incident.

What counts as a test oracle

A test file is not an oracle by itself. An oracle is the bit that decides pass or fail. That bit must come from a pinned external source.

Treat the following artifacts as real oracles:

• OpenAPI or protobuf snapshots already checked into git
• HTTP fixtures recorded from a named staging host
• Golden JSON with a human Signed-off-by trailer
• Contract tests compiled from a locked schema file
• Manual acceptance checks owned by a named reviewer

Treat the following artifacts as untrusted input:

• Asserts an agent invented inside the same product diff
• Mocks the agent derived from its own new patch
• Coverage totals produced only on a free runner
• Review comments from a free model with merge rights
• Snapshots regenerated without a human fixture diff

Red flags: when not to use a free lane

Teams refuse a free lane when any rule below matches.

1. The agent authors tests and production code together.
2. The suite has no fixture sourced outside the model.
3. Failures retry until the model rewrites the assert.
4. The runner can reach deploy keys or release signing.
5. The change touches money, identity, or deletion paths.
6. Coverage is the only merge requirement on the branch.
7. No human is named as oracle owner on the pull request.
8. The free server also stores secrets or recorded payloads.

A free model can draft a test sketch. A free server can run that sketch in isolation. Neither should own the bit that opens the merge queue.

Teams sometimes park exploratory agents on MonkeyCode. The open-source project offers free model access and a free server option for disposable scratch work.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Those options help when the task is throwaway exploration. They do not help when the task is certification. Keep any free lane, including that one, off the oracle path.

Decision table

Teams apply this table before generated tests touch main.

The table is a refuse map, not a permission map. Ambiguous rows stay in the No column.

Artifact: a fail-closed oracle gate

The workflow below is a labeled, unexecuted example. Operators should try it on a throwaway branch first. It does not claim production metrics.

Pin owners beside the fixtures

# .oracle-owners
# format: glob<TAB>owner<TAB>source
contracts/*.openapi.json    payments-oncall    git-pin
fixtures/http/*.json        payments-oncall    staging-record
tests/oracle/*.py           qa-lead            human-assert

Oracle files stay small and boring. Product tests may import them. Product tests may not rewrite them in the same commit.

Reject mixed authorship

# oracle_gate.py — proposal / unexecuted example
from __future__ import annotations

import os
import subprocess
import sys
from pathlib import Path

ORACLE_OWNERS = Path(".oracle-owners")
SCRATCH_PREFIX = "tests/scratch/"
MIXED_EXIT = 12


def git_lines(diff_range: str) -> list[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", diff_range],
        text=True,
    )
    return [line.strip() for line in out.splitlines() if line.strip()]


def load_oracles() -> list[str]:
    globs: list[str] = []
    for raw in ORACLE_OWNERS.read_text().splitlines():
        line = raw.strip()
        if not line or line.startswith("#"):
            continue
        globs.append(line.split()[0])
    return globs


def matches(path: str, glob_pat: str) -> bool:
    return Path(path).match(glob_pat)


def main() -> int:
    lane = os.environ.get("AGENT_LANE", "unknown")
    diff_range = os.environ.get("ORACLE_DIFF", "origin/main...HEAD")
    changed = git_lines(diff_range)
    oracles = load_oracles()

    oracle_hits = [
        path for path in changed
        if any(matches(path, g) for g in oracles)
    ]
    product_hits = [
        path for path in changed
        if path.endswith((".py", ".ts", ".go"))
        and path not in oracle_hits
        and not path.startswith(SCRATCH_PREFIX)
    ]

    print(f"lane={lane} oracles={oracle_hits} product={product_hits}")

    if lane == "free" and oracle_hits:
        print("refuse: free lane touched an oracle file")
        return MIXED_EXIT

    if oracle_hits and product_hits:
        print("refuse: mixed oracle and product authorship")
        return MIXED_EXIT

    if lane == "free" and product_hits:
        unpinned_tests = [
            path for path in product_hits
            if "/test" in path or path.startswith("tests/")
        ]
        if unpinned_tests:
            print(f"refuse: free lane wrote tests {unpinned_tests}")
            return MIXED_EXIT

    return 0


if __name__ == "__main__":
    sys.exit(main())

The script fails closed on mixed diffs. A free lane may still write files under tests/scratch/. Those files never gate merge.

Wire the gate in CI

# .github/workflows/oracle-gate.yml — proposal
name: oracle-gate
on:
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  refuse-mixed-oracle:
    runs-on: ubuntu-latest
    env:
      AGENT_LANE: ${{ vars.AGENT_LANE }}
      ORACLE_DIFF: origin/${{ github.base_ref }}...HEAD
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Fail if a free lane owns the oracle
        run: python3 oracle_gate.py

vars.AGENT_LANE stays a human-set variable. An agent must not write that variable. A missing value should be treated as unknown and refused on oracle files.

Record fixtures outside the model

# record_fixture.sh — proposal / unexecuted example
set -euo pipefail
HOST="${STAGING_HOST:?set STAGING_HOST}"
OUT="fixtures/http/charge-409.json"

curl -sS -D - -o /tmp/body.json \
  -H "Authorization: Bearer ${STAGING_TOKEN}" \
  -H "Content-Type: application/json" \
  "${HOST}/v1/charges" \
  -d '{"amount":0,"currency":"usd"}' \
  > /tmp/headers.txt

python3 - <<'PY'
from pathlib import Path
import json
headers = Path("/tmp/headers.txt").read_text().splitlines()
status = headers[0]
body = json.loads(Path("/tmp/body.json").read_text())
Path("fixtures/http/charge-409.json").write_text(
    json.dumps({"status": status, "body": body}, indent=2) + "\n"
)
PY

git add fixtures/http/charge-409.json
echo "Recorded ${OUT}. A human must commit this file."

The recorder talks to staging, not to a model. The agent may later write a test that reads the file. The agent may not edit the file in that same change.

Keep scratch tests quarantined

# tests/oracle/test_charge_conflict.py — human-owned oracle import
import json
from pathlib import Path

FIXTURE = Path("fixtures/http/charge-409.json")


def test_zero_amount_matches_recorded_409():
    recorded = json.loads(FIXTURE.read_text())
    assert "409" in recorded["status"]
    assert recorded["body"].get("code") == "amount_zero"

# tests/scratch/test_agent_sketch.py — never a merge gate
def test_sketch_only():
    # Agent sandbox. CI must not require this file.
    assert True

Scratch files stay optional. Oracle files stay required. Mixing them is the bug.

Better alternatives

Teams that still want agent help keep a split workflow.

• A free lane drafts tests under tests/scratch/ only.
• A human promotes one sketch into tests/oracle/ after reading it.
• A recorded fixture or locked schema remains the pass/fail bit.
• A second, pinned model may review prose, never fixtures.
• Property tests use explicit generators checked in by a person.
• Merge queues require the oracle gate, not coverage alone.

The split costs a review cycle. That cycle is cheaper than a bad charge path.

Exit criteria

Teams leave free-lane test generation when any exit fires.

1. An incident traces to an agent-written assert on main.
2. Oracle files and product files share more than one commit.
3. The free runner has ever seen a deploy key or customer payload.
4. Retry loops have rewritten asserts to match broken code.
5. Reviewers can no longer name the human oracle owner.
6. Coverage rises while contract fixtures stay stale for a week.

After an exit, scratch tests remain allowed. Merge tests do not return to the free lane without a new pin.

Limitations of this gate

The script only sees git paths. It cannot see prompt injection inside a fixture. It cannot prove a staging record is current. It cannot stop a human from rubber-stamping a bad golden file.

Lane labels are only as honest as CI variables. A mis-set AGENT_LANE defeats the refuse path. Teams still need a CODEOWNERS file on contracts/ and fixtures/.

The gate also ignores test quality. A human-owned assert can still be wrong. Pinning authorship is necessary. It is not sufficient.

Who should not use this approach

This field guide is a refuse map for agent-authored tests. It is not a license to skip tests.

Skip a free-lane test workflow entirely when:

• The repo ships payments, health records, or identity changes
• Compliance needs a named evaluator for every release artifact
• The team cannot isolate a no-secret CI worker
• Staging cannot record fixtures without production data
• No person will own .oracle-owners week after week

Those teams keep agents off tests. They write oracles first. They generate code against the oracle, never the reverse.

What this guide is not claiming

This article does not rank vendors. It does not publish latency numbers. It does not claim a free server is safe for secrets. It does not treat generated coverage as evidence.

Vibe-written tests can still teach a design. Calling those tests a merge gate is the error. Engineering keeps the oracle in a pinned file with a named owner.

Scratch agents can stay on a free model and a free server. Certification stays on recorded fixtures, locked schemas, and a fail-closed gate.