Build a Fail-Closed Evidence Binder for Free-Model Jobs

The goal was rude and small. Block real files until evidence exists on disk. A zero exit from the CLI is not proof.

Where is the proof this run was reviewed? A README checklist would not answer it. I wanted files on disk, or a hard stop.

The goal, not the vibe

This week's noise is familiar. Vibe coding is not the actual crime. Calling that output engineering is the trap.

I am a solo builder with a cheap canary. I need receipts before real input. Can a folder stop me from lying?

That is the whole experiment today. Forty-five minutes on the clock. Zero dollars if the model path stays free.

Abandon the idea at one point only. If five files cannot be produced, stop. Do not negotiate with missing evidence.

What the binder is

The binder is one directory per run. One run id. Five required artifacts. A scanner exits non-zero when anything is missing.

No scanner pass, no real input. Ever. Why five files, not twelve? Four still felt like vibes. Six became homework.

evidence/<run_id>/
  input.fixture.json
  output.sample.json
  shape.ok
  review.stamp
  rollback.sh

That is the contract on disk. Empty files do not count as proof. The scanner checks size and a few headers.

Who this is for

You run a tiny AI CLI at night. You might use a free model lane. You might park a canary on a free server.

You still need to prove the run happened. Who should skip this binder entirely? Teams with a real platform should skip it.

Skip it for production auth paths too. Skip it if you need a formal audit packet. This is a solo brake, not a compliance program.

Where the free path fits

I needed a place to burn canary runs. Not a lecture. A cheap, disposable lane.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project with free model access. It also offers a free server option for experiments. I treat both as a canary lane, never as production.

I will not invent model names in this post. I will not invent token ceilings either. Those numbers go stale faster than READMEs.

Check the project before you plan capacity. The binder does not care which model wrote the sample. It cares that you kept the sample at all.

Fail-closed decision table

Copy this table into the repo. If a cell fails, real input stays blocked.

Ask the obvious question here. Would you enable real files if any row is red? I would not. That is the whole product.

Numbered build

1. Make the run id boring

Do not use latest as a folder name. Latest is how evidence vanishes overnight.

RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)-canary"
BINDER="evidence/${RUN_ID}"
mkdir -p "$BINDER"
echo "$RUN_ID"

If this print is empty, stop immediately. You have no identity for the run. Would you debug a nameless log later?

2. Freeze an input fixture

Real input waits behind the scanner. The fixture is fake, tiny, and copyable. That is the point of a canary.

cat > "$BINDER/input.fixture.json" <<'EOF'
{
  "ticket_id": "T-1042",
  "title": "CLI exits 0 on empty stdin",
  "body": "Repro: echo -n '' | tool run",
  "expect_exit": 2
}
EOF

Does this fixture match production shape? If not, the binder is already lying. Fix the shape before you touch a model.

3. Capture one output sample

Run the worker against the fixture only. Save the raw bytes. Keep stderr nearby.

python worker.py --input "$BINDER/input.fixture.json" \
  > "$BINDER/output.sample.json" 2>"$BINDER/output.stderr.log" || true

Zero bytes means the model went quiet. Quiet is not success on a canary. The scanner should fail that file.

Do not pretty-print away the failure. Raw JSON is the receipt. Pretty JSON is a story you tell later.

4. Write a shape gate

I do not parse vibes from the model. I parse keys. Missing keys mean exit 2.

Labeled example below, not a hosted service. Copy it. Do not expand it yet.

# shape_gate.py — example gate, not a service
import json, sys, pathlib

p = pathlib.Path(sys.argv[1])
data = json.loads(p.read_text())
required = {"ticket_id", "decision", "reason"}
missing = required - set(data)
if missing:
    print("shape fail:", sorted(missing))
    sys.exit(2)
pathlib.Path(sys.argv[2]).write_text("ok\n")

python shape_gate.py "$BINDER/output.sample.json" "$BINDER/shape.ok"

No shape.ok means the chain stops. Good. Feeling lucky is not a gate.

5. Human review is a file

A chat thumbs-up is not evidence. Write a stamp with your name on it. Would you ship without that name?

cat > "$BINDER/review.stamp" <<EOF
reviewer: sam
run_id: ${RUN_ID}
verdict: pass
notes: empty-stdin path returns decision=reject
time_utc: $(date -u +%Y-%m-%dT%H:%M:%SZ)
EOF

If you cannot write an honest verdict, stop. That is an abandon criterion, not a delay. Do not backfill the stamp tomorrow.

6. Rollback must be executable

A paragraph in a doc is hope. A script is a plan you can run. One command. No extra ceremony.

cat > "$BINDER/rollback.sh" <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
# Fail closed: never point real files at the worker.
ln -sfn /dev/null .accept_real_input
echo "real input disabled"
EOF
chmod +x "$BINDER/rollback.sh"

If rollback is not executable, fail closed. Can you run it without reading a wiki? That is the only rollback test that counts.

The scanner

This is the artifact. Copy it. Run it. Watch it fail on purpose.

Proposal code, not a production service. If it passes a missing stamp, throw it away.

#!/usr/bin/env python3
"""Fail-closed evidence binder scanner. Example only."""
from pathlib import Path
import sys

REQUIRED = {
    "input.fixture.json": 40,
    "output.sample.json": 20,
    "shape.ok": 2,
    "review.stamp": 20,
    "rollback.sh": 40,
}

def scan(binder: Path) -> list[str]:
    errors = []
    if not binder.is_dir():
        return [f"missing binder: {binder}"]
    for name, min_bytes in REQUIRED.items():
        path = binder / name
        if not path.is_file():
            errors.append(f"missing {name}")
            continue
        size = path.stat().st_size
        if size < min_bytes:
            errors.append(f"too small {name}: {size}B")
    stamp = binder / "review.stamp"
    if stamp.is_file():
        text = stamp.read_text(errors="replace")
        if "verdict:" not in text or "reviewer:" not in text:
            errors.append("review.stamp missing verdict or reviewer")
    rollback = binder / "rollback.sh"
    if rollback.is_file() and not (rollback.stat().st_mode & 0o111):
        errors.append("rollback.sh is not executable")
    return errors

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("usage: scan_binder.py evidence/<run_id>", file=sys.stderr)
        sys.exit(2)
    errs = scan(Path(sys.argv[1]))
    if errs:
        print("FAIL-CLOSED")
        for e in errs:
            print("-", e)
        sys.exit(2)
    print("PASS", sys.argv[1])

Wire it before any real input path. No pass, no files. That rule is the product.

python scan_binder.py "$BINDER" || { echo "no real input"; exit 2; }

Failure fixtures

Delete the stamp. Run the scanner. Confirm exit code 2. Do it today.

rm -f "$BINDER/review.stamp"
python scan_binder.py "$BINDER"; echo exit:$?

Expected output looks like this.

FAIL-CLOSED
- missing review.stamp

Did it print PASS instead? Then the scanner is the bug. Fix that before you add features.

Second fixture: poison the output file. Restore the stamp first, then empty the sample.

: > "$BINDER/output.sample.json"
python scan_binder.py "$BINDER"; echo exit:$?

Too-small files must fail closed. Quiet models produce quiet files. The binder should yell at you.

Time, cost, rollback, abandon

Time box: forty-five minutes. If the scanner is not fail-closed, stop. Do not extend the clock for extra gates.

Cost box: stay on the free model lane. Use the free server option for the canary host. If you need a paid GPU to prove a folder, quit.

Rollback command is one file.

"$BINDER/rollback.sh"

Real input stays disabled until a new PASS. Abandon if you cannot write review.stamp. Abandon if output never matches the three keys.

Abandon if you edit the scanner to ignore gaps. That last one is the tell. You are back to theater.

Limitations

This does not measure model quality. It measures whether you kept receipts. Wrong-but-pretty JSON still passes the key check.

It will not replace tests. shape.ok is not a proof of truth. Free model lanes change without email.

Free servers go away. Do not build a business on a canary. Do not store secrets in the binder.

Fixtures should be fake tickets. Never drop customer data into input.fixture.json. Existing files are receipts, not theater.

I will call a README checklist theater. That is the joke. The joke is also the bug.

What I am not claiming

I am not claiming a benchmark. I am not claiming uptime numbers. I am not claiming token ceilings.

If the free path dies tomorrow, keep the scanner. That is the exit I want from this design. The receipts still live in evidence/.

Copy the five files tonight. Break the stamp on purpose. Watch exit 2, then decide on real input.

Need a cheap canary host for that first PASS? MonkeyCode's free model access and free server option can burn the fixture. After that, the binder is yours.

What sixth file would catch your last silent failure, without turning this into homework?