The Twin Bench Workshop: Proving a Self-Hosted Model Can Take Over

A prototype passes demo on Thursday. On Monday at 02:40 the batch job that feeds the morning report starts collecting retries, and the queue drains at half speed because the hosted endpoint is throttling. The obvious fix is to point the client at a self-hosted server and move on. The non-obvious part is proving that nothing quietly regressed in the three days since anyone last read a raw response.

This is the workshop I would run for that Monday-morning team. It takes ninety minutes, produces one reusable artifact, and — done properly — answers a question that a vibe check cannot: does the second bench pass the same cases as the first, and where does it fail?

Why a second bench is not a diff

Two endpoints that speak the same chat-completions dialect are not interchangeable. Prompt templates get rewritten by the server, stop sequences land differently, JSON modes are optional in practice, and a tokenizer boundary can turn a clean extraction into a trailing explanation. Swapping the base URL is a five-line change. Trusting the result is a different project.

MonkeyCode's free model access and free server option make the cheap half of this genuinely cheap: both benches can exist before lunch, so the workshop can spend its budget on adjudication instead of provisioning. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The ninety minutes

The first ten minutes are framing only — write the swap decision on a whiteboard and leave it there. From minute ten to twenty-five, the room builds a frozen golden set. The next twenty minutes go to the harness, the twenty after that to hard-fail checks, and the last twenty-five to running both benches and writing the verdict card. Nobody is allowed to look at model output during the first forty-five minutes; that is the discipline that keeps the bench honest.

Exercise 1: freeze twelve cases (15 min)

Twelve is the right number for a workshop because everyone can read all twelve out loud. Pull them from the existing test suite and the bug tracker, not from imagination — the cases that bit you in production are the only ones with predictive value. Cover schema-following, numeric extraction, refusal boundaries, one long-context needle, one unicode-and-code-fence case, and at least one deliberately ambiguous prompt.

golden.jsonl, one object per line:

{"id":"amount-01","kind":"number","prompt":"Extract the total from: 'Widgets x 12 @ 103.375 = 1,240.50'","expect":"1240.50"}
{"id":"schema-03","kind":"json","prompt":"Return JSON with keys id and status for ticket 9912, status closed.","required":["id","status"]}

The kind field matters more than it looks. It decides which hard check runs, and it stops the room from arguing about scoring before anyone has collected data.

Exercise 2: one harness, two output files (20 min)

Resist the temptation to write two scripts. One script, two environment variables, two JSONL files — that symmetry is what makes the later diff readable.

import hashlib, json, os, time, urllib.request

BASE = os.environ["BENCH_BASE_URL"]
KEY = os.environ.get("BENCH_API_KEY", "not-needed")
OUT = os.environ["BENCH_OUT"]  # bench-hosted.jsonl or bench-server.jsonl

def call(prompt: str) -> dict:
    body = json.dumps({
        "model": os.environ["BENCH_MODEL"],
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        BASE.rstrip("/") + "/v1/chat/completions",
        data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}"},
    )
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=120) as r:
        payload = json.loads(r.read())
    choice = payload["choices"][0]
    return {
        "text": choice["message"]["content"],
        "finish_reason": choice.get("finish_reason"),
        "latency_ms": round((time.perf_counter() - t0) * 1000),
        "usage": payload.get("usage", {}),
    }

with open(OUT, "w", encoding="utf-8") as sink:
    for line in open("golden.jsonl", encoding="utf-8"):
        case = json.loads(line)
        rec = {"id": case["id"], "kind": case["kind"],
               "prompt_sha": hashlib.sha256(case["prompt"].encode()).hexdigest()[:16]}
        rec.update(call(case["prompt"]))
        sink.write(json.dumps(rec, ensure_ascii=False) + "\n")

Two details earn their keep. The prompt hash proves both benches saw the same input even if someone edits the golden set mid-run. And temperature: 0 is a request, not a guarantee — write that on the whiteboard, because a self-hosted stack may sample differently and the room needs to expect that.

Exercise 3: hard checks before fuzzy ones (20 min)

The most common workshop mistake is reaching for similarity scoring first. A hard check is a veto; fuzzy scoring only ranks cases that already pass. Keep them in separate files so nobody can soften a failure by accident.

import json, re
from decimal import Decimal

def normalize_amount(text: str):
    m = re.search(r"-?\d[\d,]*\.\d{2}", text)
    return str(Decimal(m.group(0).replace(",", ""))) if m else None

def check_number(case, rec):
    got = normalize_amount(rec["text"])
    want = str(Decimal(case["expect"]))
    return ("PASS" if got == want else "FAIL", got, want)

def check_json(case, rec):
    try:
        obj = json.loads(rec["text"])
    except json.JSONDecodeError as e:
        return ("FAIL", f"unparseable: {e.msg}", None)
    missing = [k for k in case["required"] if k not in obj]
    return ("PASS", None, None) if not missing else ("FAIL", f"missing {missing}", None)

The normalizer in check_number is the worked example worth slowing down for. One bench returns 1,240.50 and the other returns 1240.50; compared as strings they look like a regression, compared as Decimal they are the same number. Half the "failures" in a first parity run are bench bugs, not model bugs — so before the room blames the server, it must prove the checker can pass the hosted output.

Then run both benches and diff:

BENCH_BASE_URL=https://hosted.example/v1 BENCH_MODEL=free-tier BENCH_OUT=bench-hosted.jsonl python run_bench.py
BENCH_BASE_URL=http://localhost:8000 BENCH_MODEL=local BENCH_OUT=bench-server.jsonl python run_bench.py
python checks.py bench-hosted.jsonl bench-server.jsonl --fail-fast

Exercise 4: the verdict card (25 min)

Sort every case into one of three buckets and write them down, not into a chat thread.

The card is one page and lives next to the harness: which cases passed on both, which failed on both (pre-existing — do not blame the swap), which failed only after the swap, the p50 latency of each run, and the finish-reason histogram. Anything that only drifted in wording goes to a human reviewer with three examples, because a script that scores prose will score the wrong thing.

Limitations and who should skip this

Twelve cases are a regression net, not a benchmark. They tell you whether your known failure modes survived the swap; they say nothing about capability in general, and a clean parity run is not evidence that one model matches another. Free access and free server capacity are availability claims at a moment in time, so re-run the harness rather than trusting last month's card. Self-hosting also moves operations onto you — cold starts, model download time, and whatever hardware you actually got — and a shared free server is not a place to assume throughput.

Skip this workshop if you need a throughput guarantee today, or if your product output is subjective prose that a human reads anyway; for that, three reviewers and a rubric are cheaper than a harness. Skip it too if the swap is a temporary experiment you will revert on Friday.

For everyone else, the order is the lesson: freeze the cases, build the symmetric harness, let hard checks veto, and only then look at the output. If you want both benches running before you commit to any provisioning, MonkeyCode's free model access and free server option are enough to get from empty repo to two JSONL files in an afternoon.