Case Study: A Verdict Contract That Keeps a Triage Bot Honest When the Model Changes

You can move a small bot to a different model in an afternoon, but only if that bot asks the model for almost nothing. That is the lesson behind the reference design below, and it sits oddly against this week's DEV front page, where several popular AI posts argue about what should sit between a developer and a model. A narrow, testable contract does more for you than a cleverer prompt, because the contract is the part that can fail loudly on your terms.

Background: the endpoint moves before your code does

Model endpoints change faster than the tools that depend on them. A snapshot gets retired, a JSON mode gets stricter or looser, and a workflow that looked stable starts returning prose where your parser expects an object. Most teams only discover this in production, because their single test of the model is the happy-path call they made on the day they wired it up.

This case study is a reference design, not a production report. The project is a small GitHub issue triage bot that labels incoming issues and decides whether a human should look at them. The code below is a working skeleton you can run yourself, and the results section is deliberately shaped as an interpretation table rather than a wall of numbers I did not measure.

Goal: one narrow output contract

The bot's job is deliberately boring, and that is the point. It reads an issue title and body, assigns one label from a fixed list, returns one to three reason codes, and escalates to a human when it is unsure. Nothing downstream of the bot reads free text, so the whole decision path is a single small object you can validate before it touches a queue.

Three constraints shaped the design:

1. The label set must be finite and versioned, because label drift is the failure mode nobody notices.
2. Every decision must carry at least one reason code, so a reviewer can audit the bot without re-reading the model's prose.
3. Any unparseable reply must degrade to a human queue, never to a guessed label.

The verdict contract

Save this as verdict.schema.json. If you cannot write your bot's output this tightly, the model is doing too much work for you.

{
  "type": "object",
  "additionalProperties": false,
  "required": ["label", "confidence", "reasons", "needs_human"],
  "properties": {
    "label": { "enum": ["bug", "docs", "question", "feature", "duplicate", "spam"] },
    "confidence": { "enum": ["low", "medium", "high"] },
    "reasons": {
      "type": "array", "minItems": 1, "maxItems": 3,
      "items": { "enum": ["repro_steps_present", "stack_trace_present", "version_missing", "asks_how_to", "no_actionable_detail", "links_prior_issue"] }
    },
    "needs_human": { "type": "boolean" }
  }
}

Implementation: one call function, one validator, one repair

The most useful structural decision here is that exactly one function knows about HTTP. Everything else in the bot works with a string and a parsed object, so switching providers means editing one file. Keep the transport dumb and let the validation layer carry the personality.

# client.py - the only file that knows what an endpoint looks like
import json, os, urllib.request


def raw_completion(system, user, repair_note=''):
    base = os.environ['LLM_BASE_URL'].rstrip('/')
    payload = json.dumps({
        'model': os.environ['MODEL_ID'],
        'temperature': 0,
        'messages': [
            {'role': 'system', 'content': system},
            {'role': 'user', 'content': user + repair_note},
        ],
    }).encode('utf-8')
    req = urllib.request.Request(
        base + '/chat/completions',
        data=payload,
        headers={'Content-Type': 'application/json',
                 'Authorization': 'Bearer ' + os.environ['LLM_API_KEY']},
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        return json.loads(resp.read())['choices'][0]['message']['content']

If your endpoint speaks a different request shape, change this function and nothing else. That constraint is what makes a swap drill cheap, because the rest of the codebase stays constant while the backend moves underneath it.

Validate first, repair once, then escalate

One repair attempt catches most formatting slips without hiding a model that genuinely does not follow instructions. Two attempts is usually one too many, because a second repair pass doubles the cost of exactly the cases that are already going badly.

# verdict.py
import json
from jsonschema import Draft202012Validator

SCHEMA = json.load(open('verdict.schema.json', encoding='utf-8'))
VALIDATOR = Draft202012Validator(SCHEMA)

SYSTEM = (
    'You classify GitHub issues. Reply with one JSON object and nothing else. '
    'label in {bug, docs, question, feature, duplicate, spam}; '
    'confidence in {low, medium, high}; reasons: 1-3 codes from '
    '{repro_steps_present, stack_trace_present, version_missing, asks_how_to, '
    'no_actionable_detail, links_prior_issue}; needs_human is a boolean.'
)

FALLBACK = {'label': 'question', 'confidence': 'low',
            'reasons': ['no_actionable_detail'], 'needs_human': True}


def parse(raw):
    try:
        obj = json.loads(raw)
    except json.JSONDecodeError:
        return None, 'reply was not JSON'
    errors = sorted(VALIDATOR.iter_errors(obj), key=lambda e: list(e.path))
    if errors:
        return None, 'schema: ' + '; '.join(e.message for e in errors[:3])
    return obj, ''


def classify(issue, call):
    user = issue['title'] + '\n\n' + issue['body']
    obj, why = parse(call(SYSTEM, user))
    if obj:
        return obj, 'first_pass'
    note = '\n\nYour previous reply was rejected (' + why + '). JSON only.'
    obj, why = parse(call(SYSTEM, user, note))
    if obj:
        return obj, 'repaired'
    return FALLBACK, 'fallback'

Note that the fallback is not an error path you hope never runs. It is a first-class outcome that the drill counts, because a rising fallback rate is the earliest honest signal that a candidate model is not compatible with your contract.

The drill is cheap to run because the candidate side does not need to be the endpoint you pay for in production. I point candidate calls at MonkeyCode's free model access and keep the scheduled job small enough for the free server option, so a swap evaluation costs review time instead of budget. MonkeyCode is an open-source project, and the operator advertises a free token allowance alongside that free server option; read both figures off the project page before you plan around them, because free-tier numbers age faster than any code in this article.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The swap drill

A drill is four numbered steps, and none of them need a new tool. The discipline comes from running the same golden set through the old backend and the candidate backend, then comparing outputs row by row instead of eyeballing a few examples.

1. Curate golden.jsonl from issues you already closed, with one id, the human-assigned label, title, and body per line. Aim for at least a few dozen rows and make sure every label in your enum appears.
2. Run the drill against the model currently in production and save the CSV as your baseline.
3. Run the identical drill against the candidate, changing only MODEL_ID and the base URL.
4. Review the disagreements by hand before you look at the aggregate score, because a model can be wrong in a more useful way than your old one.

# drill.py
import csv, json, time
from client import raw_completion
from verdict import classify


def run(golden_path, out_path, call):
    rows = []
    for line in open(golden_path, encoding='utf-8'):
        issue = json.loads(line)
        started = time.monotonic()
        verdict, status = classify(issue, call)
        rows.append({
            'id': issue['id'],
            'expected': issue['label'],
            'got': verdict['label'],
            'agrees': verdict['label'] == issue['label'],
            'status': status,
            'needs_human': verdict['needs_human'],
            'seconds': round(time.monotonic() - started, 2),
        })
    with open(out_path, 'w', newline='', encoding='utf-8') as f:
        writer = csv.DictWriter(f, fieldnames=list(rows[0]))
        writer.writeheader()
        writer.writerows(rows)
    n = len(rows)
    print('rows=' + str(n))
    print('agreement=%.3f' % (sum(r['agrees'] for r in rows) / n))
    print('schema_ok=%.3f' % (sum(r['status'] != 'fallback' for r in rows) / n))
    print('escalated=%.3f' % (sum(r['needs_human'] for r in rows) / n))


if __name__ == '__main__':
    run('golden.jsonl', 'drill.csv', raw_completion)

Wrap the environment plumbing in a script so a drill is one command and not a five-line incantation you will eventually mistype.

#!/usr/bin/env bash
# run_drill.sh
set -euo pipefail
export LLM_BASE_URL=${LLM_BASE_URL:?copy the base URL from the provider docs}
export LLM_API_KEY=${LLM_API_KEY:?set the key for this endpoint}
export MODEL_ID=${MODEL_ID:?set the model id the endpoint documents}
stamp=$(date -u +%Y%m%dT%H%MZ)
python drill.py --golden golden.jsonl --out drill-${stamp}.csv

A decision table you can argue about

Thresholds turn a vague feeling into a reviewable decision. Set them once with your team, then let the table decide instead of whoever is most tired that afternoon.

Running it on the free server option

A scheduled triage job is a good candidate for a free server, because it sleeps most of the day and a slow start costs you minutes rather than requests. Keep the deployment boring: one directory, one cron entry, and logs you can read with tail.

# crontab -e on the free server option
17 3 * * * cd /srv/triage && ./run_drill.sh >> drill.log 2>&1

Do not assume the free server behaves like the box under your desk. Measure startup time, confirm that the job finishes inside whatever window the host gives you, and keep the state in files you can rebuild from scratch. If a run is interrupted, the next run should still produce a complete CSV, which is why the drill writes one file per timestamp instead of appending to a shared one.

Results: what the output actually tells you

I am not going to print agreement scores I did not measure, so treat the table below as the reading guide for your own drill output. The pattern of results matters far more than any single number, and each pattern points at a different fix.

The original bot moved from background to reference design because every swap question turned into the same question: can you test this without rewriting the bot. Once the drill existed, that answer was a CSV comparison a reviewer could audit in ten minutes.

Lessons learned

• A narrow contract is portable; a wide one is a rewrite waiting for a new endpoint. Keep the schema small enough that you can recite it in a review.
• Free tiers are excellent for candidate evaluation and poor for assumptions. Verify allowances and server limits the same day you plan around them.
• One repair pass is a formatting safety net; two passes are a hidden cost multiplier on your worst requests.
• The fallback path deserves tests of its own, because it runs exactly when everything else is misbehaving.
• Golden sets rot, so add a row every time a human overrides the bot, and re-baseline after each promotion.

Who should not use this approach

If you need guaranteed determinism, a formal SLA, or full control over where inference happens, a free-tier endpoint and a schema check will not get you there. Teams handling regulated data should confirm the host's data handling terms before sending a single issue, and high-volume workloads will likely outgrow a free server quickly. Anyone unwilling to curate a few dozen labelled examples should also skip the drill, because without a golden set the comparison is just two opinions.

If you want to try the workflow without committing a paid key, run the drill against MonkeyCode's free model access first and keep the scheduled job on the free server option until the numbers earn something bigger. Check the project page for the current allowance and server terms on the day you start, and let the decision table, not the announcement, tell you when to swap.