48-Hour Field Notes: Same Prompt, Two JSON Shapes, and a Validator That Passed Both

Six hundred markdown files landed in an inbox folder, and my only job was to turn them into a one-page morning report. The plan looked trivial: send each file to a model, get back four fields, append a line of JSONL, render a table. I gave myself forty-eight hours to make that loop boring, so like an idiot I built the happy path first.

The happy path held for about six hours. The full run exited 0, the report rendered, and nobody on the receiving end noticed that one column had quietly stopped meaning anything. These are the field notes: what I tried, where it broke, and the small harness I would keep for next time.

The constraints I actually had

Before the timeline makes sense, here is the shape of the problem.

1. Roughly 600 small text files per night, growing by maybe 5% a week.
2. Four output fields per file: category, severity, needs_human, summary.
3. A morning report that a human reads, so a wrong column is worse than a missing row.
4. No budget, which meant a free tier and a free machine, and no assumption of either staying stable.

That last one is the constraint people skip, and it produced every bug below. If your model access can change shape between Tuesday and Wednesday, then your real interface is not the API — it is the contract you wrote down and test.

Hour 0–6: the validator that agreed with everything

I wrote the extraction call, a parser, and a validator, and all three passed on the first twenty files. The validator was a dataclass with type annotations checked by isinstance, which felt rigorous and was mostly theater.

# validate_v0.py -- the version that lied to me
FIELDS = {"category": str, "severity": str, "needs_human": bool, "summary": str}


def validate_v0(payload: dict) -> list[str]:
    errors = []
    for key, expected in FIELDS.items():
        if key not in payload:
            errors.append(f"missing {key}")
        elif not isinstance(payload[key], expected):
            errors.append(f"{key} is {type(payload[key]).__name__}")
    return errors

Do you see what is missing? Nothing checks the values. severity: str accepts "high" and "P1" and "HIGH " with a trailing space, and all three pass the same test. My validator was answering "is this a string" when the report needed "is this one of three levels".

Hour 6–18: moving to the free server and 19 quiet repairs

I moved the batch onto MonkeyCode's free server option, which is where the overnight run was supposed to live, and used its free model access for the extraction step.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I treat both of those as conveniences rather than guarantees, which is why nothing below hardcodes a model name, a quota, or a machine size. The only thing I actually rely on is that a call returns text, and that text usually contains a JSON object.

Nineteen of six hundred responses failed to parse, so I added a repair loop with a budget of one extra call. Nineteen became zero, the job exited 0, and I went to bed feeling clever. That feeling cost me a morning.

def extract_json(text: str) -> dict:
    start, end = text.find("{"), text.rfind("}")
    if start == -1 or end <= start:
        raise ValueError("no JSON object found in model output")
    return json.loads(text[start:end + 1])

Hour 18–30: the report that looked fine and was not

The morning report rendered beautifully and the severity column was garbage. Two thirds of the rows said high, medium, or low, and the rest said P1, Sev-2, and one memorable critical!. No exception fired, no row was dropped, and the chart at the top just looked like a bad week.

This is the failure mode I keep relearning: a schema that only checks structure will happily certify a semantic regression. The model had not broken. My contract had never specified the value domain, so the model was free to answer the question it thought I asked.

What would you have checked at hour 18? I checked row counts, because row counts are the thing that goes wrong in demos. Row count was perfect.

Hour 30–42: the artifact I should have written first

I spent the next twelve hours building a shape checker, and it is the only part of this project I would keep. The idea is simple: fingerprint the structure of every response, pin the domain of every enumerated field, and make any deviation a non-zero exit code.

1. Fingerprint the shape, not the text

# fingerprint.py
from __future__ import annotations

import hashlib
import json
from typing import Any


def type_name(value: Any) -> str:
    if value is None:
        return "null"
    if isinstance(value, bool):      # before int: bool subclasses int
        return "bool"
    if isinstance(value, int):
        return "int"
    if isinstance(value, float):
        return "float"
    if isinstance(value, str):
        return "str"
    if isinstance(value, list):
        return "list"
    if isinstance(value, dict):
        return "dict"
    return type(value).__name__


def shape_of(obj: Any, prefix: str = "") -> dict[str, str]:
    """Map every JSON path in a payload to a type name."""
    out: dict[str, str] = {}
    if isinstance(obj, dict):
        for key in sorted(obj):
            path = f"{prefix}.{key}" if prefix else key
            out[path] = type_name(obj[key])
            out.update(shape_of(obj[key], path))
    elif isinstance(obj, list):
        if not obj:
            out[f"{prefix}[]"] = "empty"
        else:
            out[f"{prefix}[]"] = type_name(obj[0])
            out.update(shape_of(obj[0], f"{prefix}[]"))
    return out


def fingerprint(payload: dict) -> str:
    blob = json.dumps(sorted(shape_of(payload).items()), separators=(",", ":"))
    return hashlib.sha256(blob.encode()).hexdigest()

Note the bool-before-int ordering, because isinstance(True, int) is True and that single line has burned me before. Note also that the fingerprint hashes key paths and type names only, so a reworded summary does not trip the alarm while a nested object where a string used to be does.

2. Pin the value domains

Structure alone would not have caught the P1 problem, so the contract also declares allowed values and fails on anything outside them.

# contract.py
from __future__ import annotations

from dataclasses import dataclass, field

from fingerprint import shape_of

EXPECTED_SHAPE = {
    "category": "str",
    "severity": "str",
    "needs_human": "bool",
    "summary": "str",
}

ALLOWED_VALUES = {
    "category": {"billing", "bug", "account", "other"},
    "severity": {"low", "medium", "high"},
}


@dataclass
class Report:
    missing: list[str] = field(default_factory=list)
    unexpected: list[str] = field(default_factory=list)
    retyped: list[str] = field(default_factory=list)
    out_of_domain: list[str] = field(default_factory=list)

    @property
    def ok(self) -> bool:
        return not (self.missing or self.unexpected or self.retyped or self.out_of_domain)

    def render(self) -> str:
        lines = []
        if self.missing:
            lines.append(f"missing: {self.missing}")
        if self.unexpected:
            lines.append(f"unexpected: {self.unexpected}")
        if self.retyped:
            lines.append(f"retyped: {self.retyped}")
        if self.out_of_domain:
            lines.append(f"out of domain: {self.out_of_domain}")
        return "\n".join(lines) or "ok"


def check(payload: dict) -> Report:
    report = Report()
    shape = shape_of(payload)

    for path, expected in EXPECTED_SHAPE.items():
        actual = shape.get(path)
        if actual is None:
            report.missing.append(path)
        elif actual != expected:
            report.retyped.append(f"{path}: {actual} != {expected}")

    for path in shape:
        if path not in EXPECTED_SHAPE:
            report.unexpected.append(path)

    for path, allowed in ALLOWED_VALUES.items():
        value = payload.get(path)
        if isinstance(value, str) and value.strip() not in allowed:
            report.out_of_domain.append(f"{path}={value!r}")

    return report

Yes, this contract is a flat dictionary, and yes, that is deliberate — the moment I added nesting I needed a second fingerprint for each child shape.

3. Replay saved responses instead of calling the model

The part that saved the most time was writing every raw response to disk with a content hash, then testing the validator against those files. Validation bugs are cheap to fix offline and expensive to find at 3 a.m.

mkdir -p fixtures/raw out

python run_batch.py \
  --inbox inbox/ \
  --out out/report.jsonl \
  --raw fixtures/raw \
  --repair-budget 1

# what actually landed in the report?
jq -r '.severity' out/report.jsonl | sort | uniq -c | sort -rn

# tests/test_contract_replay.py
import pathlib

import pytest

from contract import check
from run_batch import extract_json

RAW = sorted(pathlib.Path("fixtures/raw").glob("*.txt"))


@pytest.mark.parametrize("path", RAW, ids=[p.name for p in RAW])
def test_saved_response_matches_contract(path):
    payload = extract_json(path.read_text())
    report = check(payload)
    assert report.ok, f"{path.name}\n{report.render()}"

Forty saved responses run in under a second and cost nothing, which means I can change the contract and immediately see which historical responses I would now reject. That is a much better feedback loop than re-running the batch and squinting at a table.

4. Make drift a non-zero exit code

# run_batch.py (core loop, trimmed)
import hashlib
import pathlib
import sys

from contract import check
from fingerprint import fingerprint

FAIL_RATIO = 0.02


def handle(path: pathlib.Path, call_model, raw_dir: pathlib.Path, repair_budget: int) -> dict:
    prompt = build_prompt(path.read_text())
    payload = None
    text = ""
    for attempt in range(repair_budget + 1):
        text = call_model(prompt if attempt == 0 else repair_prompt(prompt, text))
        digest = hashlib.sha256(text.encode()).hexdigest()[:12]
        (raw_dir / f"{path.stem}-{digest}.txt").write_text(text)
        try:
            payload = extract_json(text)
            break
        except ValueError:
            continue
    if payload is None:
        raise ValueError(f"no parseable JSON for {path.name}")
    return {"source": path.name, "fingerprint": fingerprint(payload), **payload}

results = [handle(p, call_model, raw_dir, args.repair_budget) for p in inbox]
drift = [r for r in results if not check(r).ok]
if drift:
    sys.exit(2)
if repairs / max(len(results), 1) > FAIL_RATIO:
    sys.exit(3)

Exit code 2 is intentionally separate from 3, because drift means "the world changed" and a high repair rate means "the model is struggling" — different humans should be woken up for those.

Hour 42–48: three live runs and the second drift

On the second live run, needs_human came back as the string "false" on four rows. Under my old validator that would have been a truthy string that marked everything as needing a human. Under the new contract it was a retyped path, exit code 2, and a one-line message on the terminal.

Three runs later I had a golden set of forty responses, a contract that fails loudly, and a repair budget I could tune without lying to myself about failure rates. Nothing about that is clever, and that is exactly the point.

Decision table: repair, retry, or fail closed

This is the table I wish I had written on hour one.

The pattern is boring on purpose: repair is only for unparseable text, and everything semantic fails closed. Every time I let a validator "fix" a value, I created a bug that took longer to find than the original failure did.

What I would repeat, and what I would throw away

Repeat: writing the contract before the prompt, saving raw responses with hashes, replaying fixtures offline, and splitting exit codes by failure class. Throw away: the repair-on-any-error loop, the str-typed enums, and the row-count smoke test that told me nothing.

I would also keep the free tier for what it is good at — a nightly batch at this size, on a machine I do not have to own. It is not a plan for ten times the volume, and I have not tested what happens then.

Limitations and who should skip this

Three honest limits.

• A fingerprint detects change, not correctness. If the model returns the wrong category with the right shape, this harness passes it.
• The domains are hand-written. {"billing", "bug", "account", "other"} came from reading fifty files, so it is a guess with a changelog.
• The fixtures are a snapshot. Forty saved responses are evidence about those forty responses, not a distribution.

Skip this if you read every output yourself, or if your provider offers a strict schema mode you can enforce at generation time — that is a stronger guarantee than a post-hoc checker. Skip it too if the output reaches customers without a human seeing it first, because a contract test is not a review process.

If you only take one thing: write down the shape and the allowed values before you write the prompt, then make drift fail the build. Your future self gets to sleep through the night, and that is worth twelve hours of anyone's weekend.