48-Hour Field Notes: The Free Model Returned Valid JSON, and My Schema Said Nothing

Two days ago I shipped a small nightly extraction job, watched it exit 0, and went to bed feeling clever. The next morning the downstream report was empty, the logs were clean, and every single row in my database had records: []. The model had returned perfectly valid JSON. My schema had simply never told it that an empty array was the one answer I could not use.

So here are my field notes from the next 48 hours: what I tried, what actually broke, and the parts I would repeat without hesitating. If you are pointing a batch job at a free model tier and a free server instance, this is the shape of the hole you are about to fall into.

What I was actually asking for

The job itself is boring on purpose. It reads a batch of small text blobs, asks a model to pull out line items (id, amount, currency), validates the result, and writes one row per blob. The worker runs on whatever host I give it, and for this run the host was a free server option rather than a box I rent by the month.

I used MonkeyCode's free model access for the extraction calls, and the worker itself ran on the free server option the same product offers. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am deliberately not quoting a quota, a token ceiling, or a latency number here, because whatever I measured last night on one instance is not a promise about what you will get.

That is the first lesson, and it cost me about six hours: on a free tier, the number you measure is not the number that is guaranteed. Measure it anyway, but treat the measurement as a floor you validate against, not a ceiling you can rely on.

What broke, in the order I found it

Hour 3: the empty success

My JSON Schema said "records": {"type": "array"}. An empty array is a valid array. The model, faced with a blob it could not parse confidently, made a reasonable choice and returned nothing rather than inventing line items. I had written no rule that said "empty is a failure", so nothing failed.

The one-keyword fix is "minItems": 1, and it is the single highest-value line in the whole article. Every schema I write now carries at least one business constraint, because a structural schema only proves the shape, not the meaning.

Hour 11: the repair loop that repaired nothing

My first reflex was to retry on validation failure. I wrote a three-attempt loop with backoff, ran it across 400 blobs, and burned an embarrassing amount of time watching the same malformed payload come back three times. Retrying a content failure is not a retry; it is a coin flip with extra steps.

Hour 19: the config typo that would have run all night

A mistyped model name produced a 4xx on every request. My retry wrapper treated everything that was not a 200 as transient, so the job cheerfully backed off and retried for hours. There is no backoff schedule that fixes a wrong env var, and there is no reason to discover that at 7 a.m.

Hour 27: the box that was not mine

Mid-run, the worker process disappeared and the journal I had been appending to went with it. The extraction calls were fine. The host was ephemeral, which is normal for a free server option, and I had been treating someone else's machine as my own disk.

The failure taxonomy I now use

Before I touch a retry policy, I write down what kind of failure I am looking at. This table is the whole design:

The rule I care about most is the last row. A config error should kill the run in the first minute, not decay quietly into a thousand logged retries that look like progress in your dashboard.

Pre-flight probe: 40 minutes before the first real run

Here is the test plan I run before pointing any nightly job at a free model tier. It is a plan, not a benchmark, and it is meant to be run by you on your own account.

1. Send ten identical requests and record whether all ten satisfy the schema.
2. Send one deliberately ambiguous blob and check whether the model invents a number or returns an empty list.
3. Send one oversized blob and note whether you get truncation, a refusal, or a 4xx.
4. Send one request with a deliberately wrong model name and confirm your code stops instead of retrying.
5. Sleep past any idle window you suspect, then send one more request.
6. Kill the worker mid-run and confirm the journal survives somewhere you control.

The probe is small enough to read in one sitting:

# probe.py - run before you trust a free tier with a nightly job.
import os, json, time, requests

ENDPOINT = os.environ["MODEL_ENDPOINT"]   # documented chat endpoint from your provider
MODEL    = os.environ["MODEL_NAME"]       # the name listed today, not a year ago
HEADERS  = {"Authorization": f"Bearer {os.environ['MODEL_KEY']}"}

SCHEMA = {
    "type": "object",
    "required": ["records"],
    "properties": {
        "records": {
            "type": "array",
            "minItems": 1,               # <-- the keyword that saved me
            "items": {
                "type": "object",
                "required": ["id", "amount", "currency"],
                "properties": {
                    "id":       {"type": "string", "minLength": 1},
                    "amount":   {"type": "number"},
                    "currency": {"type": "string", "enum": ["USD", "EUR", "JPY"]},
                },
            },
        }
    },
}

PROMPT = (
    "Extract every line item from TEXT into JSON matching SCHEMA. "
    "Never invent an amount. If a line has no amount, skip it. "
    "If the text has no line items, return an empty records array.\n\n"
    "SCHEMA: {schema}\n\nTEXT:\n{text}\n"
)

def ask(text, timeout=60):
    body = {
        "model": MODEL,
        "messages": [{"role": "user", "content": PROMPT.format(
            schema=json.dumps(SCHEMA), text=text)}],
    }
    started = time.monotonic()
    r = requests.post(ENDPOINT, headers=HEADERS, json=body, timeout=timeout)
    elapsed = round(time.monotonic() - started, 2)
    return r.status_code, elapsed, r.text

if __name__ == "__main__":
    cases = {
        "clean":    "2 x widget @ 19.00 USD\n1 x cable @ 4.50 USD",
        "ambiguous": "some widgets, price on invoice",
        "empty":    "no line items in this document",
    }
    out = open("probe_results.jsonl", "a")   # keep this OFF the free box
    for name, text in cases.items():
        status, elapsed, raw = ask(text)
        out.write(json.dumps({"case": name, "status": status,
                              "elapsed_s": elapsed,
                              "bytes": len(raw)}) + "\n")
        out.flush()
        print(name, status, elapsed, len(raw))

Run it, fill in what you see, and let the numbers argue with your assumptions. Mine argued with several of mine.

The worker skeleton I would repeat

The second artifact is the loop I now reuse whenever a model output has to satisfy a contract. It classifies before it retries, and it keeps state where the box cannot delete it.

import json, time

RETRYABLE = {"transport", "rate_limited"}
CONTENT   = {"schema", "semantic", "truncated"}
FATAL     = {"config"}

def classify(status, parsed, violations):
    if status in (401, 403, 404):      return "config", status
    if status == 429:                  return "rate_limited", status
    if status >= 500:                  return "transport", status
    if parsed is None:                 return "truncated", status
    if violations:                     return "schema", violations[:3]
    return "ok", None

def handle(status, parsed, violations, repair_used):
    kind, detail = classify(status, parsed, violations)
    if kind == "ok":            return "accept", None
    if kind in FATAL:           return "stop_run", detail
    if kind == "rate_limited":  return "sleep_then_retry", detail
    if kind == "transport":     return "backoff_retry", detail
    if kind == "truncated":     return "shrink_input_retry", detail
    if kind == "schema" and not repair_used:
        return "repair_once", detail   # send the validator error back, once
    if kind == "schema":        return "fail_row", detail
    return "quarantine", detail        # semantic: do not auto-fix

The repair step is one extra call that includes the validator's error message and the offending payload, and it gets exactly one attempt. If that attempt fails, the row fails loudly and lands in a quarantine file I can read over coffee.

The check that is not a schema check

Structure is not truth. After validation I reconcile: sum the extracted amounts and compare them against a total I already know from the source document. If the difference exceeds a tolerance I set per currency, the row is quarantined rather than retried. That one comparison caught more bad rows than every schema keyword combined, and it needs no model access at all.

Results template

I am not going to paste my latency numbers as if they were yours. Instead, here is the table I fill in after every probe run:

If two or more rows come back as schema or semantic, do not scale the batch up yet. Fix the contract first, because a free tier will happily give you a thousand well-formed wrong answers at no charge.

Limitations, and who should not do this

• You cannot treat a free server option as durable storage. Keep the journal and the results off the box, or accept that a restart eats your progress.
• This workflow needs a validator and a reconciliation source. If you have no ground truth to compare against, quarantine has nowhere to send the row.
• If you need hard latency SLOs or guaranteed concurrency, a free tier is the wrong tool and no amount of retry logic will fix that.
• Regulated data that cannot leave your network should not go to any hosted model, free or paid.
• If your output must be byte-for-byte deterministic, a model plus a validator is still a probabilistic system with a gate in front of it.

What I would repeat

Write the business constraint into the schema before the first run. Classify the failure before you retry it. Stop the run on config errors. Keep the journal somewhere the host cannot delete. And run the probe every time you change a model name, because the cheapest bug to fix is the one that fails in minute one instead of hour nineteen.

If you want to try the probe yourself, it runs as-is against MonkeyCode's free model access, and the worker can sit on the free server option while the journal stays on your own disk. Just fill in the results table honestly before you trust it with 400 blobs.

What is the last "valid" response that quietly ruined your morning?