Nine Probes I Run Before a Model-Backed Batch Job Leaves My Laptop

A batch job that calls a model looks finished long before it is portable. It runs on the machine where it was written, against paths that happen to exist, with secrets that happen to be exported. Move it one host away and the same code reports success while writing nothing useful.

This is the checklist that replaced my guesswork. The job reads a JSONL file, sends one record at a time to a model, and writes one output file per record. It runs on MonkeyCode's free models and, eventually, on its free server option, so the code has to be correct on a host I do not control.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Everything below is a probe I run myself; where I cannot verify a platform behaviour, the probe says so instead of assuming.

Each probe is short, loud, and runnable before the first deploy. Run them in order. The early ones are cheap and catch the boring mistakes.

Probe 1 — Run with an empty environment

Local shells leak. A export API_TOKEN=... in a dotfile makes the job pass for weeks and fail on the first clean host.

# fails unless secrets come from an explicit source
env -i PATH=/usr/bin:/bin HOME=/nonexistent ./run_batch.sh --dry-run
# also check the source tree for hardcoded paths
if grep -rnE '/(Users|home)/[a-z0-9_]+' ./src; then
  echo "FAIL: developer-specific absolute path in src"; exit 1
fi

A dry run that needs a real HOME should say so in its own error message. Silent fallbacks to ~/.cache/job are the reason a job works locally and dies remotely.

Probe 2 — Delete the working directory, then run again

State hides in three places: process memory, the working directory, and /tmp. Copy the job into a fresh directory with only its inputs and run one record.

tmpdir=$(mktemp -d)
cp -r ./src ./fixtures ./run_batch.sh "$tmpdir"/
( cd "$tmpdir" && ./run_batch.sh --limit 1 ) || echo "FAIL: not relocatable"

Anything that must survive a restart belongs outside that directory. Decide once, in a table, and write the decision into the README.

Where state is allowed to live

Treat any remote filesystem as replaceable until you have verified persistence yourself. That includes the free server option: verification is a probe, not an assumption.

Probe 3 — Feed one deliberately broken record

Build a fixture file where record 4 of 20 is malformed. The batch must produce 19 outputs, one error line, and exit non-zero only if you asked it to.

jq -c 'select(.id != "rec-04")' fixtures/twenty.jsonl > /tmp/good.jsonl
echo '{"id":"rec-04","text":null}' >> /tmp/good.jsonl
JOB_ROOT=$(mktemp -d) ./run_batch.sh --input /tmp/good.jsonl
ls "$JOB_ROOT/out" | wc -l   # expect 19

If the count is 3, the job dies on first error. If the count is 20, it wrote a file for a null input, which is worse.

Probe 4 — Interrupt at half and resume

The resume rule has to be an ordering rule. Write the output file first, then append the done record. A crash between the two steps costs one repeated call; the reverse order loses a record and marks it complete.

import hashlib, json, os, pathlib

ROOT = pathlib.Path(os.environ["JOB_ROOT"]).resolve()
OUT, DONE = ROOT / "out", ROOT / "done.jsonl"

def key_of(record):
    blob = json.dumps(record, sort_keys=True, ensure_ascii=False)
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()

def write_atomic(path, text):
    part = path.with_suffix(path.suffix + ".part")
    part.write_text(text, encoding="utf-8")
    os.replace(part, path)  # atomic on one filesystem

def commit(record, text):
    key = key_of(record)
    write_atomic(OUT / f"{record['id']}.txt", text)
    with DONE.open("a", encoding="utf-8") as fh:
        fh.write(json.dumps({"key": key, "id": record["id"]}) + "\n")
        fh.flush()
        os.fsync(fh.fileno())

Kill the process with SIGKILL, restart it, and assert the total output count never exceeds the input count. Duplicate rec-07.txt files are a resume bug, not a model bug.

Probe 5 — Reject empty and truncated responses

The failure mode is a 200 status with a useless body. Validate before writing, not after.

def validate(text, min_chars=40):
    if text is None:
        raise ValueError("null body")
    body = text.strip()
    if len(body) < min_chars:
        raise ValueError(f"short body: {len(body)} chars")
    return body

Free models make this path cheap to reproduce, so there is no budget excuse for shipping it untested. Run the validator against a fixture set of empty, whitespace-only, and cut-off strings.

Probe 6 — Compare two runs byte for byte

Identical output across two runs is not promised by anyone. The point is to measure it, so a change becomes visible.

run() { JOB_ROOT="$1" ./run_batch.sh --input fixtures/smoke.jsonl >/dev/null; }
a=$(mktemp -d); b=$(mktemp -d)
run "$a"; run "$b"
if diff -r "$a/out" "$b/out" >/dev/null; then echo "identical"; else echo "differs: record it"; fi

Record the result per model alias in the repo. A sudden flip from identical to different is a signal about your prompt or your inputs, not a moral judgement of the endpoint.

Probe 7 — Grep the log for a sentinel

S="SENTINEL-DO-NOT-LOG-$$".
echo "{\"id\":\"s1\",\"text\":\"$S\"}" > fixtures/sentinel.jsonl
JOB_ROOT=$(mktemp -d) ./run_batch.sh --input fixtures/sentinel.jsonl 2>&1 | grep -q "$S" \
  && echo "FAIL: record body reached stdout or the log"

Log the record id, the model alias, the duration, and the status. Never the body. The sentinel is the only way to prove it.

Probe 8 — Count the files touched outside the job directory

This probe is Linux-only and needs strace. Container users can replace it with read-only bind mounts and watch what breaks.

strace -f -e trace=openat -o probe.strace ./run_batch.sh --limit 1
awk '{print $2}' probe.strace | grep -oE '"[^"]+"' | tr -d '"' \
  | grep -v "$PWD" | grep -vE '^/(usr|lib|proc|sys|dev|etc/ld)' | sort -u

Every unexpected path is either a dependency to declare or a write to remove.

Probe 9 — Rehearse the stop button

JOB_ROOT=$(mktemp -d) ./run_batch.sh --input fixtures/smoke.jsonl &
pid=$!
sleep 2 && kill -TERM "$pid"
timeout 5 tail --pid="$pid" -f /dev/null || echo "FAIL: ignored SIGTERM for 5s"
JOB_ROOT=$(mktemp -d) ./run_batch.sh --input fixtures/smoke.jsonl  # must still work

A job that ignores SIGTERM cannot be deployed on any host where you do not have a console. The follow-up run proves the interrupted state did not poison the directory.

What these probes do not cover

• They do not measure throughput, cost, or quality. Those need a benchmark with a fixed fixture set.
• They do not prove remote persistence. They only prove the job does not depend on it silently.
• They assume single-writer access. Two workers on one directory need a real lock, not a probe.
• They say nothing about rate limiting or quotas, which are the one thing you should read from the provider instead of guessing.

Who should skip this

Teams already running the job in a container with declared volumes, a migration tool, and a staging environment have solved these problems structurally. A solo script that will never leave one laptop does not need probe 4. Anyone shipping to a regulated environment needs more than a checklist: log retention and data classification come first.

Start with probe 3 and probe 4. They catch the two failures that make a batch job report success while quietly losing records, and both run in under a minute.