A Cron Entry Is Not a Job: Five Overnight-Batch Myths and a Run Ledger

Last month I woke up to a dashboard that was half empty. The nightly batch had "run". The logs said so. So why was there no data?

Because a schedule is a wish, not a job. Here are five myths I keep hearing, plus a ledger you can run tonight.

Myth 1: "The schedule is the job"

A scheduler entry only promises an attempt. It never promises completion.

Idle hosts sleep. Containers restart. Deploys interrupt. What actually happened is buried in a run record you did not write.

Ask yourself: could I prove this job ran last Tuesday? If the answer needs a scroll through logs, you have no answer.

Myth 2: "A lock is optional because the job is fast"

Fast jobs overlap the moment they get slow. One slow dependency and two runs share a window.

Then you get double writes, double charges, double emails. Nobody notices until a customer does.

• One lock per host is the minimum.
• One run record per window is the real fix.
• "It usually finishes in a minute" is not a control.

Myth 3: "Retries are free"

Retries inside a batch multiply your request count. Five retries on a hundred items is five hundred chances to be told no.

And a 429 is not an invitation to spin. It is a stop signal.

My rule is boring: on 429, stop the batch, write the fact down, exit with a distinct code. The next window picks up the work.

Sleeping in a loop past your own window is how you get two runs in the same hour.

Myth 4: "Exit code 0 means the work happened"

Exit code 0 means your process did not crash. That is a much smaller claim.

Here is a nightly batch I have written too many times. Any resemblance to your code is your problem.

# done.sh — the shape of a confident job
curl -sS "$API/export" -o /tmp/export.json
echo "ok"
exit 0

What if curl returned a 200 with an error body? What if the file is truncated? What if the export is stale by one day?

Exit 0 answered none of those. Write the completion marker yourself, after you check the payload.

Myth 5: "My clock is my window"

Local time, DST switches, and drifting host clocks all move your window. A cron line at 02:00 is 02:00 in whichever timezone the runner believes in.

Compute the window key in UTC, from the data you are processing. Not from now() in a memoized script.

• Window key: 2026-11-07T02 in UTC.
• Job version: a commit or a config hash.
• Both go into the ledger, or you cannot explain a rerun.

The artifact: a run ledger, not a log line

I want three facts per run: window, start, outcome. Append them as JSON lines. Never rewrite the file.

#!/usr/bin/env bash
# run_once.sh — illustrative. Adapt paths and work step.
set -euo pipefail

LEDGER="${LEDGER:-/var/tmp/jobs/runs.jsonl}"
DIR="$(dirname "$LEDGER")"
mkdir -p "$DIR"

# 1. One lock per host. flock needs util-linux; mkdir -p works too.
exec 9>"$DIR/.lock"
if ! flock -n 9; then
  echo "lock busy, exiting" >&2
  exit 75
fi

# 2. Window is computed in UTC, never taken from the scheduler.
WINDOW="${WINDOW:-$(date -u +%Y-%m-%dT%H)}"

# 3. Refuse a second completed run for the same window.
if grep -q "\"window\":\"$WINDOW\",\"event\":\"done\"" "$LEDGER" 2>/dev/null; then
  echo "window $WINDOW already done"
  exit 0
fi

printf '{"event":"start","window":"%s","ts":"%s"}\n' \
  "$WINDOW" "$(date -u +%FT%TZ)" >> "$LEDGER"

# 4. Do the work. Fail loudly.
if ! ./do_work.sh --window "$WINDOW"; then
  printf '{"event":"fail","window":"%s","ts":"%s"}\n' \
    "$WINDOW" "$(date -u +%FT%TZ)" >> "$LEDGER"
  exit 1
fi

# 5. Only now is the window complete.
printf '{"event":"done","window":"%s","ts":"%s"}\n' \
  "$WINDOW" "$(date -u +%FT%TZ)" >> "$LEDGER"

The start line is the important one. A start with no done or fail is a crash you can see.

The probe: find gaps, duplicates, zombies

Now the part people skip. Read the ledger back and compare it to the windows you expected.

# probe_runs.py — illustrative. No results claimed; run it on your ledger.
import json
from datetime import datetime, timedelta, timezone
from pathlib import Path

LEDGER = Path("/var/tmp/jobs/runs.jsonl")
END = datetime(2026, 11, 7, 6, tzinfo=timezone.utc)
HOURS = 48

rows = [json.loads(l) for l in LEDGER.read_text().splitlines() if l.strip()]
by_window = {}
for r in rows:
    by_window.setdefault(r["window"], []).append(r)

expected = [
    (END - timedelta(hours=i)).strftime("%Y-%m-%dT%H")
    for i in range(HOURS, 0, -1)
]

missing, dupes, zombies = [], [], []
for w in expected:
    events = by_window.get(w, [])
    done = [e for e in events if e["event"] == "done"]
    if not events:
        missing.append(w)
    elif not done:
        zombies.append(w)
    elif len(done) > 1:
        dupes.append(w)

print(f"missing={len(missing)} zombies={len(zombies)} dupes={len(dupes)}")
for label, bucket in (("missing", missing), ("zombie", zombies), ("dupe", dupes)):
    for w in bucket:
        print(f"{label} {w}")

Run it in the morning. Run it after every deploy. It takes a second and it answers the only question that matters.

Three buckets, three different bugs:

1. Missing — the scheduler never fired, or the host was asleep.
2. Zombie — it started and died without a verdict.
3. Dupe — two completions, so your lock or your idempotency is broken.

Where free model access fits in

A nightly batch is a decent fit for free model access, because latency is not the product. The morning result is.

MonkeyCode offers free model access and a free server option. Those are availability claims from the operator, not benchmarks, and I am not quoting numbers here.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What matters for this workflow is narrower. Can the batch stop on 429 and resume next window? Can you set a model config hash in the ledger? Those two answers decide more than raw speed.

Add the hash to every record. When output quality shifts, you will know whether the model or your prompt moved.

export MODEL_CONFIG_HASH="$(sha256sum config/model.yaml | cut -c1-12)"
export WINDOW="$(date -u +%Y-%m-%dT%H)"
./run_once.sh

Decision table: does your batch belong here?

Limitations, and who should not do this

This pattern gives you evidence. It does not give you high availability.

• A single host means a single point of failure. The ledger tells you it failed, not how to survive it.
• A JSONL file on one disk is not a distributed lock. Two hosts need a real one.
• If your batch must run within seconds of a trigger, a scheduler is the wrong tool.

Skip this if you need sub-minute guarantees, multi-region failover, or contractual uptime. Skip it if you cannot run a morning probe.

If you can run a probe, start there. Two files, one lock, one UTC key. Then tell me which bucket your batch lands in first: missing, zombie, or dupe?

If you want a place to try it without a card, the free server option is the cheapest way to find out. Just write the ledger before you trust the cron line.