Where Did the Job Go? Checkpointing Model Work on a Free Server

A job calls a model, writes an output file, and dies halfway through.
You restart it, and nobody knows which units already finished.

The interesting question is not why the process died.
It is what your code remembers after the restart.

Most teams answer with logs and hope.
Logs help humans, but they do not resume work.

The setup: one job, many units

Slice the work into units before you run anything.
Each unit has a stable key and one artifact.

• Key: something deterministic, like doc-014#summary.
• Input: the exact prompt and parameters for that key.
• Output: the text or file you want to keep.
• Status: pending, running, done, or failed.

A unit is the smallest thing you are willing to redo.
Keep it small, and restarts get cheap.

Five myths developers repeat

Myth 1: "If the process is alive, the job is progressing"

Liveness is a process fact, not a work fact.
A worker can sit inside one long model call for minutes.

Corrected model: progress is a persisted state transition.
If no row changed, no unit advanced.

Myth 2: "The instance disk is my database"

Free and short-lived servers may lose local files on restart.
Check the vendor's current terms before you rely on that disk.

Corrected model: durable state lives outside the worker.
If it must live on disk, accept loss explicitly and document it.

Myth 3: "A checkpoint is a log line"

A log line records an observation for humans.
A checkpoint records a decision the next run can read.

Corrected model: checkpoint the unit key, its status, and its owner.
Without a key, resume cannot skip anything.

Myth 4: "Restarting is the same as resuming"

A restart starts a cold process with empty memory.
A resume reads persisted state and skips finished units.

Corrected model: resume is a query, not a feeling.
Select units where status is not done, then work.

Myth 5: "Free means one worker, so no races"

A stray manual run or an overlapping instance breaks that assumption.
Two workers can claim the same unit and double your spend.

Corrected model: claim with a lease and a conditional update.
Then treat every side effect as at-least-once.

An illustrative checkpointed runner

The code below is illustrative and was not executed.
It uses stdlib sqlite3 only to keep the demo readable.
Swap in your real database, and keep the same shape.

# Illustrative only. Python 3.10+, stdlib sqlite3, no external deps.
import sqlite3, time, uuid

DB = "jobs.db"

def init(db=DB):
    con = sqlite3.connect(db)
    con.execute("PRAGMA journal_mode=WAL")
    con.execute("""CREATE TABLE IF NOT EXISTS units (
        job TEXT, unit TEXT, status TEXT DEFAULT 'pending',
        owner TEXT, epoch INTEGER DEFAULT 0, lease_until REAL,
        output TEXT, PRIMARY KEY (job, unit))""")
    con.execute("""CREATE TABLE IF NOT EXISTS attempts (
        job TEXT, unit TEXT, epoch INTEGER, owner TEXT, ts REAL)""")
    con.commit(); con.close()

def claim(db, job, owner, lease_s=120):
    """Atomically take one unfinished unit or return None."""
    now = time.time()
    con = sqlite3.connect(db, isolation_level=None)
    con.execute("BEGIN IMMEDIATE")
    row = con.execute("""SELECT unit FROM units
        WHERE job = ? AND (status = 'pending'
              OR (status = 'running' AND lease_until < ?))
        ORDER BY unit LIMIT 1""", (job, now)).fetchone()
    if row:
        con.execute("""UPDATE units
            SET status = 'running', owner = ?, epoch = epoch + 1,
                lease_until = ?
            WHERE job = ? AND unit = ?""",
            (owner, now + lease_s, job, row[0]))
        con.execute("INSERT INTO attempts VALUES (?, ?, ?, ?, ?)",
            (job, row[0], _epoch(con, job, row[0]), owner, now))
    con.execute("COMMIT"); con.close()
    return row[0] if row else None

def finish(db, job, unit, owner, output):
    """Record the artifact only if this owner still holds the unit."""
    con = sqlite3.connect(db, isolation_level=None)
    con.execute("BEGIN IMMEDIATE")
    cur = con.execute("""UPDATE units SET status = 'done', output = ?
        WHERE job = ? AND unit = ? AND status = 'running' AND owner = ?""",
        (output, job, unit, owner))
    con.execute("COMMIT"); ok = cur.rowcount == 1; con.close()
    return ok  # False means the lease moved on; discard your output.

Two details matter more than the code.
First, finish checks ownership in the same statement.
Second, epoch gives you a fencing token for external writes.

If a downstream system accepts writes, pass the epoch with them.
Reject writes whose epoch is older than the stored one.

A restart drill you can adapt

Run this once per job type, on purpose.
It takes minutes and answers the only question that matters.

1. Seed the job with at least ten units.
2. Start the worker, then kill it hard mid-run.
3. Resume the worker with the same job name.
4. Confirm no unit is stuck in running.
5. Confirm each unit wrote exactly one artifact.

python runner.py --job demo &
PID=$!
sleep 3
kill -9 $PID
python runner.py --job demo --resume

-- Every unit should end up done, none left running.
SELECT status, COUNT(*) FROM units WHERE job = 'demo' GROUP BY status;

-- Any unit with zero attempts never got claimed.
SELECT unit, COUNT(*) AS n FROM attempts
WHERE job = 'demo' GROUP BY unit HAVING n < 1;

Add one more check that SQL cannot do for you.
Re-run the same unit twice and compare output bytes.
If they differ, your side effects are not idempotent yet.

Where the free model and free server fit

This workflow needs two cheap things: model calls and a worker.
MonkeyCode offers free model access and a free server option.
That pairing is convenient for a drill like the one above.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat those availability claims as vendor statements and verify them.
Free tiers change, so read the current terms before you depend on them.
Keep the model call inside one unit of work, never across several.
Then a stopped server costs you a lease, not a whole run.

When not to use this

• Sub-second latency budgets. A database round trip per unit adds overhead.
• Truly idempotent, cheap jobs. If a full rerun takes seconds, just rerun.
• Strict exactly-once requirements. This pattern is at-least-once by design.
• Multi-tenant isolation needs. A shared SQLite demo will not satisfy audits.

The corrected mental model

The worker is disposable, and the state is not.
Persist the unit, the owner, and the epoch before you call the model.

Run the restart drill before you add a queue.
If you try it, keep the two SQL checks in your repo.