Case Study: A Nightly Maintenance Worker That Rebuilds Its Own State After Every Restart

A nightly maintenance worker needs very little: a process that starts on schedule, one file of durable state, and a cheap way to label what it finds. What it really needs is to stay boring under failure, so you can ignore it for a month and still trust its output.

This case study covers one small project end to end — a worker that scans a handful of repositories for dependency drift, labels each finding, and writes a short digest. The scanning is easy. The hard part is the state handoff that lets the worker restart at any moment, on a host you do not administer, without double-posting or losing its cursor.

The trend feed this week is full of AI feature tours and open AI fatigue, and both reactions are reasonable. The useful counter-move is duller: wrap the probabilistic part in a deterministic shell, so a bad model day cannot corrupt your data.

Goal and non-goals

The goal fits in one sentence: given a list of repositories and pinned versions, produce one digest per day, at most once, with a resumable cursor and an auditable reason for every label.

The non-goals matter just as much, because they are what keeps the design small:

1. No web dashboard, no database, and no queue broker.
2. No multi-worker parallelism in version one.
3. No upstream writes — the worker only reads tags and appends to its own state file.
4. No strict dependence on any single model endpoint for correctness.

That last point is the design hinge. Classification quality is allowed to vary; the digest itself must not.

Decide where the worker lives before you write any code

Most teams skip this table and regret it later. Score your own options against your own constraints.

If your job touches regulated data, or another team depends on its output at a fixed hour, skip the free tier row entirely. The rest of this article assumes you accepted restarts as normal and designed for them.

The state file is the only durable contract

Write every piece of resumable knowledge into one JSON document, and treat that document as the interface between runs. Anything not in the file is reconstructed on startup.

{
  "schema_version": 1,
  "last_completed_run": "2026-10-15T02:10:04Z",
  "cursor": {"repo": "acme/etl", "last_tag": "v3.14.0"},
  "lease": {
    "owner": "worker-7f3c",
    "acquired_at": "2026-10-15T02:09:41Z",
    "heartbeat_at": "2026-10-15T02:09:52Z"
  },
  "recent_labels": [{"repo": "acme/etl", "tag": "v3.14.0", "label": "routine"}]
}

Two rules keep this file trustworthy. First, bump schema_version whenever the shape changes, and refuse to run on an unknown version. Second, only ever commit the file with an atomic rename, so a kill signal cannot leave a half-written document on disk.

import json
import os
import tempfile


def save_state(path, state):
    directory = os.path.dirname(os.path.abspath(path)) or "."
    fd, tmp = tempfile.mkstemp(dir=directory, prefix=".state-", suffix=".tmp")
    try:
        with os.fdopen(fd, "w") as handle:
            json.dump(state, handle, sort_keys=True)
            handle.flush()
            os.fsync(handle.fileno())
        os.replace(tmp, path)  # commit point
    except BaseException:
        os.unlink(tmp)
        raise

The os.replace call is the commit point, and everything before it is invisible to readers. If the process dies during the write, you lose a temporary file and nothing else. If it dies after the rename, the new state is simply the current state.

A lease that fails loudly instead of double-running

Two workers racing on the same cursor is the classic way a nightly job becomes a nightly outage. A lease with a heartbeat solves this without any coordination service, as long as you accept a bounded risk of a stale takeover.

from datetime import datetime, timedelta, timezone

LEASE_TTL = timedelta(minutes=15)


def parse_ts(value):
    return datetime.fromisoformat(value.replace("Z", "+00:00"))


def lease_blocks_run(state, now, owner):
    lease = state.get("lease")
    if not lease:
        return False, "no lease recorded"
    age = now - parse_ts(lease["heartbeat_at"])
    if lease["owner"] == owner:
        return False, "we already own the lease"
    if age > LEASE_TTL:
        return False, "stale lease from %s" % lease["owner"]
    return True, "held by %s for %s" % (lease["owner"], age)

Heartbeat every few minutes, always with the current UTC timestamp, and log the reason string on every exit path. When someone asks why a run stopped, that line is the whole answer.

Model calls behind a deterministic fallback

The labeling step is where a model is genuinely useful and where a free tier is genuinely risky. Keep the call behind a function that can always return a value, and let a rules-based classifier own the default path.

ALLOWED = {"routine", "risky", "unknown"}


def classify(title, call_model, rules_fallback):
    try:
        label = call_model(title, timeout=20)
        if label in ALLOWED:
            return label, "model"
    except Exception as exc:
        print("model tier unavailable: %s" % exc)
    return rules_fallback(title), "rules"

Record the source (model or rules) next to every label. That single field turns a vague quality complaint into a queryable ratio, and it makes provider changes measurable instead of anecdotal.

The code above is a proposal you can run locally with a stub call_model, not a published benchmark. Nothing in this design depends on a specific endpoint, model name, or provider — which is exactly why it survives a free tier changing underneath you.

Where the free tier fits this design

The worker needs two things outside your laptop: a host for the process and an endpoint for the labeling step. MonkeyCode provides free model access and a free server option, which matches both needs in the shape this design already expects. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat free availability as operator-supplied and time-sensitive. Token allowances, rate limits, data retention, and instance lifetimes on free tiers change, so read the current terms on the official site before you schedule anything that matters, and keep the rules-based fallback in place so a policy change degrades quality instead of halting the job.

Results: what each failure does to the run

The value of this structure is not speed; it is a short, predictable list of failure behaviors. Verify each row yourself with the harness below.

Notice what is absent: no partial digest, no duplicated post, no silent data loss. Every failure mode is either a clean refusal or a degraded-but-labeled result.

Two commands to run before you trust it

Write a preflight check and run it as the first step of every scheduled execution. It should parse the state file, compare schema_version, verify the lease, and check clock skew against a known time source.

python -m worker.preflight --state .state/worker.json && \
  python -m worker.run --once

Then test idempotency directly, because this is the property that actually breaks in production. Run the worker twice in a row against an unchanged input set, and diff the state file after each pass.

python -m worker.run --once && cp .state/worker.json /tmp/after-first.json
python -m worker.run --once && diff /tmp/after-first.json .state/worker.json

A clean run should update last_completed_run and nothing else. If the cursor moves on the second pass, your idempotency assumption is wrong, and no amount of logging will save you.

Limitations and who should not use this

This approach trades throughput and guarantees for simplicity, and that trade is not always right.

• Do not use it if your job must run at an exact minute with an SLA, because free hosts restart, throttle, and cold-start without warning.
• Do not use it for regulated or customer-identifying data, because you do not control retention on a free endpoint.
• Do not use it for multi-writer workflows, because a single lease file assumes one logical writer at a time.
• Do not use it if you need sub-minute latency or persistent local disk, since both assumptions are gone on ephemeral hosts.
• Do not skip the fallback classifier, because a free model tier is a shared resource with shared availability patterns.

If your job sits in any of those categories, the table at the top of this article points you elsewhere, and that is a legitimate outcome.

What I would change next

The next iteration is to make the digest itself a contract: fix the key order, the label vocabulary, and the timestamp format, then assert them in preflight the same way you assert schema_version. After that, add one integration test that starts two processes against a shared state directory and asserts that exactly one of them reaches the labeling step. Those two checks catch the overwhelming majority of failures you will actually see in the first year.

If you want to try the loop without provisioning anything, start by running the preflight check on a free server instance and letting it complete a single pass before you add a schedule.