Dev.to AI 🤖 Ai 👁 0 📖 6 min read

I Let My Agent Schedule Its Own Jobs. Three Weeks Later: 6,048 Runs, 12 Emails Answered Six Times.

The agent proposed this line and I approved it in about four seconds: */5 * * * * /home/sean/venv/bin/python /home/sean/agent/inbox.py Every five minutes. Check the inbox. It reads fine, doesn't it? That is exac

The agent proposed this line and I approved it in about four seconds:

*/5 * * * * /home/sean/venv/bin/python /home/sean/agent/inbox.py

Every five minutes. Check the inbox. It reads fine, doesn't it? That is exactly the problem — a crontab line reads like a sentence, so I reviewed it like a sentence. I never converted it into the only number that matters: how many times does this actually run?

6,048 times, over three weeks. My intent was "a few times an hour, during the day." I got every five minutes, around the clock, including 3 a.m. on Sundays. And because each run took longer than five minutes once there was mail to answer, I got overlapping runs that read the same twelve unprocessed emails and replied to each one six times.

Two incidents in one. The schedule was wrong, and the code underneath it couldn't survive a wrong schedule. Both were mine.

Act one: four days of silence nobody noticed

The job ran zero useful times for the first four days. load_dotenv() and a relative config.yaml path meant every run died instantly, because cron's working directory is $HOME, not my project directory. Cron also doesn't load your shell profile, so PATH is roughly /usr/bin:/bin and your .env does not exist as far as the job is concerned.

Nothing alerted me, because the end of the crontab line looked like this:

*/5 * * * * /home/sean/venv/bin/python /home/sean/agent/inbox.py > /dev/null 2>&1

I added 2>&1 at some point, feeling responsible, then aimed the whole thing at the void. Four days of a dead job producing no output, which in a log directory looks identical to a job that had nothing to do.

The monitoring was a customer asking "did you get my message?"

Fix: absolute interpreter path, cd /home/sean/agent at the top of the wrapper, set -euo pipefail, and every line appended to a log with a timestamp and exit code. Straightforward, twenty minutes. Then it started actually sending email, which is when it got worse.

Act two: the check-then-act loop

Here is the loop the agent filled in from my pattern:

pending = db.query("SELECT * FROM messages WHERE replied = 0")

for m in pending:
    reply = generate_reply(m)                    # 40-90s, model call
    send(reply)
    db.execute("UPDATE messages SET replied = 1 WHERE id = ?", m.id)

The flag is written at the end. A run that starts at 14:05 and finishes at 14:06 overlaps a run that started at 14:10 and read the same rows while they were still replied = 0. Check-then-act, with a 90-second window between the check and the act. Twelve messages, six runs each, seventy-two outgoing replies.

I found out because a customer forwarded two of my replies, four minutes apart, and asked politely whether she was talking to a robot. She was, in the sense that mattered.

The mail provider didn't stop me. My rate limiting didn't stop me — it was per-run, and each run was well under its own limit. Nothing on the box was in a position to say "you already answered this," because the only component that knew the truth was the database, and I never asked it.

What I changed, in order of how much it mattered

1. Claim the window before doing the work, not after.

CREATE TABLE agent_windows (
  window_key  TEXT PRIMARY KEY,   -- 'inbox:2026-11-07T14:05'
  started_at  TIMESTAMPTZ DEFAULT now(),
  finished_at TIMESTAMPTZ
);
row = db.execute(
    "INSERT INTO agent_windows (window_key) VALUES (%s) "
    "ON CONFLICT DO NOTHING RETURNING window_key", (key,)
).fetchone()
if row is None:
    sys.exit(0)   # someone else owns this window

And a UNIQUE(message_id) on the replies table, so even a duplicate attempt fails loudly at the storage layer instead of reaching a human's inbox. Both of these are two minutes of work. Either one alone would have prevented all seventy-two messages.

2. flock as a belt, not as the plan.

flock -n /tmp/agent-inbox.lock -c '/home/sean/bin/inbox-wrapper.sh'

This costs one line and stops same-host overlap. It is not a fix: /tmp is per-host and cleared on reboot, and a run that dies mid-flight leaves a stale started_at row that only the ledger can help you clean up. I use both. I trust one.

3. Write the schedule as a number, then verify the log.

  • */5 * * * * → 288 runs/day → 8,640/month.
  • */5 8-18 * * 1-5 → 132 runs on a weekday, zero at 3 a.m.

I now count runs instead of reading the crontab, because a crontab is a wish and a log is evidence:

journalctl -u agent-inbox --since "-7 days" | grep -c "run start"

If that number doesn't match what you expected, you've found your bug before a customer does.

4. Jitter, because every */5 job on the box fires at :00.

sleep $((RANDOM % 45))

Cheap, and it keeps the model API from seeing twenty simultaneous requests from one Raspberry Pi at the top of every five minutes.

5. Alert on the absence of runs, not just on errors.

A heartbeat file touched at the end of every successful run, and an alert when its mtime is older than twice the interval. I also alert when a single run exceeds three times the median duration — that's the leading indicator for overlap, and it fires minutes before duplicates do. My old alerts only triggered on exceptions. Every expensive failure in this story was a success: exit code 0, seventy-two times.

6. systemd timers for anything that does real work.

Type=oneshot won't start a second copy of the same unit while the first is still running — that property alone would have made this incident impossible. Add RandomizedDelaySec=30 for jitter, Persistent=true if you want catch-up after downtime, and check your calendar expression before you trust it:

systemd-analyze calendar 'Mon..Fri 08..18:00/5:00'

That command prints the next elapse times. Run it. It's the check I skipped.

What I actually got wrong

I let the agent write its own infrastructure, then reviewed the result as prose. For years I've refused to merge a cron change from a human without a "runs per day" number in the description, and I gave a language model a pass because the diff was one line and the one line sounded reasonable.

The agent wasn't wrong, either. "Check every five minutes" is very close to what I said out loud. It was obedient, literal, and fast — which is precisely the combination that punishes a vague review. The sloppiness was entirely in the approval.

Two things I'd tell myself from three weeks ago:

  • The non-idempotent loop was mine. The agent copied my pattern. In my incident log, six of the last nine entries are my code being ordinary, not the model being strange. "The agent did it" is a story I tell because it's more interesting than "I wrote a race condition on a Tuesday."
  • Make the schedule self-reporting. Mine now computes and prints runs_per_day at startup, so every deploy logs a number I can disagree with. That one line has caught two bad schedules since, before they shipped.

The whole fix is about forty lines: a primary key, a unique constraint, a lock, a jitter sleep, and a heartbeat that complains when it stops being touched. What it replaced was my belief that a schedule I hadn't measured was a schedule I understood.

Go count your runs. grep -c your own log right now. If the number surprises you, the duplicates are already in someone's inbox and you're just waiting for the forward.

I write up the specific playbooks in The Solo Operator's AI Agent Playbook — code LAUNCH90 at checkout makes it $1.90. Enter it in the discount field on the checkout page; the link won't apply it for you. If it doesn't save you 5 hours in week one, reply to the receipt for a refund.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.