Scheduled jobs that survive a restart
Scheduled jobs that survive a restart I rebooted a box in the middle of the night for a kernel update and killed a job I had been quietly relying on for months. Nothing crashed. Nothing emailed me. The next morning the
Scheduled jobs that survive a restart
I rebooted a box in the middle of the night for a kernel update and killed a job I had been quietly relying on for months. Nothing crashed. Nothing emailed me. The next morning the index was simply stale, and the only reason I noticed was that a query returned a document I had deleted two days earlier. The job ran at 03:00. The reboot happened at 02:58. Cron looked at the clock, saw that 03:00 had already gone past while the machine was down, and moved on with its life.
That was the moment I stopped thinking of "scheduled" as "guaranteed" and started thinking about what a scheduler actually promises.
Cron promises the clock, not the work
Plain cron has no catch-up mechanism at all. If the moment passes while the machine is off, the job is skipped, and there is no record anywhere that it should have run. Worse, MAILTO only fires on an actual run β so the failure mode of a missed job is indistinguishable from the failure mode of a healthy job that had nothing to say. Silence means both "all good" and "never happened."
I had built my monitoring around the assumption that no news is good news. That assumption was wrong in exactly the case I cared about.
anacron remembers, but only roughly
anacron exists precisely for machines that are not always on. Instead of asking "is it 03:00?", it asks "has it been more than a period of days since the last run?" β and it keeps that answer in a timestamp file under /var/spool/anacron, one line per job. If the machine was off for three days and the period is one day, anacron notices the gap and runs the job.
Two details in that design matter more than the feature itself. First, -u updates the timestamp without running anything, which is how you mark a job as satisfied when you have caught it up by hand. -f forces a run regardless, and -s serialises jobs so two of them do not overlap. Second, and this is the part people miss: if you were down for several periods, anacron runs the job once. It does not replay a backlog of missed invocations. That is usually the right call β a daily digest does not need five copies β but it is a decision baked in, not something you configure.
systemd gets closest, with caveats worth knowing
systemd timers with Persistent=true do the thing I originally wanted. The last trigger time is written to disk, and when the timer is activated after a period of inactivity, a missed elapse causes an immediate catch-up run. It has been available since v212, it defaults to false, and β this is the trap β it only applies to realtime timers driven by OnCalendar=. A monotonic timer built from OnBootSec= or OnUnitActiveSec= has no wall-clock schedule to be late for, so Persistent= does nothing there. I have watched someone set Persistent=true on a OnBootSec= timer and then spend an afternoon wondering why the flag had no effect.
The catch-up is also exactly one run. systemd.timer(5) says the unit is triggered immediately if it would have been triggered at least once during the inactive window. So say a job runs hourly and the host is down for six hours: you get one run back, not six. Anything that accumulates per-interval β counters, rolling windows, sequential steps β needs to tolerate a gap rather than assume a fixed stride.
There is a second, quieter guarantee I did not expect. If the timer elapses while the unit is still running from a previous elapse, systemd leaves it running; it does not start a second copy. So a job that occasionally runs long does not stack instances behind itself. And if you are on systemd 257 or later, DeferReactivation=true flips the catch-up behaviour: instead of firing late, it waits for the next scheduled elapse. That is the right setting for a job whose output is only meaningful at a particular time of day.
The state file is what actually makes it safe
None of the above saves you if the job itself cannot tell whether its work has already been done. Schedulers are unreliable narrators; the work has to be its own witness.
The pattern I ended up with is a marker file that records the identity of the completed unit of work β a date, a version, a hash of the input β rather than the fact that some run happened. The job starts, reads the marker, compares it against what it is about to do, and exits quietly if they match. The marker is written only after the work succeeds, and it is written atomically: write a temp file, then rename() it into place. A crash halfway through leaves the old marker intact, which means the next run (catch-up or scheduled) retries the work instead of skipping it.
With that in place, it stops mattering whether the trigger came from a catch-up, from anacron, or from me typing the command by hand. Running it twice is not an event. Running it zero times is the only real failure, and the marker makes that visible.
Silence should mean something
The other half is output. My jobs print nothing when there is nothing to report. No "starting", no "0 items processed", no cheerful summary. If a job produces a line, that line is a signal that someone should read it. This is the watchdog shape: the absence of output is the heartbeat, and any output is an alarm.
That only works if the absence itself is monitored β otherwise I am back to MAILTO's blind spot, where a dead machine and a quiet night look identical. The fix is a dead-man's switch: after a successful completion the job pings a healthcheck endpoint, and the alarm is the missing ping rather than a message. Now the failure mode I actually hit β job never ran because the box was down β produces an alert, because the ping for that window never arrived.
Reloading without gutting a run
The last piece was reloads. Config changes used to mean restarting the unit, which meant killing a job mid-pass and leaving half-written state behind. Now a reload request sets a flag, and the job checks that flag at safe points: finish the current item, write the marker, exit. The timer's next elapse starts a fresh process with the new config. Because systemd does not re-execute an active unit on elapse, the reload cannot accidentally produce two workers fighting over the same marker file β the drain happens first, and the schedule picks up afterwards.
I still reboot that box. I just no longer assume the schedule owes me anything.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.