My Agent Wrote 1,842 'Success' Logs to a Full Disk. Python's Logging Module Never Complained Once.

I found out my agent's observability was fake on a Tuesday afternoon, while trying to answer a boring customer question: "did the nightly report run on the 14th?"

I grepped the log file. Nothing after the 9th.

The dashboard said 100% success. The agent had posted done in Slack every single night. The queue table said the job completed. And there was not one byte of evidence for six days of work.

The disk was full. It had been full since the previous Wednesday.

The numbers, because the numbers are the point

df -h on the Pi:

/dev/root        29G   29G     0  100% /

Then du -xh --max-depth=2 / | sort -rh | head:

5.1G  /var/log
4.2G  /home/agent/logs
1.8G  /home/agent/.cache

Two log directories. 9.3GB of logs on a 29GB card. The agent's own file was agent.log, and next to it sat agent.log.1, agent.log.2.gz... except logrotate had never been installed, so there was no rotation at all. Just one file, 4.2GB, opened with logging.FileHandler in append mode since August.

Why so big? Because I had turned on DEBUG in March to chase a bug and never turned it off. Every HTTP request logged headers. Every prompt logged in full. Every model response logged in full, twice — once on receipt, once in the summary line that included the whole body "for convenience."

That's roughly 700MB a day of a language model explaining itself to me in a file nobody reads.

Why nothing screamed

This is the part that actually scared me. Four separate things failed to tell me the truth, and each one is a normal, defensible default.

1. Python's logging module swallows handler errors by design.

import logging

log = logging.getLogger("agent")
log.addHandler(logging.FileHandler("/home/agent/logs/agent.log"))
log.setLevel(logging.DEBUG)

When the disk fills, FileHandler.emit() raises OSError: [Errno 28] No space left on device. Handler.handleError() catches it. If logging.raiseExceptions is True — the default — it prints a traceback to sys.stderr and returns. The logging call itself does not raise. Your log.info(...) returns None. Your agent continues, cheerfully, into the void.

And my stderr? It was redirected to agent.err, on the same full disk. So the traceback about the disk being full could not be written to the disk that was full.

2. My success markers were in the same place as the logs. The "job done" record was a JSON line appended to a file. The append failed. The code didn't check the return value, because open(..., "a").write() doesn't return one. The Slack ping fired anyway, because the Slack call came before the state write in my code, not after.

3. The queue thought the job never finished. My worker marked tasks complete by writing to SQLite. sqlite3.OperationalError: database or disk is full — caught by my broad except Exception: and logged at WARNING. Which, per point 1, went nowhere. The task stayed pending. It ran again the next night. And the next. Five nights of duplicate runs of a job that emails a summary.

4. My uptime monitor checked the process, not the outputs. systemctl is-active agent.service returned active the entire time. Of course it did. The process was healthy. It was just blind, mute, and repeating itself.

The near-miss

The duplicate runs were the dangerous part, not the disk. The nightly job compiles a per-customer usage summary and mails it. On the fifth night I happened to be awake when the send started, noticed the same subject line twice in my own inbox, and killed it.

I checked the send-log table. Nothing duplicate — because that table lived in the same SQLite file that couldn't be written. The dedupe key I was so proud of was INSERT OR IGNORE into a table that was failing every insert. Deduplication that depends on a write, on a disk that can't write, is not deduplication. It's a coin flip.

If I hadn't been awake, forty-one customers would have gotten the same email twice, on a Saturday, with numbers that were five days stale.

What I changed, in order of how much it mattered

1. A canary write at the top of every run.

def preflight():
    probe = Path("/home/agent/state/.canary")
    try:
        probe.write_text(str(time.time()))
    except OSError as e:
        notify_critical(f"agent cannot write state: {e}")
        raise SystemExit(78)  # fail closed, not open
    free_gb = shutil.disk_usage("/").free / 1e9
    if free_gb < 2:
        notify_critical(f"disk low: {free_gb:.1f}GB free")
        raise SystemExit(78)

The critical detail is SystemExit. My first version logged a warning and continued. A warning on a broken logging system is a thought you had and didn't say out loud.

2. logging.raiseExceptions is not enough — use a handler that escalates.

class LoudFileHandler(logging.FileHandler):
    def handleError(self, record):
        # Do not swallow. Route to a place that outlives this disk.
        try:
            urllib.request.urlopen(
                "https://ntfy.sh/my-agent-alerts",
                data=f"log write failed: {sys.exc_info()[1]}".encode(),
                timeout=5,
            )
        finally:
            raise

An out-of-band channel is the whole trick. Anything that reports failure through the system that is failing is decoration.

3. Rotation, and a cap on verbosity that isn't a comment in a file.

/home/agent/logs/*.log {
    daily
    rotate 7
    maxsize 200M
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

copytruncate because my agent holds the file descriptor open and I didn't want to restart it to rotate. Yes, it can lose lines written during the copy. For debug logs, that's an acceptable trade. For the audit trail, I moved to a separate append-only file with O_APPEND and its own budget.

4. Success is a row, not a message. The Slack ping now fires after a committed write, and it includes the row id. If the write fails, there is no ping. Silence is the alarm.

5. df is a metric, not a command I run when things feel weird. One line in the existing health check, exporting free bytes. It's the cheapest monitoring I have and it was the one thing I wasn't doing.

The honest part

Two confessions, because the fixes above are the easy half.

First: I had a disk-space warning for three weeks before this. It was a WARNING line in a log I wasn't reading, emitted by a check I'd written as a placeholder. I wrote the check, felt productive, and never wired the output anywhere. I have now done this enough times to state the rule plainly — a check that reports into the system it's checking is not a check. It's a diary entry.

Second, and worse: during the six-day window, a customer reported that the agent had given them a wrong number. I spent an afternoon writing about context windows and model drift, and I posted a careful internal note about "increased hallucination rates on long prompts." There were no logs for that window. There was no prompt to inspect. I had invented a theory to explain an absence of evidence, and I'd started to believe it. The actual cause was a stale input file — the upstream fetch had been failing silently since the 9th for the same disk reason — and the model had done something reasonable with five-day-old data.

I blamed the model for my plumbing. That's the most common mistake I make, and this time I have a 4.2GB log file with a hole in it to prove it.

The checklist I now actually run

• df free bytes exported as a metric, alert at 15%
• canary write before every job, fail closed
• log write failures escalate out-of-band
• rotation configured, DEBUG behind an env var, never a code edit
• success notification sent only after a committed state write
• dedupe keys stored somewhere with a different failure mode than the send path

None of it is clever. All of it took an afternoon. The six days of missing evidence cost me a week of debugging and one customer's trust, which is a much worse exchange rate.

If you run an agent unattended on a Pi, go check df -h right now. Then check whether your logger can tell you it's broken. Mine couldn't, for six days, while reporting success 1,842 times.

I keep the operational patterns that came out of incidents like this in The Agent Prompt Vault — $3, lifetime updates. Steal the ones that fit your workflow.