I chmod 600'd My Agent's .env and Called It Done. The Key Was in Four Other Places, Starting With My Shell History.

The audit started because a reader asked me a question I couldn't answer

A reader emailed me in October: "How do you rotate your agent's API key?"

My honest answer was: I open .env, paste a new key, restart the service. That's not a rotation, it's a replacement. And I had no idea whether the old key still worked anywhere.

So I did the laziest possible audit. One grep, across my home directory and /etc:

grep -rIl -E 'sk-[A-Za-z0-9]{20,}|ghp_[A-Za-z0-9]{20,}|AKIA[0-9A-Z]{16}' \
  ~ /etc/systemd/system 2>/dev/null

Nine files. My .env was one of them. The other eight were the interesting part, and none of them were a file I would have called a secrets file.

This is the post I wish I'd read seven months ago, when I thought protecting a secret meant protecting one file.

What "chmod 600" actually buys you

Here's the threat model I thought I had: another user on the box reads my key.

That's a real threat and 600 closes it. What it does not close is: anything running as the same user reads my key.

My agent runs as a dedicated user, agent. Fine. But it also has a shell tool: it can run bash, printenv, python3 -c "import os; print(os.environ)", and read /proc/<pid>/environ for any sibling process — same UID, and /proc doesn't stop you.

So 600 is not a lock. It's a fence between users. If the thing you're most afraid of is your own agent being manipulated into exfiltrating something — a prompt injection, a malicious dependency, a tool call you didn't intend — then chmod 600 protects you from exactly nothing, because the attacker is already inside the fence. It's running as agent.

I knew this in the abstract. I did not know it in the way that makes you change a systemd unit.

Place 1: my shell history, at 644

In August I spent three hours debugging a 401. At some point I ran something like this:

export OPENAI_API_KEY=sk-proj-XXXXXXXXXXXX python3 test_call.py

It worked. I moved on. ~/.bash_history still had that line, and unlike my .env, it was mode 644 — world-readable on a box where I'd once given a friend an SSH account to look at a Grafana dashboard.

The irony is specific: I had spent an afternoon hardening a file, then left the same string in plaintext in a file I never thought of as containing secrets. A history file is a log of everything I've ever typed, which makes it a worse secrets file than .env by every measure.

What I do now: a HISTCONTROL=ignoreboth plus HISTIGNORE='*KEY*:*TOKEN*:*sk-*:*password*' in my shell rc, and if I need to test a key interactively I read it:

read -rs KEY && curl -H "Authorization: Bearer $KEY" ...

It's four extra seconds and keeps the string out of the file that syncs to every backup I own.

Place 2: the systemd unit file, which is world-readable by default

For a while my unit had this in it:

[Service]
Environment=OPENAI_API_KEY=sk-proj-XXXXXXXXXXXX
Environment=AGENT_DB=/home/agent/state.db

/etc/systemd/system/agent.service is created 644, and systemctl cat agent.service prints the whole thing to any local user without a password prompt. I had put the secret in the one place on the machine designed to be readable.

The migration was small:

[Service]
User=agent
EnvironmentFile=/etc/agent/secrets.env
# secrets.env: root:root, 0600, and systemd reads it as root

That's better. It is not perfect — the process still has the key in its environment, and the process is what I'm worried about. So for the key that matters most I went further: Debian 12 ships systemd 252, so LoadCredential= exists:

LoadCredential=openai_key:/etc/agent/openai.key

The agent reads $CREDENTIALS_DIRECTORY/openai_key, a tmpfs path scoped to the unit, and the value never enters the environment block at all. systemctl show -p Environment returns nothing useful. It took twenty minutes, and it's the single change I'd keep if I had to throw the rest away.

Place 3: a crash dump that printed my headers

In September a requests call in my worker raised on a bad gateway. My handler logged it with exc_info=True, and the exception repr included the PreparedRequest — whose repr includes headers. Authorization header, in full, into journald.

Survivable, except for two settings I'd made months earlier and forgotten:

# journald.conf
Storage=persistent

and, worse, I had added agent to the adm group back in June to read logs without sudo, and never removed it. So the box had a persistent, on-disk copy of the key in /var/log/journal, in a file the compromised process could read.

You can't stop a library printing a header in a traceback. You can stop it at the formatter:

import logging, re

SECRET_RE = re.compile(r'(sk-[A-Za-z0-9_\-]{8})[A-Za-z0-9_\-]+')

class Redact(logging.Filter):
    def filter(self, record):
        msg = record.getMessage()
        if SECRET_RE.search(msg) or 'Authorization' in msg:
            record.msg = SECRET_RE.sub(r'\1…REDACTED', msg)
            record.args = ()
        return True

logging.getLogger().addFilter(Redact())

It's not clever and won't catch everything. It catches the case that actually happened, which is how I judge every control on this box.

Place 4: the backup tarball on a share with guest read

My nightly backup was a tar of /home/agent into /home/agent/backups, rsync'd to a NAS whose share had guest read because I set it up in 2024 to move photos and never revisited it. The tarball inherited the default umask, 022, so it was 644 on the NAS too. And of course it contained .env.

The whole hardening exercise sat downstream of a pipeline that mailed the key to a share my TV can browse.

The fix that mattered wasn't file modes, it was exclusion plus encryption:

umask 077
tar --exclude='.env' --exclude='*.key' --exclude='backups' \
    -czf - /home/agent | age -r $AGE_PUBKEY > agent-$(date +%F).tar.gz.age

Excluding .env means a restore needs one manual step, which I've written down. That's a real tradeoff and I'd rather have it than a searchable copy of my keys on a consumer NAS.

The honest part: I fixed this and then broke it again

Here's the failure I'm least proud of.

After the audit I rotated the key properly. New key created, old key revoked via the provider's API, .env updated, service restarted, a note in the changelog with the date. I felt genuinely good about it.

Eleven days later staging broke. Staging had its own .env, in a directory I'd forgotten existed, still holding the revoked key. My fix — I want to be precise about this — was to create a new key and paste it into staging/.env, prod/.env, the unit file's Environment=, and a Postman environment. I recreated the exact problem I had just spent a weekend solving, because the pressure was "make staging work in the next ten minutes," and the correct fix (one key per environment, one source of truth) was a two-hour job I did not have two hours for.

Two weeks later I found the new key in a .env.bak I'd created while editing. Which is how secrets actually escape: not through one dramatic leak, but through a series of reasonable ten-minute decisions made by a tired person who has already solved this problem once and resents solving it again.

The thing that broke the loop wasn't discipline. It was making rotation a single script that writes the file, restarts the unit, and prints the revocation command — so the fast path and the correct path became the same path.

The takeaways, in the order I'd apply them

1. Write down your threat model before your controls. Mine was "another user reads the file." Every control I had was correct for that threat and useless for the real one: my own agent, running as me.
2. Grep for the key shape, not the filename. sk-, ghp_, AKIA, plus your provider's prefix. Secrets live in history, unit files, logs, backups, and .bak files, and none of those look like secrets.
3. Default to umask 077 in anything that writes files: services, cron jobs, backup scripts. One line, closes a whole class.
4. Get keys out of Environment=. EnvironmentFile= with root-owned 0600 is the ten-minute version; LoadCredential= is the twenty-minute version and strictly better.
5. Redact at the formatter — you cannot control what a library puts in an exception string.
6. Treat group memberships as secret surface. adm can read persistent journald. I added myself for convenience and never subtracted.

The uncomfortable summary is that chmod 600 made me feel done, and feeling done is what stopped me looking for seven months. A permission bit answers a question about other users. It says nothing about the process you're trying to contain, and that process is usually the one with the shell.

The full checklist + scripts are in Ship Safe — The Launch-Day Security Kit — code LAUNCH90 at checkout makes it $1.50. Enter it in the discount field on the checkout page; the link won't apply it for you.

If you've run this audit on your own box, I'd like to know where your key turned up. Mine was shell history.