Nine Keys Can SSH Into Production. Five Owners Don't Work Here Anymore.

I cat'd authorized_keys on our oldest box for a boring reason and kept reading: the 2023 contractor, a dead CI pipeline, two ex-engineers, an 'ops-temp' nobody can attribute. The sweep, the smoke alarm, the one-week evidence trick, and keys-that-expire culture.

I opened the file for a boring reason — a connectivity weirdness — and then just kept reading. ~/.ssh/authorized_keys on our oldest production box: nine lines, nine doors.

Matching comments to memory: the 2023 migration contractor (still there), the CI pipeline decommissioned last spring (still there), two engineers who left — one warmly, one not (still there), a key commented ops-temp that nobody including our ops lead can attribute to a human, and one line with no comment at all, which is its own small horror genre.

Four keys belonged to current humans who still needed access. Five were sediment. The door remembered everyone, because nobody had ever asked it to forget.

authorized_keys is an attendance sheet
The oldest access-control file in our profession is a plain-text list on disk: appended to for a decade, reviewed never. Every line was added on someone's worst day — incident, deadline, "just for tonight, add my key." Keys arrive during emergencies; emergencies don't do cleanup; departures miss the file entirely, because it isn't in any system. It's wallpaper. And unlike a password, a key never expires out of embarrassment — it waits for its laptop to be sold, lost, or inherited.

The sweep

#!/bin/bash
# keys-audit.sh — every authorized_keys on every box, with its "who/why" comments
for host in $(cat hosts.txt); do
  ssh "$host" 'for f in /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys; do
    [ -f "$f" ] && {
      echo "== $HOSTNAME $f — $(grep -cvE "^[[:space:]]*$|^[[:space:]]*#" "$f") keys"
      grep -vE "^[[:space:]]*$|^[[:space:]]*#" "$f" | awk "{print \"   \", \$NF}"
    }; done'
done | tee keys-audit.txt
# then the human step: diff the comment column against the current roster.
# every line that doesn't match a living owner is sediment.

Ours diffed to five. Yours will too; that number is your real access list.

The four changes that stuck

1. Keys got names and reasons. One comment format, enforced at add-time:

echo "$(cat ana.pub) ana@laptop — incident-4821 — 2026-11-03" \
  | sudo tee -a /home/ubuntu/.ssh/authorized_keys
# a key that can't state its reason in six words doesn't get added.
# 'ops-temp' failed this test immediately and was deleted with a small ceremony.

1. Offboarding gained a server step. Badge, laptop, SSO — and this, on every box:

grep -rn 'ana@' /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys

The step exists because the 2023 contractor taught us it doesn't happen by osmosis.

1. The file got a smoke alarm. Any change pages someone — not blocks, pages; additions are sometimes legitimate at 2 a.m., silent additions never are:

# auditd: watch writes and attributes to authorized_keys on every box
echo '-w /root/.ssh/authorized_keys -p wa -k ssh_sediment' | sudo tee /etc/audit/rules.d/ssh.rules
sudo augenrules --load
# wire the audit event to your pager; our worst file became our best-monitored one

1. Deletion ran on evidence, not folklore. We almost kept the unattributed key "in case something depends on it." Instead: a week of fingerprints.

sudo sed -i 's/^#LogLevel INFO/LogLevel VERBOSE/' /etc/ssh/sshd_config && sudo systemctl reload ssh
# after a week — which fingerprints actually authenticated?
sudo journalctl -u ssh --since "-7 days" | grep -oE 'SHA256:[A-Za-z0-9+/=]+' | sort | uniq -c > used.txt
# and per line in the file — which keys never appeared in used.txt?
awk '{print $1" "$2}' /root/.ssh/authorized_keys | while read -r t k; do
  fp=$(echo "$t $k" | ssh-keygen -lf - 2>/dev/null | awk '{print $2}')
  grep -q "$fp" used.txt || echo "UNUSED ALL WEEK: $fp"
done

Nothing used it. Measure before you mourn.

New boxes start at one
The habit stuck because new machines made it easy: on the Cubes we run at Krova Cloud, a box boots with exactly one key — the one handed at creation — and root login is off by default. Additions become a decision you make, not sediment you inherit. Humans increasingly skip static keys entirely: short-lived certs through a bastion for people, per-pipeline keys rotated with the pipeline for robots. Zero sediment, not zero keys.

The honest part

• Static keys aren't a sin. Plenty of honest estates run them fine because they run the reviews. The sin is the unreviewed list, not the mechanism.
• Ceremony has a cost. Certs and bastions add moving parts and one more thing to fail during the incident that justified the shortcut. We tiered: short-lived access on sensitive boxes, named-and-alarmed keys elsewhere. Perfection is a budget line, not a value.
• The alarm will page you once legitimately, at 2 a.m., during a real incident. Thank it. That page is the system proving it can still tell the difference.

The door remembers. You don't have to.
Go cat the file — the actual one, on the actual oldest box, right now. Count the lines; count the owners who are gone. Ours was nine and five. We deleted four, adopted one (the CI key got a name, a rotation, and an apology), and the ninth — the comment-less line — turned out to be me, from a forgotten 2022 incident.

The door remembered me too.