Moving a production API to Hetzner, part 2: hardening and configuring the server

Series: Part 1: choosing and creating the server · Part 2: hardening and configuring it · Part 3: migrating data and cutting over

In part 1 we created a Hetzner CX23 behind a Cloud Firewall, with a
cloud-init config meant to harden it on first boot. This part does that hardening
by hand, one step at a time, explains every line, and does it without ever
risking a lockout.

(cloud-init is the tool in every cloud image that runs once on first boot and
applies the config you paste into the Cloud config box: users, files,
packages, commands. Part 1, section 5 has the full config and how it works.)

Why by hand, if cloud-init can do it?

• You'll know what each setting does, so you can debug it when it misbehaves.
• It works anywhere: on a provider without cloud-init, on a server someone else created, or when cloud-init failed or was never given its config.
• It doubles as the verification. If you did use cloud-init, skip to section 5 ("Verify like an attacker") and run the checks. They prove the config was applied, not just pasted.

Placeholders, as before: the new server is 198.51.100.20 and the old one is
203.0.113.10.

1. ~/.ssh/config: two servers, two keys, short commands

The problem it solves

During a migration you constantly talk to two machines, each with its own
address, port and key. Typing
ssh -p 2222 -i ~/.ssh/hetzner_ed25519 [email protected] every time is slow
and error-prone. ~/.ssh/config gives each server a short name:

Host old-box
  HostName 203.0.113.10
  Port 2222
  User ubuntu
  IdentityFile ~/.ssh/oldhost_key.pem
  IdentitiesOnly yes

Host new-box
  HostName 198.51.100.20
  Port 2222
  User ubuntu
  IdentityFile ~/.ssh/hetzner_ed25519
  IdentitiesOnly yes

Now ssh new-box means all of that. So do scp file new-box:, rsync … new-box:,
and git clone new-box:…. Every OpenSSH-based tool reads this file.

You can append a block without opening an editor:

printf '\nHost new-box\n  HostName 198.51.100.20\n  Port 2222\n  User ubuntu\n  IdentityFile ~/.ssh/hetzner_ed25519\n  IdentitiesOnly yes\n' >> ~/.ssh/config

Why IdentitiesOnly yes

Without it, the client offers the server every key in your ssh-agent, one by
one, before the one you named. sshd allows only a few attempts per connection
(MaxAuthTries, default 6). With enough keys loaded you get
Too many authentication failures before the right key is ever tried.
IdentitiesOnly yes means "offer only the IdentityFile named here". It also
stops you from accidentally presenting your work key to a personal server.

2. First login, and the plan for not locking yourself out

On a stock Hetzner image, before any hardening, there is exactly one account you
can reach: root, on port 22, with the SSH key you selected at creation.
Hetzner writes that key into /root/.ssh/authorized_keys and sets no root
password. So that's where we start:

ssh -p 22 -i ~/.ssh/hetzner_ed25519 -o IdentitiesOnly=yes [email protected] 'id; ss -tln'

(If cloud-init did run, root login is already disabled and this is refused. That's
your cue to skip to the checks in section 5.)

What happened to me: I created the server and forgot to paste the cloud
config. My first ssh -p 2222 got Connection refused, because nothing was
listening on 2222 (section 6 explains how to read that). Recreating the server
would have meant giving up a CX23 that was close to sold out, so I hardened it
by hand, as below.

Running a command after the host ('id; ss -tln') executes it remotely and
returns. No interactive shell is needed, which makes every step repeatable and
easy to paste into notes.

The lockout problem

Hardening SSH means changing the very thing you're connected through. Get one
line wrong, for example disabling root before the new user's key works, and the
next connection fails. On a cloud server, your remaining way in is the provider's
web console. Remember from part 1 that Hetzner sets no root password when you
use an SSH key, so even the console needs a password reset first.

The fix is to make changes in two passes, with a test in between, so that at
every moment at least one known-good way in still works:

3. Pass 1, explained block by block

The whole pass is one script, sent from the laptop and run on the server as root.
'bash -s' tells the remote bash to read its script from standard input, and the
heredoc (<<'EOF' … EOF) supplies it. The quotes around 'EOF' stop your
local shell from expanding $variables inside, so they reach the server as
written.

ssh -p 22 -i ~/.ssh/hetzner_ed25519 -o IdentitiesOnly=yes [email protected] 'bash -s' <<'EOF'
set -euo pipefail
export DEBIAN_FRONTEND=noninteractive

# --- user ---
id ubuntu >/dev/null 2>&1 || adduser --disabled-password --gecos "" ubuntu
usermod -aG sudo ubuntu
echo 'ubuntu ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/90-ubuntu && chmod 440 /etc/sudoers.d/90-ubuntu
install -d -m 700 -o ubuntu -g ubuntu /home/ubuntu/.ssh
install -m 600 -o ubuntu -g ubuntu /root/.ssh/authorized_keys /home/ubuntu/.ssh/authorized_keys

# --- packages ---
apt-get update -q
apt-get -y -q -o Dpkg::Options::=--force-confold upgrade
apt-get -y -q install ufw unattended-upgrades

# --- sshd ---
cat > /etc/ssh/sshd_config.d/10-hardening.conf <<'CONF'
Port 22
Port 2222
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitRootLogin prohibit-password
PubkeyAuthentication yes
CONF
sshd -t
systemctl daemon-reload
systemctl restart ssh.socket || true
systemctl restart ssh || true

# --- docker log rotation (read when Docker is installed later) ---
mkdir -p /etc/docker
echo '{"log-driver": "json-file", "log-opts": {"max-size": "10m", "max-file": "3"}}' > /etc/docker/daemon.json

# --- host firewall ---
ufw default deny incoming; ufw default allow outgoing
for r in 22/tcp 2222/tcp 80/tcp 443/tcp 443/udp; do ufw allow $r; done
ufw --force enable

# --- swap ---
if [ ! -f /swapfile ]; then fallocate -l 2G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile && echo '/swapfile none swap sw 0 0' >> /etc/fstab; fi

ss -tln | grep -E ":(22|2222) "
EOF

set -euo pipefail

These are bash's "fail loudly" settings: -e stops at the first failing command,
-u treats an unset variable as an error, and pipefail makes a pipeline fail
if any stage fails, not just the last. Part 3 shows pipefail biting back,
too.

The user

• adduser --disabled-password: the account has no password at all, so it can only be reached with a key. id ubuntu || … makes the line safe to run twice.
• sudoers.d/90-ubuntu with NOPASSWD: ubuntu can run sudo without a password, which is what Ubuntu cloud images give their default user. It's a trade-off: if someone gets this account they get root. But a password-less account with a sudo password needs a password set somewhere, and that's a secret to manage. With key-only SSH, the key is the boundary. chmod 440 because sudo refuses sudoers files that others can write to.
• install -d -m 700 / install -m 600: create .ssh and copy root's authorized_keys with the right owner and permissions in one step. The permissions aren't cosmetic. sshd's StrictModes (on by default) ignores an authorized_keys file that other users could modify, and the symptom is a baffling Permission denied (publickey).

Packages

DEBIAN_FRONTEND=noninteractive and --force-confold stop apt from opening an
interactive "a config file changed, which version do you want?" prompt in the
middle of a non-interactive SSH session. confold keeps the existing file.

sshd: drop-in files and "first value wins"

Instead of editing /etc/ssh/sshd_config, we add a file in
/etc/ssh/sshd_config.d/. The main config includes that directory near its
top, and the files load in name order. For most settings sshd keeps the
first value it reads, so 10-hardening.conf beats the image's own
50-cloud-init.conf and anything below in the main file. Package upgrades never
touch your drop-in, either.

Two commands make it safe:

• sshd -t tests the configuration and exits non-zero on a syntax error. Because of set -e, a typo stops the script before the restart, so the old, working sshd keeps running.
• sshd -T (used later) prints the effective configuration after all files are merged. That's the only reliable way to know which value won.

Why ssh.socket and not just ssh

Since Ubuntu 24.04, sshd is socket-activated: systemd holds the listening
socket (ssh.socket) and starts sshd when a connection arrives. Which ports that
socket listens on is generated from sshd_config by a systemd generator, and
the generator only runs on systemctl daemon-reload. So changing Port takes
three steps: reload (regenerate the socket), restart ssh.socket (rebind the
ports), restart ssh. Skip the reload and sshd's config says 2222 while nothing
listens there. || true keeps the script going on releases that aren't
socket-activated.

The script ends by printing what's listening. We got
0.0.0.0:22, 0.0.0.0:2222, [::]:22 and [::]:2222.

ufw: the host firewall

ufw ("uncomplicated firewall") is a front end for the kernel's netfilter
(nftables). Default deny incoming and allow outgoing, then open exactly
what's needed. 22 is allowed before ufw enable, otherwise enabling it would
cut the very session doing the enabling.

Docker bypasses ufw for published ports (see part 1), so ufw protects sshd and
anything else running directly on the host, and the Cloud Firewall protects
everything. Two layers, each covering the other's blind spot. That's
defence in depth.

Swap

A 4 GB server with no swap handles a memory spike (a big query, a container
starting) by invoking the kernel's OOM killer, which usually kills the biggest
process: Postgres. A 2 GB swapfile turns that crash into a brief slowdown.
fallocate reserves the space instantly, mkswap formats it, swapon activates
it, and the /etc/fstab line brings it back after a reboot.

4. Test, then pass 2

Before taking anything away, prove the new way in works:

ssh new-box 'whoami; sudo -n true && echo sudo-ok'

ubuntu
sudo-ok

sudo -n means "non-interactive": fail instead of prompting. So this proves
both the key and passwordless sudo. Only now, as ubuntu, turn root off:

ssh new-box 'sudo sed -i "s/^PermitRootLogin .*/PermitRootLogin no/" /etc/ssh/sshd_config.d/10-hardening.conf && sudo sshd -t && sudo systemctl restart ssh && sudo sshd -T | grep -E "^(port|permitrootlogin|passwordauthentication) "'

port 22
port 2222
permitrootlogin no
passwordauthentication no

5. Verify like an attacker

Checking the config proves what you intended. Negative tests prove what's
enforced: try what should fail and watch it fail.

ssh -p 2222 -i ~/.ssh/hetzner_ed25519 -o IdentitiesOnly=yes [email protected] true
# [email protected]: Permission denied (publickey).

ssh -p 2222 -o PubkeyAuthentication=no -o PreferredAuthentications=password [email protected] true
# [email protected]: Permission denied (publickey).

The second line is the interesting one. We asked for password authentication,
and the server's reply lists only (publickey) as acceptable. It doesn't offer
a password prompt at all.

Then the rest of the configuration in one go:

ssh new-box 'sudo ufw status verbose; ss -tln; timedatectl show -p NTPSynchronized; swapon --show; free -h; df -h /; cat /etc/apt/apt.conf.d/20auto-upgrades; systemctl is-enabled unattended-upgrades; ls /var/run/reboot-required 2>/dev/null && echo REBOOT-NEEDED'

What to look for, and why each matters:

Automatic updates, and the reboot they don't do

unattended-upgrades installs security updates every day. During pass 1,
needrestart restarted the services whose libraries changed and listed the ones
it deferred. It doesn't reboot: a new kernel only takes effect after a
reboot, and the system flags this with /var/run/reboot-required. Automatic
reboots stay off (the default), because a reboot in the middle of a
database write is worse than a few days on the old kernel.

The best time to reboot is now, while the box is empty. It's also a free test
that everything comes back on its own:

ssh new-box 'sudo reboot'
# wait ~30–60 s
ssh new-box 'whoami; uptime; ss -tln | grep -E ":(22|2222) "; sudo ufw status | head -1; swapon --show --noheadings; ls /var/run/reboot-required 2>/dev/null || echo no-reboot-needed'

Both SSH ports, ufw and swap all survived.

6. Reading a port test: open, refused, filtered

From the laptop, without installing nmap:

for p in 22 2222 443 5432; do timeout 5 bash -c "</dev/tcp/198.51.100.20/$p" 2>/dev/null && echo "$p open" || echo "$p closed/filtered"; done

/dev/tcp/HOST/PORT isn't a real file. It's a bash feature: redirecting from it
opens a TCP connection. timeout 5 caps the wait.

There are three possible answers, and telling them apart is the most useful
network-debugging skill I know:

Two real examples from this migration:

• ssh -p 2222 → Connection refused on the first try. The Cloud Firewall was fine. sshd simply wasn't on 2222 yet, because cloud-init hadn't run.
• Port 22 → timeout from home, even though sshd listened on 22, ufw allowed it and the Cloud Firewall allowed it. Every layer on the server said yes. That points at the path: some home routers and ISPs drop outbound port 22. The way to prove it is to test from a third machine on a different network, such as the old server (ssh old-box 'timeout 5 bash -c "</dev/tcp/198.51.100.20/22" && echo open') or a phone hotspot. This is exactly why SSH listens on two ports.

The final scan read 2222 open, 443 closed/filtered (nothing serves it
yet), and 5432 closed/filtered. The database port is invisible from outside,
which is what we want.

7. Docker

ssh new-box 'curl -fsSL https://get.docker.com | sudo sh && sudo usermod -aG docker ubuntu'

• get.docker.com is Docker's official convenience script. It adds Docker's own apt repository and installs the engine and the Compose plugin. Piping a script into sh is fine when you trust the source and it comes over HTTPS; read it first if you want (curl -fsSL https://get.docker.com | less). If it doesn't yet support a brand-new Ubuntu release, add the apt repository by hand from Docker's docs instead.
• usermod -aG docker ubuntu lets ubuntu run docker without sudo. ⚠️ The docker group is root-equivalent: anyone in it can start a container that mounts / and edit any file. Here ubuntu already has passwordless sudo, so nothing changes, but never add a less-trusted user to it.

Group membership is read at login, so open a new connection to check:

ssh new-box 'docker --version && docker compose version && docker info --format "{{.LoggingDriver}}"'

Docker version 29.x
Docker Compose version v5.x
json-file

json-file confirms Docker read our daemon.json from pass 1. That file caps
each container's log at 3 × 10 MB. Without it Docker never rotates container
logs, and on a 40 GB disk the reverse proxy's access log eventually fills the
disk. Disk-full is a classic way for a small server to die in month five.

Where we are

In part 3 we move the application: copying a secrets file from server to
server through the laptop without it ever touching the laptop's disk, pulling
the image instead of building it, rehearsing a restore from backups (which turned
up a real bug), and planning a cutover with a few minutes of downtime and a clear
rollback.