No :latest for firmware: five verbs for Monday morning
Originally published on November 19, 2026 at https://bruno.verachten.fr/2026/11/19/no-latest-for-firmware-five-verbs-for-monday-morning/ Above This Line: the lift directory 0. The lift ride -1. One setting changed,
Originally published on November 19, 2026 at https://bruno.verachten.fr/2026/11/19/no-latest-for-firmware-five-verbs-for-monday-morning/
Above This Line: the lift directory
-1. One setting changed, two lines moved (coming soon)
▶ Monday morning. No
:latestfor firmware (you are here)RISC-V. Down to the last binary (coming soon)
Previously on Above This Line: in the lift ride, we went from the VM down to the silicon and found the bottom of the building red: closed firmware, closed processors, keys held by someone else. Then in the roommate in the closet, we booted an open BMC on a laptop and asked it who it was. Lovely. But your production rack is still red at the bottom. So what do you actually do about it on Monday?
Photo: a coal miner’s hard hat from a museum display, by Mikepascoe, CC BY-SA 4.0, resized.
Monday morning, coffee in hand
You know the digest of every image you run in production. You have SBOMs, scanners, a policy that rejects anything unsigned. If someone pushes :latest to prod, a bot complains before a human even notices. (Bless that bot.)
Now, quick: which firmware version is the BIOS of your third database server running? Its BMC? Its network cards?
Yeah. Me neither, for most of my machines.
First, let’s set expectations. I’m not going to ask you to flash coreboot onto production on Monday. Your security officer would track me down, and they’d be right to. The red floors stay red. What we can change is whether we see them.
« On ne rend pas le rouge vert, on le rend visible. » We don’t turn red into green. We make it visible.
Visible, measured, and no longer trusted blindly. That’s the whole programme, in five verbs: Inventory, Measure, Attest, Pin, Isolate. None of them is exotic. You already do all five for containers. We’re just going to point them one floor lower (well, four floors lower).
1. Inventory: you can’t watch what you don’t know you have
The dumbest verb, and the most underrated. On any Linux machine, two commands get you started:
fwupdmgr get-devices # firmware that fwupd can see
sudo dmidecode -t bios # the system firmware version
fwupd is the firmware update daemon on Linux. get-devices lists every device it can see on the machine, with its firmware version: the system firmware, plus whatever else fwupd knows how to talk to on that particular box. dmidecode reads the system firmware’s own tables and gives you its vendor, version and release date. What should you look for? Version numbers you can compare across machines, anything that doesn’t match the rest of the fleet, and anything you didn’t know was there. (That last category is usually the interesting one.)
One blob neither command will show you: the processor’s own microcode, the patch the chip maker ships to fix bugs and security holes in the CPU after the fact. The CPU forgets it at every power cut, so it gets reloaded at every boot, by the BIOS or by your kernel, very early, from the initrd. And it reaches you like everything else: as a distro package (amd-ucode-firmware, on my Fedora laptop). Ask for its version:
grep microcode /proc/cpuinfo | sort -u
You’ll get a revision number, normally one line (more, if your cores disagree, and that would be worth a look). Not the code: Intel’s and AMD’s licences both forbid disassembling it. Still, a revision number is exactly what an inventory wants.
Then the BMC. Go and get its firmware version too. If it speaks Redfish, the manager resource has a FirmwareVersion field (on the emulated board in the BMC post, it was filled in). Not every BMC bothers. If yours doesn’t, or doesn’t speak Redfish at all, the web interface will tell you. Somewhere. (Happy hunting.)
And then the important bit: put all of it in the same inventory as everything else. Same database, same dashboards, same review as your container SBOMs. Firmware isn’t a separate religion. It’s software that happens to run before yours.
How many lines should you expect? On one real server I looked at for the talk, counting the BIOS, the BMC, the network cards, the disk controller and the drives, I got to at least ten distinct firmware images in one box. Ten. In one box. Most of them closed.
One more line that’s easy to forget: what’s missing. That same server had no TPM. "No TPM" is not an absence of data. It’s an inventory line, and as we’ll see in a minute, it’s the line that tells you what you can’t do next.
And on Intel, read that line twice if you’ve ever switched the ME off. Intel’s firmware TPM, PTT, is one of the ME’s own features (it’s listed among them in Intel’s CSME white paper), so it stops along with it. Dasharo’s ME menu says so in plain words: "with ME disabled, the firmware TPM will no longer be functional". A dedicated TPM chip keeps working. But close one door on floor -3, and you may find you’ve switched off the camera on floor -1. (Nobody said this building was simple.)
2. Measure: the bouncer and the camera
Secure Boot and measured boot sound like the same thing. They aren’t.
Secure Boot is the bouncer. Anything not signed by a key it knows doesn’t get in. « Tu as des baskets, tu rentres pas. » You’re wearing sneakers, you’re not getting in. It decides at the door, and it doesn’t keep a record of who it let through.
Measured boot is the security camera. It lets everyone in, but it films them: each boot step the firmware measures gets recorded in the TPM, and the event log tells that story afterwards. On Linux, you can read that log with tpm2-tools:
sudo tpm2_eventlog /sys/kernel/security/tpm0/binary_bios_measurements
I’m deliberately not showing you what comes out. That’s for the next one: in the floor -1 post, I swap two boot entries in a machine’s firmware, reboot, and count what moves in the log. (Place your bets now. One line? More? None?)
And the server with no TPM from the inventory? No TPM, no camera. You measure nothing. Which is why that inventory line matters: it’s the first thing on the list of things you can’t do on that box.
3. Attest: a camera nobody watches is decoration
A security camera that records to a disk nobody ever looks at is a very expensive piece of decoration. Same with measurements. They’re only useful if someone, somewhere else, compares them against what you expect.
That’s remote attestation, and there’s a project for exactly that, in a foundation you already know: Keylime, a CNCF project (Sandbox level), Apache 2.0, born at MIT’s Lincoln Laboratory. (Open source code keeping an eye on closed firmware. I do enjoy the irony.) An agent on each machine sends its TPM measurements. A verifier, on another machine, compares them with your policy. If they stop matching, the verifier sends revocation messages. What happens next depends on how you wire it in: you can listen for those events and pull the node out of rotation, put it in quarantine, page someone. Zero trust, all the way down to the firmware.
Now, be careful about what this buys you. Attestation tells you that something changed. It doesn’t tell you the firmware is trustworthy. It doesn’t tell you the firmware is open. A closed blob stays closed; it just doesn’t change behind your back unnoticed anymore, at least for the steps your TPM measures. That’s not "everything’s fine" versus "everything’s broken". It’s "something moved". Which, for a floor you can’t read, is already a lot.
4. Pin: friends don’t let friends use latest
You refuse :latest in production for your images. Why would you accept it for firmware? Friends don’t let friends use latest, and that goes for the BIOS too.
So, pin. Know which version you run, choose when it changes, and change it on purpose. The LVFS is the portal where hardware vendors upload their firmware updates for fwupd to apply. Where your hardware is supported, get it from there, rather than grabbing a binary by hand from some download page at two in the morning and pushing it straight onto a production box. (We’ve all been tempted. Two in the morning is when all the bad ideas look reasonable.)
And remember that microcode from the inventory? The copy your kernel loads doesn’t come from the LVFS. It comes from a package. On my laptop, a perfectly ordinary dnf update replaces firmware files in /usr/lib/firmware, microcode included. Nothing gets flashed: the files just change under you. So pinning firmware also means pinning those packages, and canarying them like the rest.
Then do what you’d do for any other software: canary it. Put the new version on an isolated test machine first, check that nothing odd happens, and only then roll it out to production. Boring. Effective. Exactly what you already do for everything above floor 0.
And if your vendor doesn’t publish on LVFS? Ask them. Ask what the new versions are, whether you can test them, whether they’re signed and attested. While you’re at it, ask for a firmware SBOM. They may not have one. The question alone moves things. Remember TP-Link in the hub post: an FCC consent decree committed it to work with the open source community, and on an Archer A7 sold in the US, OpenWrt installs from TP-Link’s own update page. I can’t prove the two are connected. But asking certainly didn’t hurt.
5. Isolate: the roommate gets their own cave
The fifth verb brings back our roommate in the closet, the BMC.
Raise your hand if at least one of your servers has its BMC on the same network as production. (You don’t have to answer. Your security officer might be reading this.)
Remember the buses? Oslo drove them into a mountain to cut them off from the world. Your BMC deserves a cave of its own:
a separate management network, never the production one;
restricted access, so only the people and machines that need it can reach it;
default accounts changed (no, not later, now);
and before your next purchase, check the list of platforms OpenBMC supports. If your machine is on it, that’s one of the rare floors with an open alternative already. Ask the salesperson too. Their answer alone will tell you something.
If your servers live at someone else’s place
Cloud-native folks, I haven’t forgotten you. All of this concerns you too. Floors -1 to -4 still exist under your cluster. You just don’t hold the keys anymore. So what can you do on Monday?
One: turn on the virtual TPM. The three big clouds each offer one: Shielded VM at Google, NitroTPM at AWS, Trusted Launch at Azure. Same treatment for all three, and no ranking (I’m not picking favourites today). Your VM does its own measured boot, and you can attest its boot chain. Its own chain, mind you. Not the host’s firmware underneath.
Two: confidential VMs. On AMD it’s SEV-SNP, on Intel it’s TDX. The processor encrypts your VM’s private memory, so the host only sees noise there (it still runs the VM and handles its I/O). Managed Kubernetes nodes can run that way too, for example with Confidential GKE Nodes. And on AMD, guess who produces the attestation report? The SEV firmware, running in the PSP, our old acquaintance from floor -3. You trust the host less and the chip maker more. That’s progress. It isn’t openness.
Three: ask your provider. Who updates the firmware and the BMCs of my hosts? How often? With what proof? Same as the firmware SBOM: just asking the question moves the line a little.
The printable checklist
Print it, stick it next to the coffee machine, tick it on Monday. (Or Tuesday. I won’t tell.)
Inventory
[ ] Run
fwupdmgr get-devicesanddmidecode -t bioson every machine type in the fleet[ ] Record each BMC’s firmware version
[ ] Record each machine’s microcode revision (
grep microcode /proc/cpuinfo | sort -u)[ ] Store it all next to your container SBOMs, missing TPMs included
Measure
[ ] Check which machines have a TPM, and which don’t
[ ] Read one event log with
tpm2_eventlog, just to see what’s in it
Attest
[ ] Try Keylime on two test machines: one agent, one verifier
[ ] Decide what a revocation event should actually do to a node
Pin
[ ] Check which of your machines get firmware through LVFS
[ ] Pin the distro’s firmware and microcode packages like any other dependency
[ ] Canary every firmware update on an isolated test box first
[ ] Ask each vendor for a firmware SBOM
Isolate
[ ] Move every BMC to a separate management network
[ ] Change every default account
[ ] Check OpenBMC’s supported platforms before the next purchase
Takeaways
Inventory first. A firmware you don’t know about is a firmware you can’t watch, and "no TPM" is a line in the inventory too.
The bouncer isn’t the camera. Secure Boot refuses at the door; measured boot records who came in.
Watch the camera. Attestation only tells you something changed. That’s exactly what you need from a floor you can’t read.
No
:latestfor firmware. LVFS where you can, a canary box always, and a polite question to every vendor.Give the BMC its own cave. Separate network, restricted access, no default passwords.
None of these verbs makes the management engine readable. Red stays red. But you go from "I trust it" to "I know when it changes", and for anyone who runs infrastructure, that’s a big deal.
audit("monday") = ? · but now you’ll know when it changes
Next in the series: floor -1, where I change one boot setting on a real machine and catch it on camera.
The talk video (in French; the five verbs start here) and the slides (sources in the appendix) are online.
Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.