Originally published on November 5, 2026 at https://bruno.verachten.fr/2026/11/05/above-this-line-a-lift-ride-under-your-hypervisor/

• ▶ 0. The lift ride (you are here)

• -1. One setting changed, two lines moved (26 November)

• -2. The roommate in the closet (12 November)

• Monday morning. No :latest for firmware (19 November)

• RISC-V. Down to the last binary (3 December)

Photo: Orphan Girl Mine, Butte, Montana, by James St. John, CC BY 2.0, resized.

In October 2025, Ruter, the public transport operator in Oslo, asked itself a very simple question: who actually controls our electric buses? So they took two of their own, a brand-new Yutong built in China and a three-year-old VDL from the Netherlands, and drove them into an isolated environment inside a mountain. No network. Just two buses, some probes, and a lot of rock.

What did they find? The new one was built for over-the-air updates: its manufacturer could reach each bus directly, battery and power-supply control system included, over the mobile network, through a Romanian SIM card. (A Chinese bus, in Oslo, talking through Romania. I’ve stopped trying to make the geography work.) The old one had no OTA at all.

Ruter’s conclusion, in its own publication, is that the manufacturer could, in theory, stop the bus or render it unusable. In theory. Nobody switched off a bus. But keep the shape of it in mind: closed firmware that somebody else can update.

Hold on to the bus. It comes back.

Ruter wondered who controls its buses. I asked the same thing about servers, mine and yours, to open my talk at Cloud Nord 2026 in Lille. This is the written version (the slides are online, in French, sources in the appendix).

Hands up, hands down

First question to the room: who has ever opened a BMC’s web interface? (The BMC is the little board that lets you power a server on remotely, and a lot more.) A handful of hands went up. Then: who has ever read the code running inside it?

Every hand dropped.

That’s the whole talk, in two questions. (If you only run cloud VMs and have never touched a BMC: lucky you, there’s a paragraph for you further down.)

I’m Bruno, community manager at Vates (we make XCP-ng and Xen Orchestra). At Cloud Nord 2025 I was turning Android phones into Jenkins agents; this year I came back to teach people to distrust their real, perfectly legit servers. « Je progresse dans la paranoïa, c’est l’âge. » I’m getting more paranoid. It’s my age.

I’m no firmware expert, by the way: nearly everything below, I discovered from summer 2026 onwards, so this series is me doing my homework in public.

The method: the "descenseur"

We take the lift from the VM all the way down to the silicon. On stage I called it « le descenseur » (French says ascenseur, and we were only ever going down). Think strace, except we don’t stop at the kernel. On each floor, we call an imaginary function:

audit(floor) ==  0;  // readable: you read it, you patch it      -> green
audit(floor) ==  ?;  // partly: an open door exists, few use it  -> orange
audit(floor) == -1;  // EACCES: you can't read it or remove it   -> red

EACCES is what Linux hands you when it refuses you a file. At Cloud Nord the room voted with raised hands before each verdict. You can vote in your head. I won’t count (promise).

Somewhere on the way down there’s a line. Above it, you read the code. Below it, you take the vendor’s word for it. That’s the line in the title, and on every floor we’ll check which side of it we’re standing on (spoiler: it doesn’t improve).

One more map, because other people’s writing uses it. x86 counts rings from 3 (apps) down to 0 (kernel), and the hypervisor gets "ring -1", which is a figure of speech, not a piece of hardware. Libreboot calls the ME and the PSP (floor -3, patience) "ring -2" or "ring -3". Not a ring at all: another processor, reaching the centre. Rings aren’t floors, but they point the same way. Down. Audit your hypervisor all you like, you still haven’t audited the bottom.

Floor 0: everything’s fine

Ground floor: your VMs, and the hypervisor underneath. Xen, KVM: GPL, public code. You read it, you patch it, you send the patch upstream. audit("hypervisor") == 0. Green.

Everything’s fine…​ until you power the machine on.

Even at home. On stage I’d promised to put my own product in orange, so here goes: the hypervisor’s code is green, but booting it is another story. Secure Boot is the bouncer at the firmware’s door ("Sneakers? You’re not getting in."). Turn it on, and an XCP-ng host won’t boot, installer or installed. I hit it in my lab, on an OptiPlex. First I blamed the USB stick. Then the ISO. Then the IP KVM. Best part? The ISO really was corrupt as well. Two problems, one symptom. With a clean image and Secure Boot on:

All bootable devices failed Secure Boot verification.

Not a bug.

So what is it, then? Partly signatures: XCP-ng’s boot chain isn’t signed with a key your firmware trusts out of the box. The 8.1 release notes say Secure Boot relies on proprietary packages, the forum says it would need a valid Microsoft certificate, and the request has been open since 2019. Guests have had Secure Boot since 8.2.1 (same name, different thing, just to keep you on your toes). And enrolling your own keys? Then you sign the whole chain, every update included.

« Le Secure Boot ne vérifie pas que le code est ouvert. Il vérifie qui l’a signé. » Secure Boot doesn’t check that the code is open. It checks who signed it.

Who holds the keys. Sounds a bit like the bus, doesn’t it?

Floor -1: open upstream, closed at your place

Knock knock. On floor -1, the firmware that boots the machine answers. And your vendor’s UEFI? A closed binary. On 22 October 2026, TianoCore published GHSA-99cj-7hg9-mp4m, a critical pre-boot flaw in the PXE driver of EDK2, the reference UEFI implementation. The fix had been on master since 15 October. In your server, though, it lands when your vendor ships a new binary. Not before. (Dan Goodin at Ars Technica explains why that reaches so many servers: most UEFI suppliers build on EDK2.)

« Ouvert en amont, fermé chez vous. » Open upstream, closed at your place.

Why orange, then, and not red? Because doors exist. coreboot does the bare minimum of hardware bring-up, then hands over to a payload such as TianoCore, which is EDK2, compiled by you this time. Libreboot is to coreboot what Debian is to Linux: a distribution of it, the purist one, mostly on older machines (hello, ThinkPad X200 from 2008). On recent Intel, memory init goes through Intel’s FSP blob, and the Libreboot FAQ says replacing it would take the world’s best developer years. (Not me, then.) Dasharo, also coreboot-based, ships on laptops and on a handful of server boards with paid support.

Dasharo can even stop the Intel ME after boot, with the HAP bit, a flag that tells the ME to halt once the hardware is up. Stopped. Not removed. We’ll meet it again two floors down.

Floor -2: the roommate in the closet

Who has ever updated their BMC’s firmware? Who knows which OS runs inside it? (Before this talk, not me. Not a clue.)

Remember the Romanian SIM? Your server has one too: the BMC. Power, console, ISOs, its own OS and its own network, even while the server is off. A roommate living in your closet, with the house keys. The keys come with the server, though not always all of them: on one machine I was lent, mounting an ISO needed a paid licence. That same machine, with 1.5 TB of RAM, spent more than four minutes in POST (the power-on self-test, the machine checking itself) before even looking for its devices. Four minutes of code you can’t read, and your OS isn’t even born yet. Long enough to wonder what it’s doing in there.

Orange anyway, thanks to OpenBMC, an open source Linux distribution for BMCs. Facebook started it in 2014, and Facebook’s own 2015 engineering post says Facebook runs it in production. My servers? None of them. So I booted the public image for Bletchley, a Meta board, in QEMU on my laptop. That one gets its own post.

Floor -3: knock knock, who’s there?

Guessing game. A reasonable one-socket server: how many processors inside? The invoice says one. The invoice is an optimist. There’s the CPU you paid for, the BMC, your SSD’s controller, and the one that answers when you knock on floor -3.

Yes, a fourth one. And it was there before you.

On Intel, it’s the Management Engine: a separate little computer in the chipset since 2006. It has handled part of hardware init since Nehalem, its firmware is signed by Intel, and on recent chips it runs a 486-derived core running MINIX (my favourite piece of trivia in the whole talk). Without valid signed firmware, the machine shuts down after 30 minutes, according to people who took it apart.

Half an hour of freedom.

And AMD? No favourites here. The PSP is an Arm core on the same die as your x86 cores, and per the coreboot and Libreboot docs its firmware must be AMD-signed, or your x86 cores never start.

Not thirty minutes: zero.

Your processor has two brains, and you only hold the keys to one.

This isn’t Intel bashing or AMD bashing (I just did both, to be fair), it’s architecture. And these locks exist to protect you: anti-rollback, for one, stops anyone dragging you back to an older, vulnerable firmware. They just don’t hand you the key. (AMD friends reading this: « cœur sur vous », much love. I meant it on stage too.)

So I looked at a real one: an AMD EPYC Genoa belonging to a partner (not the loaner from floor -2), booted from a live key. The kernel reported its PSP fused, debug locked and anti-rollback on, all of it protective, and Secure Boot had no keys enrolled at all.

No TPM fitted on that box, so fwupd’s HSI scale (built with PCs in mind) stopped it at HSI:0, on step one. Ouch. Count BIOS, BMC, network cards, disk controller and SSDs, and that’s at least ten closed firmwares. « Je ne jette la pierre à personne. » I’m not throwing stones at anyone. Yours probably looks much the same.

To be fair, I ran the same checks on my own Intel NUC: ME version 18, manufacturing mode locked, and Boot Guard fused to the board vendor’s key, so only that vendor’s firmware boots (on my NUC, yes, that one stung). It has a TPM (the small chip that keeps track of what booted), and fwupd gives it HSI:3, a better score earned partly because of those vendor locks.

« Plus sûr, oui. Plus ouvert, non. » Safer, yes. More open, no.

audit("ME/PSP") == -1. Red. Can’t read it, can’t remove it.

Two minutes, and it tells you a lot: fwupdmgr security. The HSI scale was designed for laptops and desktops, so read a server’s score with that in mind.

"But Bruno, there’s me_cleaner!" (Nobody said it. I felt it.) me_cleaner strips much of the ME firmware, and the HAP bit stops the ME once the hardware is up, but by then it has already done its job. Never run it on a machine you need. For the PSP, says Libreboot, there’s no equivalent.

In the cloud, this floor is still under your cluster (told you there was a paragraph for you). On AMD, a confidential VM’s SEV-SNP attestation report comes from firmware running in, you guessed it, the PSP. You trust the host less and the chip maker more.

And here’s the part I genuinely like: with openSIL, AMD publishes its silicon initialisation code as open source, and coreboot boots recent EPYC chips with it on a few boards. A real step, and AMD is the one taking it. The PSP itself still runs AMD-signed firmware, so openSIL opens the initialisation, not the PSP. But a door that wasn’t there is now ajar: not orange yet, but the first crack in the red.

Floor -4: the bottom of the cave

Knock knock…​ nothing. On x86 the instruction set is proprietary and core designs are secret. Nobody expected to read a Xeon’s RTL (the hardware description the chip is built from) this morning. (Good. I didn’t bring one.) audit("ISA") == -1. Red.

Not a law of physics, though: OpenPOWER is one open ISA, and on Raptor’s machines the platform firmware sources are published and auditable. There’s another open ISA, and I’m keeping it warm for later.

Express ride back up

Here’s the map you voted on: green at the top, orange in the middle, red at the bottom. So is open source pointless? (Deliberately provocative.)

No.

« L’open source, c’est une direction. Pas une case à cocher. » Open source is a direction. Not a checkbox.

I skipped a few floors on the way down (my talk, my rules). Your SSD has its own processor and closed firmware, and in 2019 Radboud University researchers showed at IEEE S&P that several self-encrypting SSDs gave up their data without the user’s password.

In October 2026, an intermediate certificate of the certificate authority eMudhra turned out to have a private key equal to 4. Yes, four. Mozilla’s bug 2083449 tracks eight weak intermediates; eMudhra attributes it to a script error. It was spotted because the public half was public (I checked it myself, I couldn’t resist). A 4 inside SSD firmware? Nobody would ever see it.

The line moves both ways. After 2015 FCC rules on 5 GHz radios, TP-Link locked out third-party firmware entirely. The 2016 consent decree brought a \$200,000 penalty (for its own firmware exceeding power limits, not for open source) and a commitment to work with the open source community and chip makers on reopening the door. On an Archer A7 sold in the US, OpenWrt installs from TP-Link’s own update page. Are the two connected? I can’t prove it.

Monday morning: make the red visible

I won’t ask you to flash coreboot onto production. Your CISO would find me, and rightly so.

« On ne rend pas le rouge vert, on le rend visible. » We don’t turn red into green. We make it visible.

Five verbs. Inventory: list every firmware you have, and "no TPM" is a line in that list too. Measure: Secure Boot, our bouncer from floor 0, checks who signed each step; measured boot is the CCTV camera, recording each step in the TPM. Attest: have something else compare those recordings with what you expect, because a camera nobody watches is decoration. Pin: friends don’t let friends use latest, firmware included. Isolate: the BMC gets its own network, and nobody else’s. Red stays red, but you go from "I trust it" to "I know when it changes". That’s not nothing. Want to watch one setting move two lines in the TPM log? That’s the floor -1 post. The rest of Monday morning gets its own.

The line moves

What if, somewhere, floor -3 opened when you knocked?

Same lift, different building: RISC-V. On floor -3, where x86 keeps its ME, RISC-V puts OpenSBI, the open source reference firmware running at the machine’s most privileged level: audit("OpenSBI") == 0. On floor -4, the ISA is an open specification. The core? It depends. SiFive sells the P550 as licensed IP; CVA6 publishes its RTL, though trying it on an FPGA needs a closed vendor tool. An open ISA isn’t an open core (a confusion I used to make myself, so no judgement here).

On my first Banana Pi BPI-F3, the bootloader is GPL source, at the exact commit my distribution says it built. OpenSBI is open source too, though built from a contributor’s tree. Lovely. But a second, 32-bit core runs esos.elf, a prebuilt binary my distribution copies in, even though SpacemiT publishes its source. My most open board, and still a binary nobody here rebuilt. It gets its own post.

The SpacemiT K3 announces everything a hypervisor needs: H for the CPU, AIA for interrupts, an IOMMU for devices. 16 cores!

Well…​ 8.

Only eight of the sixteen have the H extension. Julien Aubé ran the commands on his board for me, and on that one board AIA is there while Linux sees no IOMMU.

Still, RISC-V servers are starting to look like servers.

No board? Scaleway rents bare-metal RISC-V, the Elastic Metal EM-RV1, launched in February 2024. According to Scaleway, it boots U-Boot, OpenSBI, then Linux, on top of "minimal proprietary code". RISE’s GitHub runners run on these machines by default, and most of my riscv64 porting goes through them.

The bus comes back

Told you. After Oslo, Denmark’s Movia had EY audit electric buses from several manufacturers, and found no basis to consider Chinese buses less cybersecure than the others. And the manufacturer that could least demonstrate its cybersecurity management? The Dutch one, VDL: the maker of Oslo’s bus without OTA.

EY gives both halves: without OTA, nobody takes over the fleet remotely, and nobody patches, monitors or audits it either. Not having OTA doesn’t make you auditable.

« Injoignable, ce n’est pas vérifiable. » Unreachable isn’t verifiable.

The risk was never the bus’s passport. It’s closed firmware that someone else writes and signs, at the heart of a critical system. And you have several in every server.

Takeaways

• Know which floor you’re on. For each layer, ask whether you can read it, patch it, or only believe it.

• Spend two minutes on fwupdmgr security. It opens nothing, but it tells you where you stand.

• Treat firmware like container images. Inventory it, pin it, never :latest it.

• Put the BMC in its own cave. Separate management network, restricted access, default accounts changed.

• Watch the bottom of the stack. openSIL and RISC-V show the line can move down.

Thanks to the Cloud Nord crew for having me back, to Julien Aubé for running commands on his K3 for my slides, and to Ludovic Henry for the photo. The video (in French) and the slides are online. Next in the series: floor -2, where I boot a BMC on my laptop.

audit("this post") == 0;  // sources in the slides' appendix