My NAS runs on a Supermicro X10SDV-TLN4F — Xeon D-1541, 128 GB of ECC, a 22 TB ZFS pool, and a BIOS dated 2017. It had been up since March. I did some network configuration work one morning, rebooted, and it didn’t come back.

What it showed instead was the Supermicro splash screen with A2 in the bottom right corner, and a keyboard that did nothing. DEL wouldn’t get me into Setup, F11 wouldn’t open the boot menu, F12 wouldn’t PXE. The machine sat there indefinitely.

A2 is an AMI Aptio progress code meaning “IDE Detect” — the firmware enumerating SATA devices. That framing sent me down the wrong path for the first two hours, because “stuck detecting storage” reads like “a disk is dying.” It isn’t necessarily. A2 is the last code the firmware latched, and the firmware stops updating it once it hands off. A machine displaying A2 forever might be stuck enumerating storage, or it might be several stages further on and no longer writing new codes.

Getting into a machine whose keyboard is dead

The BMC was reachable even though the host wasn’t, which turned out to be the single most useful fact of the day. ipmitool gave me sensors, the event log, and — critically — the ability to force the next boot into Setup without pressing anything:

ipmitool -I lanplus -H <bmc> -U ADMIN -E chassis bootdev bios
ipmitool -I lanplus -H <bmc> -U ADMIN -E chassis power reset

That sets an IPMI boot flag the firmware reads at the start of BDS. It bypasses the keypress entirely. I used it maybe fifteen times over the course of the day, and without it I’d have been stuck at “the keyboard doesn’t work” with no way in. If you run Supermicro hardware and haven’t used chassis bootdev, it’s worth knowing before you need it.

The sensors were all boring, which is itself evidence:

12V     12.128      CPU Temp    50 C
5VCC     5.026      System Temp 41 C
3.3VCC   3.367      VBAT         3.226

Rails nominal, CMOS battery fine, no power overload, no drive fault, no fan fault. The System Event Log was full — 512 entries, 100% used, overflow set — and every single entry from February 2025 to the night before was a correctable ECC error, mostly on DIMMA1. That matters later, but nothing in it pointed at a boot failure.

Five theories the evidence killed

A failing drive. The obvious reading of “IDE Detect.” I got into Setup via the IPMI flag and every one of the six SATA devices enumerated with the correct model and capacity. Two 6 TB HGSTs, two 22 TB Seagates, two 1 TB Samsung SSDs. Nothing missing, nothing mis-sized.

A full /boot. The morning’s work had rebuilt the initramfs, and /boot was a 387 MB partition. This was my favourite theory for hours because the timing was perfect. When I finally got a live USB booted and looked, /boot was 61% used — 213 MB of 373 MB, all three kernels intact with matching initrds. Completely healthy. I’d been confidently wrong about it in three separate messages.

A stale NVRAM boot entry. The boot menu had four entries: UEFI OS, ubuntu, Ubuntu, and a Windows Boot Manager left over from a transplanted disk. UEFI OS was first, and it pointed at the fallback path \EFI\BOOT\BOOTX64.EFI, which chained to a 2021 copy of grubx64.efi sitting next to a 2026 shim. That’s a genuine landmine and I was pleased to find it. It also wasn’t the problem: promoting ubuntu to first changed nothing.

Boot order generally. The killer here was launching the bootloader by explicit path from the EFI Shell, which bypasses NVRAM completely:

fs0:\EFI\ubuntu\shimx64.efi

Same hang, no output. So it wasn’t the entries, the ordering, or the firmware’s choice of target. Whatever was failing was failing after the firmware handed control over.

A video artefact. By this point I wondered whether GRUB was running fine and the iKVM just wasn’t capturing the mode it selected. Cheap test: launch it, then type reboot blind. If the machine reboots, GRUB is alive and I’m staring at a stale framebuffer. Nothing happened. Genuinely hung.

The EFI Shell is a real debugger

The X10SDV has a built-in EFI Shell as a boot option, and I’d never used it for anything. It turns out to be the best tool available when you’re below the OS.

map -r lists every filesystem and block device with its full device path:

fs0: PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x4,0xFFFF,0x0)/HD(3,MBR,0x52796A31,0x800,0x32000)
fs1: PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x5,0xFFFF,0x0)/HD(2,GPT,090296AA-...,0xE1800,0x32000)

That’s readable once you know the grammar. Sata(0x4,...) is SATA port 4. HD(3,MBR,...) is partition 3 on an MBR-labelled disk. So fs0 is the ESP on the disk in port 4, and it’s partition three of an MBR disk — which told me immediately that my “modern UEFI system” had an MBR boot disk with the ESP mistyped as 0x0b (W95 FAT32) instead of 0xEF. The firmware mounted it anyway, because EDK2’s partition driver attaches to any FAT it finds, but it was wrong and had been wrong for years.

From there ls, type to read files, and the ability to execute any EFI binary by path let me bisect the boot chain properly: shim hangs, GRUB hangs, a completely different 2021 GRUB build hangs, the EFI Shell itself runs fine, Windows Boot Manager boots to a spinner. Three different loaders failing identically while two others work is a strong signal, and I still couldn’t turn it into a diagnosis.

Giving up on the diagnosis

At some point the cost of another theory exceeded the cost of rebuilding, and I stopped. That’s a judgment call rather than a discovery, and it’s worth naming as one: I had a machine down, a boot stack I didn’t trust, and no hypothesis left that the evidence hadn’t already killed.

The stack was worth replacing regardless. Count the indirections between firmware and kernel on the old setup: an EFI boot variable naming a device path, shimx64.efi, grubx64.efi, a stub grub.cfg on the ESP, a search.fs_uuid call resolving a filesystem UUID to a device, and finally the real grub.cfg on /boot. Six steps, four of them a lookup that can resolve to the wrong thing or to nothing.

The search.fs_uuid step is the one I’d been quietly relying on. GRUB’s search is a scan, so grub-install writes a hint next to it — hd2,msdos1, an ordinal position on a specific disk — to skip the scan in the common case. That’s a hardcoded disk index inside a config file, on a machine with six SATA bays, four of them hot-swap. Move a drive and the hint points somewhere else. GRUB falls back to scanning and usually recovers, but “usually” was doing real work in that sentence.

/boot was the slower problem. 387 MB, and not full — 61% — but Ubuntu keeps multiple kernels and initrds have grown past 100 MB each. It had room for about three. The partition was sized when initrds were a tenth of that, and it was going to become an outage during some future kernel update, in the middle of something unrelated.

What replaced it

GPT on both SSDs, a correctly typed 2 GB ESP on each, and the rest of each disk as a member of an mdraid1 mirror:

sdf1 ─ ESP  /boot/efi                    sde1 ─ ESP  /boot/efi2
sdf2 ─┐                                  sde2 ─┐
      └─ md127 (raid1, 974429888 blocks) ──────┘
           └─ cryptroot (LUKS2, Clevis/Tang network unlock)
                └─ LVM vg "eldir"
                     ├─ eldir-root  700 G   /
                     └─ eldir-swap    8 G   swap

There’s no separate /boot partition now. /boot is a directory on the 700 GB root LV, and the kernels and initrds the firmware actually loads live on the ESP under a machine-id directory, as systemd-boot Type 1 entries. Three kernels on a 2 GB ESP come to 261 MB. The sizing problem isn’t solved so much as made irrelevant.

The chain is now: firmware reads an EFI variable, loads systemd-bootx64.efi, which reads three-line text files in loader/entries/ naming a kernel, an initrd, and a command line. No UUID resolution anywhere — each entry points at a file on the partition the loader itself was read from.

mdadm has to assemble before LUKS can unlock

The ordering is worth stating, because the stack reads top-down and assembles bottom-up. The LUKS container is on /dev/md127, not on a partition, so the initramfs assembles the array before it can even look for a LUKS header. /etc/crypttab keys cryptroot on the array’s UUID, and mdadm.conf names the array so the initramfs doesn’t have to scan for it.

This host also unlocks over the network with Clevis and Tang rather than a typed passphrase, so the initramfs sequence is: assemble md, bring up an interface, DHCP, fetch the Tang advertisement, unlock LUKS, activate LVM, switch root. Any step failing drops to an initramfs prompt on the IPMI console. It works unattended, but it’s more moving parts than a passphrase, and Tang running on one router makes that router a single point of failure for unattended boot.

The BADRAM masks had to move

/etc/default/grub is inert once GRUB isn’t booting the machine, and kernel-install builds its entries from /etc/kernel/cmdline instead. Most of the translation was mechanical, except this:

GRUB_BADRAM="0x1f07817000,0xfffffffffffff000,0x1a664ed000,0xfffffffffffff000"

Two bad 4 KB pages, masked out years ago by me rather than replaced — which, combined with an SEL holding 512 correctable ECC events concentrated on DIMMA1 and accelerating over the preceding two months, describes a machine with known-bad memory. systemd-boot has no GRUB_BADRAM equivalent; it’s a GRUB feature, not a UEFI one. The kernel’s own memmap= does the same job:

memmap=4K$0x1f07817000 memmap=4K$0x1a664ed000

and the kernel confirmed it after the first boot:

kernel: e820: reserve RAM buffer [mem 0x1a664ed000-0x1a67ffffff]
kernel: e820: reserve RAM buffer [mem 0x1f07817000-0x1f07ffffff]

This is easy to drop in a migration and produces no error when you do. Nothing fails. The machine quietly starts using memory you’d decided not to trust. It’s a trap for automation too — the deploy scripts for this host set a kernel command line from a config variable, and the obvious port would have written KERNEL_CMDLINE over /etc/kernel/cmdline, dropping root=, ro and both memmap= entries. That script now only appends missing tokens and never rewrites the file.

For a while I thought bad memory was the answer. GRUB_BADRAM only takes effect once grub.cfg executes, so GRUB’s own load and relocation were never protected by it, and a loader landing on a bad page hangs with no output, deterministically. It fit everything I saw. It also fit nothing I could prove, and the live USB booted through shim and GRUB on the same firmware all day without trouble.

You still can’t mirror an ESP

Worth being precise about, because “mirrored boot disks” implies something stronger than what’s running.

An mdraid1 mirror is invisible to UEFI firmware, which reads raw FAT off a partition before any kernel, md module or RAID metadata handling exists. You can hide an ESP inside an array with 1.0 metadata at the end of the device and firmware will read one member as a plain FAT filesystem — but firmware also writes there, a system token or a boot counter, and then you have two members silently disagreeing with no resync.

So the ESPs aren’t mirrored, they’re copied. kernel-install writes to /boot/efi, and /etc/kernel/postinst.d/zz-esp-sync rsyncs that onto /boot/efi2 afterwards, excluding loader/random-seed and refusing to run if either side isn’t a mounted vfat filesystem. Both disks carry their own UEFI boot entry. That survives losing a disk, but it’s a scheduled copy with a failure mode: if the hook doesn’t run, the second ESP rots, and you find out during the disk failure it was supposed to cover.

Check it as root. /boot/efi is mounted dmask=0077, so an unprivileged find or du reports an empty directory and looks exactly like a mirror that never populated. I spent a while convinced the second ESP was empty and the whole exercise had failed. diff -r as root showed 14 files a side, byte-identical.

The cause, found afterwards

I originally wrote this up as unsolved. That was honest at the time and it’s no longer true, so here’s the correction.

Cleaning up UEFI boot variables weeks later, I found three entries — Boot0000, Boot0006, Boot000B — all pointing at HD(3,MBR,0x52796a31,...): partition 3 of a disk with MBR signature 0x52796a31. No disk on the machine carries that signature. Every device is GPT now. Those entries referenced a drive that had been shuffled between hot-swap bays and then wiped or pulled — one that used to hold a boot sector and a working GRUB configuration.

That fits the symptom in a way none of my five theories did. The firmware was enumerating boot devices for entries whose target no longer existed, and two of them sat between the working entries and the fallback in the boot order. It explains why launching shimx64.efi directly from the EFI Shell failed identically — that path still ran on a machine whose NVRAM was full of dangling references — and why the box was unresponsive rather than erroring. It wasn’t failing to boot an OS. It never got that far.

Deleting them is one command each, but save the right thing first:

sudo efibootmgr -v > /root/efibootmgr-before.txt
for e in 0000 0006 000B; do sudo efibootmgr -b $e -B; done

Plain efibootmgr omits the device-path hex you’d need to recreate an entry you shouldn’t have deleted. Use -v. I nearly didn’t, and I’d told myself I had a backup that turned out never to have been written.

The uncomfortable read is that the rebuild fixed the machine partly by accident. Repartitioning both SSDs and installing a new bootloader created fresh NVRAM entries that sorted ahead of the broken ones, so the symptom disappeared while the cause sat in NVRAM for another few weeks. The new boot stack is better on its own merits — fewer lookups, an ESP that isn’t rationed, both disks bootable independently, and a config that’s three lines of text per kernel instead of a generated grub.cfg with a disk hint in it. It just didn’t diagnose anything.

What I’d check on a machine like this

If you’re running a boot chain that accumulated rather than got designed, two cheap checks: does every HD(...) in efibootmgr -v resolve to a device that still exists, and how large is /boot relative to the initrds in it.

for g in $(efibootmgr -v | grep -oE 'GPT,[0-9a-f-]+' | cut -d, -f2 | sort -u); do
  printf '%s -> %s\n' "$g" "$(blkid -o device -t PARTUUID=$g || echo MISSING)"
done

Neither needs a rebuild. I’d run them before deciding whether you need one.

Notes

  • POST code A2 on AMI Aptio is “IDE Detect”, but a latched code tells you where the firmware last was, not where it’s stuck.
  • ipmitool chassis bootdev bios is the way into Setup on a machine with no working keyboard.
  • The built-in EFI Shell reads device paths, lists FAT filesystems, and executes arbitrary EFI binaries. It is the right tool for bisecting a boot chain.
  • GRUB_BADRAM does not protect GRUB itself; the mask applies only after grub.cfg runs. systemd-boot has no equivalent — use memmap=.
  • A UEFI boot entry pointing at a device that no longer exists can hang the firmware. Audit them when you move drives between bays.