Sysadmin

On a UEFI Ubuntu system there’s a 123-byte file on the ESP that almost nobody looks at. Mine was at \EFI\ubuntu\grub.cfg and contained this: search.fs_uuid 0a9ccc8b-d14c-4291-9eb8-7bf71995a901 root hd2,msdos1 set prefix=($root)'/grub' configfile $prefix/grub.cfg That’s the whole bootstrap. The signed grubx64.efi on the ESP has no idea where your real configuration lives, so this stub finds the /boot filesystem by UUID, sets $prefix to point at it, and chains to the grub.cfg that update-grub generates. Three lines, and I’d never read them before the day my NAS stopped booting. ...

My NAS runs on a Supermicro X10SDV-TLN4F — Xeon D-1541, 128 GB of ECC, a 22 TB ZFS pool, and a BIOS dated 2017. It had been up since March. I did some network configuration work one morning, rebooted, and it didn’t come back. What it showed instead was the Supermicro splash screen with A2 in the bottom right corner, and a keyboard that did nothing. DEL wouldn’t get me into Setup, F11 wouldn’t open the boot menu, F12 wouldn’t PXE. The machine sat there indefinitely. ...

While rebuilding the root filesystem of my NAS I had to decide how much swap to give it. The machine has 128 GB of ECC RAM and runs a ZFS pool, some containers, and a Spark setup that occasionally gets greedy. The old install had a 976 MB swap LV that wasn’t even in fstab. The folk rule says swap should be at least as large as RAM, which would mean carving 128 GB out of a 929 GB mirrored volume group for something I’d hope never to use. That felt obviously wrong, so I went looking for where the rule comes from. ...

I rebuilt a NAS onto a mirrored encrypted root: two SSDs, mdadm RAID1, LUKS2 on top, LVM inside. That layout is well-trodden — it’s what the Debian and Ubuntu Server installers produce when you ask for software RAID plus encryption, and the reasoning for putting RAID below LUKS is sound. You encrypt once instead of twice, resync traffic is ciphertext, and rebuilds work on raw blocks below the crypto layer. ...

I was rebuilding the boot disk of a NAS running Ubuntu 24.04 — kernel 6.8 — and doing the work from an Ubuntu 26.04 live USB, because that’s what I had on the Ventoy stick. This felt fine. A newer rescue environment is usually a better rescue environment: more recent tooling, drivers for newer hardware, fewer bugs. The plan was a mirrored encrypted root: GPT, mdadm RAID1 across two SSDs, LUKS2 on top, LVM inside. I created the array degraded on the new disk, put LUKS2 on it, made the volume group, copied 262 GB of root filesystem across, set up systemd-boot, and rebooted. ...

In the process of trying to figure out my VM performance problems, I’ve been doing a lot of filesystem benchmarking. Unfortunately, there isn’t a lot of consistency between platforms or machine classes as to benchmarking methodology, so I’ve had some trouble generating comparable numbers. However, I’ve gotten the solaris filebench suite running on linux, and bonnie++ running on solaris, so I can now generate comparable numbers across both platforms. One of my primary interests is the throughput I can get out of the 3ware 7506 raid controller in my unix nas box, both in order to optimize it and in order to compare to other solutions and determine if they will actually be an upgrade or not. In the process, I’ve been benchmarking an older Dell Precision Workstation 420. It has 4 wd1200jb drives plugged into its onboard IDE boards (yes, they are sharing ide channels), and is currently running opensolaris nv47. I played around with a few different ZFS configurations, but eventually settled on raidz, leaving me with 360G usable disk space. Here’s some of the numbers I got: ...