Why I Picked NixOS for a 24/7 Home AI Server, and the 5 Gotchas That Bit Me During Install

Posted on Oct 21 • Originally published at wada-dev.com

I built a home AI server that runs around the clock, as a playground to mess with local LLMs. The GPU is an RTX 5090 (32GB), and it's headless and unattended by design. The question was which OS to run. I use Arch on my daily machine, so I figured I'd go with Arch — but in the end I picked NixOS.

To cut to the chase: it was the right call. But because the GPU is a Blackwell-generation card — basically zero community precedent, pioneer territory — installing from the minimal ISO got me stuck a few times, as expected. This post is a write-up of "why NixOS" and "the gotchas I actually hit during install." It's not a generic NixOS intro; I'm only writing the parts where I personally got my hands dirty.

My laptop runs Arch and I have no complaints. But a 24/7 unattended server has different requirements.

There's a learning cost to the Nix language, but with a Haskell/Lua background I judged it acceptable. It also fit my rule of thumb that "systems aligned with the grain of my interests (declarative, functional) are the ones I actually keep up."

NixOS + RTX 5090 (Blackwell) has almost no precedent. Naturally you wonder, "wouldn't Ubuntu be safer, with more reports out there?" I did my homework here first.

The scariest thing about running Blackwell 24/7 is an unresolved hang bug called the GSP heartbeat timeout — and as far as I could find, it's distro-independent (reported on Mint, Fedora, etc. Blackwell forces the nvidia-open driver on every distro, so that part is shared). In other words, switching to Ubuntu would buy me more anecdotes but not avoid the crash. Giving up NixOS's rollback and declarative config for that isn't worth it.

So I landed on: " stay on NixOS + design assuming a watchdog auto-recovers + keep a single Ubuntu Live USB for triage." Just enough of a safety net to answer "is this a GPU problem or a Nix-config problem?" when something jams.

Now the main event. I did a base install from the NixOS 26.05 minimal ISO with a btrfs subvolume layout. I ran the whole thing remotely over SSH, so the traps specific to that are included.

Since it's headless, the first thing I needed was WiFi up so I could open SSH. I followed the old wiki recipe of wpa_passphrase > /etc/wpa_supplicant.conf — and it simply would not connect.

The cause: the 26.05 installer ISO assumes NetworkManager. The ISO's wpa_supplicant.service starts with -u (D-Bus mode, no -i ) as an NM-only backend, so writing your own conf never grabs the interface. Worse, even if you kill it, it comes back via D-Bus activation, so trying to drive it by hand just whiffs over and over.

nmcli device wifi connect "<SSID>" password "<pass>" Enter fullscreen mode Exit fullscreen mode One command and it connects. Trust the old recipe without questioning it and you're stuck at the very first step.

I wanted compress=zstd and noatime on btrfs, so I mounted the subvolumes and ran nixos-generate-config. Looking at the generated hardware-configuration.nix, subvol= was there but compress=zstd and noatime were gone.

This is documented behavior in the official Wiki. Make it up on the configuration.nix side:

fileSystems. "/". options = [ "compress=zstd" "noatime" ]; fileSystems. "/nix". options = [ "compress=zstd" "noatime" ]; #...one per subvolume Enter fullscreen mode Exit fullscreen mode The lists get merged, so this adds on without dropping subvol=. A textbook case of "don't take the generated output at face value — diff it."

This was the one that made me sweat. The WiFi connection info you set up on the ISO does not carry over to the install target. Reboot without noticing and the headless server comes up with no networking — and since there's no screen or keyboard attached, you are quite literally locked out.

You have to copy the NM profiles by hand before installing:

mkdir -p /mnt/etc/NetworkManager/system-connections cp -a /etc/NetworkManager/system-connections/ *.nmconnection \ /mnt/etc/NetworkManager/system-connections/ chmod 600 /mnt/etc/NetworkManager/system-connections/ * Enter fullscreen mode Exit fullscreen mode (I later rewrote this declaratively with networking.networkmanager.ensureProfiles in configuration.nix, but to survive the first reboot you need this copy.)

Once it came up, my SSH keystrokes felt sluggish. I suspected the hardware, but time bash -lc true showed shell startup at 0.002s. The shell was innocent.

Suspecting the network layer, I took a ping: average 83ms, max 257ms on a LAN. Normal LAN values are 2–5ms, so something was clearly off. This is the classic pattern of WiFi power save buffering packets until the next beacon.

The chip I was using (rtw89 / RTL8922AE) treats NetworkManager's powersave: 0 not as " disabled " but as " driver's choice," and the driver happily turns power save on. Turn it off explicitly:

# runtime check + immediate fix iw dev wlp7s0 get power_save iw dev wlp7s0 set power_save off Enter fullscreen mode Exit fullscreen mode On NixOS I made it permanent declaratively, adding wifi.powersave = 2; (2 = disable) to the profile. Result: average 8ms, max 18ms — about a 10x improvement. I learned in my bones that for a server, WiFi power save off is the default.

Minor but quietly annoying. Firing a glob command remotely like sudo cp... *.nmconnection... expands the glob in the non-root shell and fails. You need the sudo sh -c "cp... *.ext..." form, so the glob expands in root's shell instead. A hazard specific to remote work.

Pioneer territory lived up to its name — the WiFi stuff especially did not go smoothly. But every mine I stepped on was the kind where first-hand sources exist if you look, and none were fatal. And the thing I actually wanted — the peace of mind of declarative config + rollback — delivered. After that, piling on drivers and services stopped being scary. Just knowing "if it breaks I can roll back to the previous generation" drops the psychological cost of tinkering with a server by an order of magnitude.

The real boss fight on the GPU (Blackwell) side — the GSP heartbeat timeout and watchdog design — is a separate story, so I'll split that into its own post.

This post was written with AI assistance and reviewed by the author.