The Day the Generator Guy Broke Everything (and Why I Decided NixOS Was Worth Learning): Part 1 of "Building a Resilient Home Server" series
*Part 1 of "Building a Resilient Home Server" series*
4:00 PM on a Tuesday
Generator maintenance. Annual checkup. The guy was just finishing up.
Testing the handover from utility to generator, generator back to utility. Standard procedure.
Then: streaming stopped mid-show. UPS units started screaming. Something was wrong.
So what actually happened?
Turns out: the part of the generator breaker that controls handover from generator to utility? Failed. Mid-test.
Left the house stuck in "generator mode"—except the generator was off.
The maintenance guy fiddled with it. Turned it off and on manually. Got power back.
They'd have to come back with a replacement part, but hey, everything was working again, right?
Phones connected. PCs online. Router happy.
Everything was fine...except Streaming Was Just... Spinning
Wife's streaming apps: loading forever. Spinning. Not playing.
Every streaming device in the house: same thing. Just spinning.
But the phones worked. Laptops worked. Everything else was fine.
"What the heck?" Then the slow realization clicks...
Streaming devices weren't falling back to the router's backup DNS like everything else.
They were stuck waiting for the primary DNS server that... wasn't responding.
My little adblock server was down.
The Setup (That Was)
I'd been running AdGuard Home on EndeavourOS. A little mini PC sitting by the router. Network-wide ad blocking, DNS filtering, client identification. It handled DNS for the entire house.
12-18 filter lists. Custom rules for streaming apps. Everything tuned exactly how I wanted it.
Worked great. Set it up once almost a year ago, forgot about it.
Until the generator incident.
The Grub Screen of Doom
Went to the server. Hooked it up to a display.
error: no such device: [UUID]
Entering rescue mode...
grub rescue>
Grub couldn't find anything. Boot loader was toast.
The power cycling during the generator failure—when the server wasn't on the UPS (oops)—had corrupted something. Bootloader, filesystem, who knows.
The Research Phase
I had too much going on to deal with this immediately. Kicked the server out of the router's DNS settings. Switched to Quad4 temporarily. Streaming worked again.
Then: how do I recover this?
Searched: "endeavouros grub rescue recovery"
Found... a lot of conflicting advice. Live USB, chroot into the installation, reinstall grub, maybe fix fstab, possibly regenerate initramfs, hope nothing else is corrupted.
Not exactly straightforward.
And even if I got it working: then what? Reconfigure everything? Try to remember all my filter lists? Hope I didn't forget something important?
The Mastodon Connection
Was scrolling my Mastadon feed randomly a couple days later, and someone was talking about moving to NixOs. Set it up once move anywhere, etc, etc.
That sounded rather appealing so I looked into it. "Declarative configuration." "Reproducible builds." "Your entire system defined in text files." "Load it to Git."
"Fix it once. Any failure after that is 20-30 minutes max."
That got my attention.
The Promise
What if I could:
- Define my entire server setup in configuration files
- Commit those files to Git
- When hardware fails: install NixOS, clone the repo, rebuild
- Get an identical system back
Not "here's my documentation that's probably outdated."
Not "here's the list of commands I think I ran."
But "here's exactly what my system is, in code."
Hardware dies? 20-30 minutes to full recovery.
Generator incident? Install, clone, rebuild, done.
Bootloader corruption? Doesn't matter—rebuild from config.
Decided to find out if it actually worked.
The Decision
Option A: Fix the EndeavourOS installation
- Hours of recovery attempts
- Might not even work
- Then reconfigure everything from memory
- Hope I don't forget something
- Repeat this process next time something breaks
Option B: Try NixOS
- Learning curve (steep, apparently)
- But: configuration as code
- Git-versioned infrastructure
- Reproducible deployments
- Never manually rebuild again
The generator incident made the decision for me.
I wasn't going to spend another 4+ hours rebuilding a server from memory.
The Learning Curve Warning
NixOS is different. Really different.
- Packages don't go in `/usr/bin`
- You don't edit config files in `/etc`
- You don't run `pacman -S` or `apt install`
- Everything's in weird hash-based paths in `/nix/store`
- Configuration is declarative, not imperative
The documentation assumes you either:
1. Already know functional programming
2. Have infinite patience
3. Enjoy reading academic papers about build systems
I had none of these. Well I had a brief stint in a work clojure experiment years ago...however, my day job is .Net and my hobby is Flutter. So a functional developer I am not.
But I also had:
- A dead server with a grub error
- No desire to manually rebuild ever again
- 12-18 filter lists to reconstruct
- Streaming apps that needed custom rules
- The memory of this whole mess fresh in my mind
So: VM time. Learn in a safe environment. Then deploy for real.
The Plan
1. Install NixOS in a VM
2. Get AdGuard Home working
3. Document everything in configuration files
4. Test disaster recovery by trashing the VM and rebuilding
5. If it works, deploy to real hardware
6. Never manually configure a server again
Spoiler: It worked. Eventually. With some detours.
What's Coming
In the next posts, I'll walk through what actually happened:
Part 2: VM Adventures: setup Nixos on a vm and get AdGuard Home "sort of" work and find my footing
Part 3: When things break (RustDesk build errors and pragmatic solutions)
Part 4: Making it work (SSH, port forwarding, and filter list archaeology)
Part 5: Deployment to real hardware (when the bootloader lets me)
Was It Worth It?
Spoiler: Yes.
Right now, as I'm writing this, the server is humming along behind me. Filtering ads. Syncing files via Syncthing. Running exactly as configured.
It took 3 weeks to:
- Get the basics working in a VM
- Fight with deployment issues and surprise gotchas
- Fine-tune on the physical hardware
And now it's running. When hardware fails again? Install, clone repo, rebuild. 20-30 minutes.
No more generator incidents taking down my DNS server for days.
No more "which filter lists did I use?" (they're in the config)
No more manual configuration (it's all declarative)
Just: install, clone, rebuild, done.
That was the goal. And it actually worked.
Don't want to wait for the rest to jump in yourself?
Main Server (nixos): Codeberg
Second Server (nixos2): Codeberg
ISO can be gotten here.
---
← Back to Series
What do you think about NixOS for a homeserver? I'd love to hear your thoughts—find me on Mastodon at @ppb1701@ppb.social and let's talk NixOS or Linux in general.