VM Adventures - Getting hands on with NixOS in the VM : Part 2 of "Building a ResilientHome Server" series

*Part 2 of "Building a Resilient Home Server" series*

After my EndeavourOS server died, I decided to try NixOS for its declarative configuration and reproducibility. But I wasn't about to deploy straight to production hardware. Time to spin up a VM and learn.

The VM Playground

I installed NixOS 25.05 in VirtualBox with GRUB (BIOS mode). The installer was minimal but straightforward. I chose a basic configuration to start.

AdGuard Home: The DNS Loop

First mission: get AdGuard Home running declaratively. On EndeavourOS, I'd set it up manually—installing packages, editing config files, hoping I remembered everything. With NixOS, I wanted it done right.

I created /etc/nixos/modules/adguard-home.nix and configured the service. One sudo nixos-rebuild switch later, AdGuard Home was running.

Sort of.

The web interface loaded, but DNS queries weren't working. I'd set upstream DNS to 192.168.1.1 (my router), which seemed logical. My router handles DNS, so AdGuard Home should use it, right?

Wrong.

The router was forwarding DNS queries back to AdGuard Home, which was forwarding them to the router, which was forwarding them back... you get the idea. DNS loop.

The fix was pointing upstream DNS to actual DNS servers (Quad9 and Cloudflare) and using local_ptr_upstreams for local hostname resolution. Once I understood the architecture, it made perfect sense.

RustDesk: The Build That Wouldn't (And Took Home Manager With It)

With AdGuard Home working, I decided to tackle RustDesk—a self-hosted remote desktop solution. I wanted to access the server's desktop remotely without relying on third-party services.

RustDesk needed to be built from source. This meant diving into Nix derivations, build dependencies, and a whole lot of trial and error.

The build kept failing. Cryptic error messages about missing dependencies, incompatible versions, build flags that didn't work. I tried different approaches, tweaked configurations, rebuilt multiple times.

Somewhere in the chaos of troubleshooting RustDesk, I'd set up Home Manager to manage my user-level configurations—Starship prompt, Zsh aliases, the works. It was working beautifully.

Until it wasn't.

During one of my many RustDesk build attempts, I must have accidentally removed or commented out the Home Manager import while editing the configuration. When I rebuilt, Home Manager just... vanished. My custom prompt was gone, my aliases disappeared.

Because NixOS is purely declarative, when that import line was gone, Home Manager ceased to exist. No leftover files, no cached state, nothing.

After hours of fighting with RustDesk, I discovered it was a known issue in NixOS—the package had build problems that weren't going to be fixed quickly. I had two choices: wait for someone else to fix it, or pivot to something that actually worked.

PIVOT!

RustDesk could wait. I needed to get SSH working and move forward.

SSH: The First Connection

Getting SSH running was straightforward. Added it to my configuration:

Nix

services.openssh = {
  enable = true;
  settings = {
    PermitRootLogin = "no";
    PasswordAuthentication = true;
  };
};

Rebuilt, and I could SSH from my Windows machine using password authentication. Not the most secure, but good enough for learning in a VM.

Shell Improvements

With SSH working, I spent time making the shell more pleasant. I added Zsh with some basic configuration:

Nix

programs.zsh = {
  enable = true;
  enableCompletion = true;
  autosuggestions.enable = true;
  syntaxHighlighting.enable = true;
};

Better than Bash, but the prompt was boring. I'd deal with that later.

The Custom ISO Idea

At this point, the VM was working well. AdGuard Home was running, SSH was functional, and I had a basic but usable system. Time to think about deploying to real hardware.

But I had an idea: what if I could build a custom NixOS ISO with my entire configuration baked in? Instead of installing vanilla NixOS and then copying configs, I could just boot my custom ISO and have everything ready to go.

I started researching how to build custom NixOS ISOs. That's when things got interesting.

To Be Continued...

In Part 3, I'll cover the custom ISO saga, the UEFI surprise, so many stair trips (PRO TIP: Don't do this in the middle of a three-week-long paint project where you reside on a different floor than the physical machine), accidentally nuking the desktop environment, and eventually getting the server running in production.

Want to follow along or see the configs? Check out my NixOS configuration Main Server (nixos): Codeberg
Second Server (nixos2): Codeberg
ISO can be gotten here.

← Back to Series
                    
What do you think about NixOS for a homeserver? I'd love to hear your thoughts—find me on Mastodon at @ppb1701@ppb.social and let's talk NixOS or Linux in general.