Homepage, Backup Topology, and Why macOS Can't Always Be Trusted — Part 12 of Building a Resilient Home Server Series

Where We Left Off

Part 11 closed with a fairly ominous cliffhanger. I told you to back up your Syncthing certificates and said ask me how I know. Actually, don't. I'll tell you in a future post.

This is that post.

Part 12 has a few threads running through it: finishing off some things I'd been deferring (Homepage dashboard, full backup coverage for the Mac and Windows machines), a proper accounting of the disaster that taught me to care about cert.pem and key.pem, and — because it wouldn't be a homelab post without it — more things macOS does silently wrong that make you question your life choices.

I also got Collabora Online working — a web-based document editor integrated with Nextcloud. The personal Microsoft 365 subscription is on borrowed time. As little as I actually use it at home, unless something laughs directly in my face, it's hard to justify keeping it.

Let's get into it.

The Syncthing Mac Disaster (The Story I Promised)

Look, I genuinely like and enjoy using macOS. I build iOS Flutter apps on it. It's solid, it's reliable, and it doesn't break every other update the way Windows has been lately. But every now and then, macOS does something — blocks a connection silently, hides a permission behind three menus, or uses a different socket path depending on how you installed an app — and I'm sitting there staring at my screen going "WHY????" This is one of those posts.

So. macOS updated. Syncthing wouldn't start. The menu bar icon appeared, the service seemed like it was trying, but it was stuck in a "failed to acquire lock" loop. I killed all the processes, found the LOCK file, removed it. Still wouldn't start.

Eventually I found the actual error by running Syncthing with --verbose. Config file version mismatch. Specifically: config file version (51) is newer than supported version (37).

That's... a big gap. What happened?

I had two Syncthing installations running simultaneously. One from Homebrew (/opt/homebrew/opt/syncthing/bin/syncthing). One from the macOS app (/Applications/Syncthing.app). Both pointing at the same config directory. The Homebrew version had been quietly upgrading the config format to version 51. The macOS app only understood up to version 37. After the macOS update changed which one launched on login, the older binary tried to read the newer config and bailed.

The lock file error was a red herring — just a symptom of the first instance crashing without cleaning up. The real problem was buried three layers deeper.

I tried the Syncthing v2 beta to get past the version mismatch. It started, but the GUI showed up on a non-standard port (62571 instead of 8384), and more importantly: on restart it completely reset. New Device ID. No devices. No folders. Blank config. Whatever the beta was doing with state persistence, it wasn't preserving anything.

So now I had no working config and no backup of cert.pem and key.pem.

Here's why that matters. Your Syncthing Device ID is cryptographically derived from those certificates. It's not stored anywhere else. You can't recover it. You can't set it manually to match your old ID. When you lose those files, your Device ID is just gone, and every other machine on the network that trusted the old ID needs to be updated with the new one.

On a regular device that's annoying — open the web UI, update a few entries. On NixOS machines, it means editing the config files, pushing the changes, and doing a nixos-rebuild switch on each one. For two servers that's not the end of the world. It's still part of an afternoon I didn't need to spend.

I downgraded back to v1.30.0, nuked the config, and started over from scratch. Reconnected everything manually. Slowly got the sync state rebuilt.

And then I added two lines to the private-configs restic backup on both NixOS servers:

paths = [
  # ... existing paths ...
  "/home/ppb1701/.config/syncthing/cert.pem"
  "/home/ppb1701/.config/syncthing/key.pem"
];

To be clear — the Mac cert loss was the disaster, and adding these lines doesn't rewind that. What they do is make sure a NixOS crash-and-burn doesn't cause the same problem from a different direction. But adding them also prompted the obvious follow-up question: can I do this for every machine? Turns out yes. So the Windows machines and the Mac got the same treatment. The full picture is in the backup topology section later in this post.

Lesson (the version you should actually remember): cert.pem and key.pem in your Syncthing config directory are the only truly irreplaceable files it has. Everything else can be reconfigured. Those two files cannot. Back. Them. Up.

The Installation Method Problem

The specific failure here is worth calling out explicitly because it goes beyond just Syncthing: App Store vs Homebrew vs Standalone installers are not interchangeable, and they are not the same paths.

I'd been running both Syncthing installations for months without realizing it. They fight over the same ports (8384, 22000) and config directory. The newer one silently wins format upgrades. Pick one installation method and stick with it — Homebrew or macOS app, not both.

This same thing bit me with Bitwarden. More on that in a moment.

Bitwarden SSH Agent — The Socket Path Problem

While I was rebuilding Syncthing, I decided to also fix something I'd given up on months earlier: getting Bitwarden's SSH agent integration working on the Mac.

The symptom was: SSH would fail with "permission denied" and Bitwarden would never prompt for the key. Frustrating, because the whole point of a password manager with SSH agent support is that it manages your SSH keys so you don't have to.

The root cause, once I found it: the App Store version of Bitwarden and the standalone version use different socket paths.

App Store: ~/Library/Group Containers/LH4T7G6HJ7.com.bitwarden/agent.sock
Standalone: ~/.bitwarden-ssh-agent.sock

My .zshrc had the wrong path (copied from a guide written for the App Store version, while I was actually running standalone). So when SSH went to talk to the Bitwarden agent, it wasn't — it was talking to the default macOS SSH agent, which doesn't have any of my keys. Bitwarden never got involved.

The fix:

export SSH_AUTH_SOCK=/Users/ppb1701/.bitwarden-ssh-agent.sock

Full path, not the tilde shorthand — tilde expansion is inconsistent in some contexts. source ~/.zshrc, then ssh-add -l to verify keys are listed.

If the socket file doesn't exist: Bitwarden isn't running or SSH Agent isn't enabled in its settings. You can find it with find / -name "*bitwarden*ssh*" 2>/dev/null. If you get "Agent refused operation," the vault is locked — unlock it first.

There's a particularly ironic loop to this whole thing. I'd given up on Bitwarden's SSH agent because of the socket path issue — I didn't realize that was the problem, I just thought the feature was flaky. So I created a local SSH key with a passphrase instead. Then forgot the passphrase five or six weeks later. Because the password manager that would have stored it for me was the one I couldn't get working.

The tool that solves the memory problem was blocked by one wrong path in .zshrc. Now it works, and local SSH keys on disk are no longer a thing I need on the Mac.

Adding a Homepage Dashboard

Part 11 had everything monitored and backed up, but checking on things still meant opening Grafana, or tabbing through a bunch of nginx virtual hosts to see if services were alive. What I wanted was a simple launchpad — one page I could pull up and see green/red status at a glance, with links to everything.

Homepage is a self-hosted dashboard designed exactly for this. Declarative config, supports up/down status pings, clean UI, and there's a NixOS module that handles the whole setup.

The Basic Service Config

services.homepage-dashboard = {
  enable = true;
  listenPort = 8582;
  allowedHosts = "home.home";
};

The allowedHosts is required — without it, Homepage rejects requests from nginx with a "Host validation failed" error. Learned this the slightly frustrating way.

The virtual host in nginx:

"home.home" = {
  serverName = "home.home";
  listen = [
    { addr = secrets.tailscaleIP; port = 80; }
    { addr = "192.168.1.154"; port = 80; }
  ];
  locations."/" = {
    proxyPass = "http://127.0.0.1:8582";
  };
};

Both listen addresses — Tailscale IP and LAN IP — so it's accessible however you're connected.

One thing Homepage on NixOS doesn't do is generate default config files. You start with a blank dashboard if you don't provide declarative config. So the whole thing needs to be written out.

I went with ping widgets rather than API widgets. The API widgets — Syncthing's widget type doesn't exist in Homepage, Nextcloud's caused TypeErrors — were more trouble than they were worth for what I actually needed, which is just "is this thing up or down." A ping to the local port is sufficient and never has authentication headaches.

The services are also conditional. Same lib.optionals pattern I used for monitoring in Part 11 — if the service isn't enabled in the NixOS config, it doesn't show up on the dashboard. The same homepage.nix file works on both servers, showing only what's actually running.

services = [
  {
    "Services" = lib.flatten (
      (lib.optionals config.services.syncthing.enable [
        {
          "Syncthing" = {
            description = "File sync";
            href = "http://syncthing.home";
            icon = "syncthing";
            ping = "http://127.0.0.1:8384";
          };
        }
      ]) ++
      (lib.optionals config.services.gitea.enable [
        {
          "Gitea" = {
            description = "Git hosting";
            href = "http://git.home";
            icon = "gitea";
            ping = "http://127.0.0.1:3300";
          };
        }
      ])
      # ... and so on for each service
    );
  }
];

One gotcha: AdGuard. If you ping it via the DNS hostname (http://adguard.home), you're asking AdGuard Home's DNS to resolve the address of AdGuard Home, which creates a circular dependency and it shows as down. Ping the local port directly instead:

ping = "http://127.0.0.1:3000";

Dark Reader, if you use it, will mess with the theme. Disable it for home.home.

The end result is a clean dashboard with live up/down indicators for everything, resource widgets showing CPU/RAM/disk usage, and one-click links to all the services. It auto-hides services that are disabled. Survives rebuilds. Takes about fifteen seconds to glance at and know the current state of the fleet.

Full Backup Topology

Alright. Let's actually write out what's backed up across every machine, because Part 9 covered the NixOS server backups but I never laid out the complete picture.

NixOS Servers

The restic backup strategy from Part 9 covers the data-heavy stuff:

nixos1 (main): Vaultwarden (hourly), Nextcloud DB (daily), Linkwarden (daily), private configs including Syncthing certs (daily)
nixos2: Gitea (daily), private configs including Syncthing certs (daily)

All restic repos live at /var/local/backups/restic and get chmod g+rX + chgrp syncthing after each backup so Syncthing can pick them up. From there:

NixOS servers → Restic → /var/local/backups/restic → Syncthing → Mac → iCloud

The important addition since Part 9: both servers now back up cert.pem and key.pem from /home/ppb1701/.config/syncthing/ as part of the private-configs job. For reasons I've already explained at length.

Windows Machines

Both Windows boxes run a minimal restic backup on a daily Task Scheduler job. Here's what's actually worth backing up when most things are already in the cloud:

C:\Users\pboyd\AppData\Local\Syncthing\cert.pem
C:\Users\pboyd\AppData\Local\Syncthing\key.pem
HKCU registry export (reference for settings)
winget list output (reference for reinstalling apps)

That's it. Browser stuff is cloud-synced. SSH keys are in Vaultwarden. Everything else syncs via Syncthing or lives in GitHub. The Syncthing certs are the only genuinely irreplaceable things.

The backup script:

Start-Transcript -Path C:\Temp\backup-log.txt -Append

# Export reference data
reg export HKCU C:\Temp\user-settings.reg /y
winget list > C:\Temp\installed-apps.txt

# Backup
restic -r C:\ResticRepo backup `
  C:\Users\pboyd\AppData\Local\Syncthing\cert.pem `
  C:\Users\pboyd\AppData\Local\Syncthing\key.pem `
  C:\Temp\user-settings.reg `
  C:\Temp\installed-apps.txt `
  --password-file C:\rp\password.txt

# Prune
restic -r C:\ResticRepo forget `
  --keep-daily 7 `
  --keep-weekly 4 `
  --keep-monthly 12 `
  --prune `
  --password-file C:\rp\password.txt

Stop-Transcript

One gotcha on Windows: Task Scheduler's GUI has an XML-parsing issue with certain tasks written by third-party software (AMD and ASUS updater apps are common culprits). If you open Task Scheduler and everything is greyed out with an "index out of range" error, the GUI is broken but the tasks themselves are fine. Create new tasks via PowerShell to bypass the problem entirely:

$action = New-ScheduledTaskAction -Execute "powershell.exe" -Argument "-ExecutionPolicy Bypass -File C:\ResticRepo\backup.ps1"
$trigger = New-ScheduledTaskTrigger -Daily -At "12:34PM"
$settings = New-ScheduledTaskSettingsSet -RunOnlyIfNetworkAvailable
Register-ScheduledTask -TaskName "ResticBackup" -Action $action -Trigger $trigger -Settings $settings -RunLevel Highest

The repos live in a Syncthing folder that syncs to nixos1, which then propagates to Mac and iCloud through the existing chain. I thought briefly about also writing them to the external SSDs on the NixOS boxes, but that felt like overkill — these repos already exist on iCloud and are spread across close to half a dozen machines. The redundancy was already there. No new infrastructure needed.

One note on public NixOS configs: I didn't name the Syncthing folders win1restic, win2restic, macrestic in the public repo. I used a generic path (/var/local/clientbackups) and kept the actual device/folder mappings in the private syncthing-devices.nix that doesn't get pushed publicly. Putting that detail in a public repo is accidentally publishing a hardware inventory.

Mac

The Mac runs two parallel backup strategies, which a blog reader noted is "reversed" from their setup — valid point, there are multiple ways to do this. Here's the thinking:

Time Machine handles macOS system state: keychain, app permissions, sandbox data, preferences, bare-metal restore capability. Apple has deep hooks into all of this that third-party tools can't easily replicate. Time Machine is a black box — no retention control, no backend choice — but for macOS-specific state it does things restic simply can't.

Restic covers Syncthing certs (exactly as above, just different paths):

restic -r /Users/ppb1701/Downloads/ClientBackups/ResticRepo backup \
  "/Users/ppb1701/Library/Application Support/Syncthing/cert.pem" \
  "/Users/ppb1701/Library/Application Support/Syncthing/key.pem" \
  /tmp/brew-apps.txt \
  --password-file /Users/ppb1701/rp/password.txt

The restic repo sits in ~/Downloads/ClientBackups/ which is in an iCloud-watched folder AND picked up by Syncthing. That's double coverage with zero extra effort — iCloud grabs it directly, Syncthing sends it to the NixOS boxes independently. Either path alone would be sufficient. Both running simultaneously costs nothing.

Schedule via crontab. If you try to edit crontab and suddenly find yourself in vim, you can bypass it:

EDITOR=nano crontab -e

Look, vim is genuinely powerful and I get why people love it. I just lived in a Windows world for years and my muscle memory is very much "something more notepad-y." I'm not learning vim for a 90-second crontab edit. No judgment if you do — I'll just be over here with nano. If you have micro installed, that works too, but nano ships with macOS so it's the safer default.

The Time Machine target is an SMB share on nixos2 — a 2TB cap on the 6TB SSD. Initial backup took about 1.5 days (server is on WiFi). That's normal. Subsequent incremental backups are fast. I set the frequency to weekly since most things that matter are already in the cloud or on Syncthing.

The full topology:

NixOS servers     → Restic → /var/local/backups/restic → Syncthing → Mac → iCloud
Windows machines  → Restic → C:\ResticRepo             → Syncthing → NixOS → Mac → iCloud
Mac               → Restic → ~/Downloads/ClientBackups  → iCloud (direct) + Syncthing → NixOS
Mac               → Time Machine → nixos2 Samba share

Every machine now backs up its Syncthing certs. The lesson was expensive in time, not in data — but still.

The Channel Chaos That Preceded All of This

Before I get to Collabora actually working, I need to tell you about the adventure that preceded it. Because nothing about this was straightforward, and it started before I even touched Collabora config.

I went to do a rebuild and it failed. Poked around and discovered my system was on nixos-unstable (26.05pre) but home-manager was pinned to release-25.11. Not intentionally mismatched — this had drifted at some point and I hadn't noticed because things had been working fine.

My first instinct was to downgrade everything to 24.11 stable and get onto a properly pinned channel. This turned into a cascade of errors that I will present to you in the order they arrived, like a series of increasingly annoyed error messages from a system that did not want to be touched:

Error 1: services.homepage-dashboard.allowedHosts doesn't exist in 24.11. The option landed later. Comment it out and try again.

Error 2: home-manager.users.ppb1701.programs.zsh.initContent doesn't exist in 24.11 — it's initExtra in that version. Rename, restructure, try again.

Error 3: nextcloud32 not available in 24.11, which only goes up to nextcloud31. Okay, try nextcloud30 — except that breaks the richdocuments app because the version compatibility matrix doesn't line up.

Error 4: linkwarden package doesn't exist in 24.11 at all.

At some point you have to accept that the config has just outgrown a channel version. Four options deep into fixing a downgrade, I stopped and made the pragmatic call: the system is on unstable, it's been working on unstable, lean into unstable properly and stop fighting it.

sudo nix-channel --add https://nixos.org/channels/nixos-unstable nixos
sudo nix-channel --add https://github.com/nix-community/home-manager/archive/master.tar.gz home-manager
sudo nix-channel --update

Both channels on unstable/master, properly matched. Then rebuild. Eventually I'd like to get back onto a stable channel properly — but that's a "a few months from now when I have the patience to work through whatever the config has outgrown next" problem, not a today problem.

The rebuild hung. This is apparently normal behavior after a large channel jump — services stall during activation and the switch never completes. Hard reboot. Everything came up clean on the next boot. If you're doing large channel jumps on a headless machine, the rebuild-safe alias I added handles this:

rebuild-safe = "nh os switch -f '<nixpkgs/nixos>' -- -I nixos-config=/etc/nixos/configuration.nix || sudo reboot -f";

If the switch fails or hangs, it force-reboots. The system comes back up on whatever generation actually activated. Blunt but effective.

While all this was happening, the channel chaos had knocked Nextcloud sideways too. The richdocuments app reference in the config had gotten into a state where Nextcloud was trying to auto-update a previously installed app that no longer matched what was in the config. The fix was a git revert to a clean state, then rebuilding forward from there.

Then Redis was down. Because of course it was.

sudo systemctl start redis-nextcloud
sudo systemctl start phpfpm-nextcloud

Every error in nextcloud.log during that period was RedisException: Redis server went away. Redis is a hard dependency of Nextcloud — if Redis dies, Nextcloud dies, and everything in the logs will point at symptoms rather than the actual cause. Check Redis first.

After all of that, I was back to a clean, stable, properly channeled system. I also mirrored the channel change to nixos2, which had been on nixos-25.11 stable. Same process, same rebuild-hang-reboot pattern, same clean result on second boot.

The NixOS generation rollback is worth mentioning here too. At one point during the worst of the channel chaos, I rolled back to generation ~150 (pre-chaos) just to get back to a known working state and regroup. nixos-rebuild list-generations to find it, then select it from the boot menu. Homepage and Nextcloud were already broken before the chaos started, so they didn't come up — everything else did. That's the value of generations: you always have a retreat option.

Okay. Now Collabora.

Collabora Online — Replacing Microsoft 365 at Home

The last piece of this update: Collabora Online is running, integrated with Nextcloud. Full disclosure — I set it up and it worked immediately on the things I tested, but I haven't exactly had a flurry of documents to edit since then, so I'm not going to oversell it. What I did open worked great. And work still runs Microsoft 365 so I'm not escaping it entirely on work machines anyway. But for personal use, the option is there without a subscription.

Collabora is essentially LibreOffice in a web browser, integrated with Nextcloud via the WOPI protocol. You open a .docx or .xlsx in Nextcloud, and instead of downloading it, it opens in a full editing interface right in the browser. Track changes, comments, formatting — it works.

The NixOS service config:

services.collabora-online = {
  enable = true;
  port = 9980;
  settings = {
    ssl."@enable" = false;
    ssl."@termination" = false;
    ssl.enable = false;
    ssl.termination = false;
    server_name = "collabora.home";
    storage.wopi."@allow" = true;
    storage.wopi.alias_groups."@mode" = "groups";
    storage.wopi.host = [ "cloud\.home" "127\.0\.0\.1" ];
  };
};

That doubled-up SSL configuration is not a typo. The NixOS module sets XML attributes (@enable, @termination) but the underlying coolwsd process reads inner element values (enable, termination). If you only set one or the other, coolwsd will still think SSL is enabled, proxy through nginx will fail, and you'll get 502s. Both must be false.

The systemd service name is coolwsd.service, not collabora-online.service. Useful to know when you're doing systemctl status and wondering why nothing is coming up.

The virtual host:

"collabora.home" = {
  locations."/" = {
    proxyPass = "http://127.0.0.1:9980";
    proxyWebsockets = true;
    extraConfig = ''
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
    '';
  };
};

In Nextcloud's admin settings, set the Collabora Online server to http://collabora.home. Then set the WOPI allow-list — and here's the part that tripped me up — to 127.0.0.1, not the LAN IP of the server. Coolwsd connects back to Nextcloud from the loopback interface, not from the network interface. If you put the LAN IP in the allow-list, coolwsd will be denied because the connection is coming from somewhere else.

The richdocuments app is how Nextcloud integrates with Collabora:

extraApps = {
  inherit (config.services.nextcloud.package.packages.apps) richdocuments;
};

Add a DNS rewrite for collabora.home in AdGuard. That's it.

It takes roughly 500MB of RAM at baseline, adds about 2.7GB to the Nix closure (LibreOffice fonts and all), and is completely stateless — no additional backup needed beyond whatever Nextcloud already has.

The practical test so far: the documents I opened loaded cleanly and edited without drama. It's not going to replace Excel for power users with complex pivot tables, and work has 365 on work machines so I'm not escaping it entirely. But for personal use, where I'm paying a subscription for something I barely touch? The math stops making sense. We'll see if something laughs in my face when I actually need it for something heavier, but setup-wise it went smoother than almost anything else in this series.

The macOS Silent Failure Pattern

I'll tie a bow on the recurring theme here. Across Parts 11 and 12, macOS has silently failed in four separate and distinct ways:

Vivaldi Local Network permission (Part 11) — "address unreachable" because macOS blocked local network access without mentioning it
Syncthing dual-install config version mismatch — "failed to acquire lock" masking the real problem, which was a version incompatibility
Bitwarden SSH socket path — "permission denied" because SSH was talking to the wrong agent entirely
macOS update resetting launch state — the thing that kicked off the whole Syncthing disaster in the first place

The pattern is consistent: the error message is never actually about what's wrong. The fix is always a path, a permission, a setting, or an installation method conflict. Check the boring stuff first. Check which installation method you're running. Check app-specific paths before assuming the configuration is broken. And check System Settings → Privacy & Security → Local Network before you spend two hours debugging a network connection that's working perfectly fine.

Lessons Learned

Back up your Syncthing certificates. I know I said this in Part 11. I'm saying it again. cert.pem and key.pem. Two lines in your backup config. Do it now.
Pick one installation method per app. Homebrew, App Store, or standalone — not multiple. They don't share socket paths, config directories, or binary versions, and the conflicts are silent.
macOS error messages are decorative. The actual problem is always one layer deeper. Treat every macOS error as a starting point for investigation, not a description of what's wrong.
Conditional config pays dividends. The same lib.optionals pattern from monitoring in Part 11 made Homepage significantly less work to maintain — it adapts to which services are running automatically.
Simple is better for dashboards. Ping widgets that just check if a port is open are more reliable than API widgets that need credentials and specific service integrations. I care about up/down at a glance, not detailed stats in the dashboard (Grafana is there for that).
Know what channel you're actually on. Channel drift is silent. nix-channel --list before a big change, not after things are already broken.
If the config has outgrown a channel version, stop fighting the downgrade. Four cascading errors trying to go back to 24.11 was the universe telling me something.
Redis is a hard Nextcloud dependency. If Nextcloud is throwing cryptic errors, check Redis first. Everything cascades from it.
Collabora is genuinely good enough for personal document editing — at least for what I've tested. Work still uses 365 and that's fine. But paying a personal subscription for something I barely touch at home is harder to justify now.

What's Next

The stack is in a solid state. Both servers fully monitored and backed up, every machine in the fleet has cert backups, there's a dashboard to check the overall health at a glance, and there's a web document editor so I'm not dependent on a personal Microsoft subscription for basic productivity.

I've been poking at some things around VM/ISO storage — the 6TB SSD on nixos2 has plenty of headroom and I've been thinking about what to actually do with it beyond Time Machine. Related thought: the Samba infrastructure is already there from the Time Machine setup, so repurposing part of it as a general file server for drag-and-drop transfers between machines isn't much of a stretch. Whether that's actually useful or just another thing to maintain, I genuinely don't know yet. We'll see what sounds appealing versus what I can justify actually needing. Also need to build a new ISO. We'll see what breaks first.

One more thing since I finished writing this: my Windows machine had other ideas. Fresh install, dropped the backed-up Syncthing certs in place, everything reconnected like nothing happened. The backup paid for itself before the post even went up. New ISO is on hold until I get that sorted — but that's probably Part 13's problem.

Find me at @ppb1701@ppb.social on Mastodon. Let me know if you've been following along or if something in the configs helped you out — or cost you an afternoon the same way Syncthing cost me one.

← Back to Series

Main Server (nixos): Codeberg
Second Server (nixos2): Codeberg
ISO can be gotten here.