Automated Backups with Restic and Syncthing: Part 9 of "Building a Resilient Home Server" Series

Part 9 of "Building a Resilient Home Server" series

Where We Left Off

In Part 8, I got Vaultwarden running - self-hosted password management via Tailscale Funnel. My passwords were mine again, syncing to my server instead of someone else's datacenter.

The server was doing its job: AdGuard Home blocking ads, Nextcloud storing files, Vaultwarden managing passwords, monitoring keeping an eye on everything.

But here's the uncomfortable truth I'd been avoiding for weeks: I had no backups.

Sure, I had monitoring. I'd know when things broke. But that doesn't help if the drive dies and takes my Nextcloud database with it. Or if I accidentally nuke my AdGuard filter lists during a config change. Or if the entire server catches fire (metaphorically... hopefully).

Time to fix that.

The Backup Strategy

I wanted something that would:

Encrypt everything - My data, my encryption keys
Deduplicate efficiently - I don't need 30 full copies of the same files
Work automatically - Backups that require manual intervention don't happen
Replicate offsite - Local backups are great until the house burns down
Be simple - Complex backup strategies fail when you need them most

The plan: Restic for encrypted backups, Syncthing for offsite replication.

Restic would handle the heavy lifting - encrypted, deduplicated snapshots of databases and config files. Then Syncthing would sync those backups to my Mac, which backs up to iCloud. Three-tier approach: server → Mac → cloud.

Eventually I'd add a second NixOS server (nixos2) for extra redundancy. But first, get the basics working.

What Needs Backing Up?

Three things I absolutely couldn't afford to lose:

PostgreSQL Databases - Nextcloud's database
System Configs - Everything in /etc/nixos/private/ - passwords, API keys, SSH keys, the works
Nextcloud Data - 150GB+ of actual user files

Everything else I could rebuild from Git. But these three? Irreplaceable.

Setting Up Restic

Created modules/backups.nix to keep things organized. Generated a strong password and stashed it in /etc/nixos/private/restic-password:

Bash

openssl rand -base64 32 > /tmp/restic-pass
sudo mv /tmp/restic-pass /etc/nixos/private/restic-password
sudo chmod 600 /etc/nixos/private/restic-password

The Nextcloud database backup was straightforward - dump the database, let Restic snapshot it:

Nix

services.restic.backups.nextcloud-db = {
  repository = "/var/local/backups/restic";
  passwordFile = "/etc/nixos/private/restic-password";
  paths = [ "/var/backup/nextcloud-db" ];

  backupPrepareCommand = ''
    mkdir -p /var/backup/nextcloud-db
    ${pkgs.sudo}/bin/sudo -u nextcloud ${pkgs.postgresql}/bin/pg_dump nextcloud > /var/backup/nextcloud-db/nextcloud.sql
  '';

  timerConfig = {
    OnCalendar = "02:15";
    Persistent = true;
  };

  pruneOpts = [
    "--keep-daily 7"
    "--keep-weekly 4"
    "--keep-monthly 12"
  ];
};

System configs were even simpler - just snapshot /etc/nixos/private/ nightly:

Nix

services.restic.backups.private-configs = {
  repository = "/var/local/backups/restic";
  passwordFile = "/etc/nixos/private/restic-password";
  paths = [ "/etc/nixos/private" ];

  timerConfig = {
    OnCalendar = "03:15";
    Persistent = true;
  };

  pruneOpts = [
    "--keep-daily 7"
    "--keep-weekly 4"
    "--keep-monthly 12"
  ];
};

Added directory creation rules:

Nix

systemd.tmpfiles.rules = [
  "d /var/local/backups 0755 root root -"
  "d /var/local/backups/restic 0700 root root -"
  "d /var/backup 0755 root root -"
  "d /var/backup/nextcloud-db 0755 nextcloud nextcloud -"
];

And permission fixes after backups complete:

Nix

systemd.services.restic-backups-nextcloud-db.postStart = ''
  chmod -R a+rX /var/local/backups/restic/
'';

systemd.services.restic-backups-private-configs.postStart = ''
  chmod -R a+rX /var/local/backups/restic/
'';

Retention: 7 daily, 4 weekly, 12 monthly snapshots. Restic's deduplication is shockingly good - 31 snapshots consuming only 22MB total.

Applied the config:

Bash

sudo nixos-rebuild switch

Backups started running. Finally.

The Syncthing Disaster

Okay, backups are running locally. Now I need them replicated offsite to my Mac. Enter Syncthing.

I'd used Syncthing before - it worked great for my Obsidian notes. Adding another sync folder should be trivial, right?

Added the backup folder to my config... and nothing synced.

Problem #1: Wrong Import Path

Checked modules/services.nix. Found this:

Nix

syncthing-secrets = import ../private/syncthing-secrets.nix;

Then tried to reference syncthing-secrets.devices. But devices weren't in that file - they were in syncthing-devices.nix.

Copy-paste error. Fixed the import:

Nix

syncthing-devices = import /etc/nixos/private/syncthing-devices.nix;

Rebuilt. Still no sync.

Problem #2: Incorrect File Structure

Even after fixing the import, Syncthing couldn't see any devices. Opened syncthing-devices.nix and found this:

Nix

{ config, pkgs, lib, ... }:
{
  devices = {
    # ... device configs
  };
}

This is structured like a full NixOS module - function signature and all. But when you're importing a file to use as a value (not a module), it needs to be just the data:

Nix

{
  devices = {
    # ... device configs
  };
}

No function signature. No module wrapper. Just the attribute set.

This distinction isn't obvious when you're learning NixOS. I'd structured it wrong because that's how most NixOS files look. But imports work differently depending on context, and getting it wrong means Syncthing gets a function instead of data.

The actual working Syncthing config in modules/services.nix looks like this:

Nix

services.syncthing = {
  enable = true;
  user = "ppb1701";
  dataDir = "/home/ppb1701";
  configDir = "/home/ppb1701/.config/syncthing";
  guiAddress = "0.0.0.0:8384";
  overrideDevices = true;
  overrideFolders = true;
  settings = import /etc/nixos/private/syncthing-devices.nix;
};

Fixed the structure. Rebuilt. Backups started syncing.

Testing From the Mac

Once Syncthing caught up, I had the entire Restic repository on my Mac at ~/Downloads/nixosbackups/restic/.

Listed all snapshots:

Bash

restic -r ~/Downloads/nixosbackups/restic snapshots --no-lock

The --no-lock flag is necessary because the folder is read-only from Restic's perspective - it can't create lock files in a Syncthing-managed directory. Safe for read-only operations like viewing snapshots and restoring.

All the snapshots showed up - database backups, config backups, everything. Tested a restore:

Bash

restic -r ~/Downloads/nixosbackups/restic restore latest:/etc/nixos/private --target ~/restore-test --no-lock

Got all my files back. The pipeline works end-to-end.

Enter nixos2: The Second Server

With backups working on the main server, I started thinking about redundancy. What if the main server's drive dies? Sure, I've got backups on the Mac and in iCloud, but restoring 150GB+ of Nextcloud data over the internet would take forever.

Better solution: A second NixOS server with a 6TB drive, syncing Nextcloud data continuously via Syncthing. If the main server dies, nixos2 has a live copy of everything.

I had a Minisforum MS-01 sitting around. Installed the 6TB drive. Time to set it up.

The Flake Saga

I originally built the ISO to match the VM - auto-connect working from the GUI, everything configured properly. It worked great.

Then I decided to convert nixos2 to a flake for cleaner module imports from the main repo.

Converted to flake. Built it. Network didn't work. Couldn't connect to WiFi, couldn't reach the internet, couldn't do anything.

Realized I'd forgotten to import system.nix. And somewhere in the process, the config ended up with the wrong username. I was logged into a session, but nothing ran under my username. sudo didn't work. su didn't work. I was completely locked out.

Booted a previous generation. Everything came back up. Fixed the configs.

Tried to continue with the flake approach, but now I was fighting a different battle: "Oh, you made changes? Better commit them before you can build!" Every. Single. Time.

And nh flat-out refused to work with the flake setup.

Finally decided to abandon the flake approach entirely. Copied the nix files over, removed the flake wrapper, and built it the traditional way.

Eight minutes later, nixos2 was running. Just had to mount the external drive.

Lesson learned: Sometimes the "proper" way is just fighting you for no good reason. Direct configs work fine.

The Declarative Syncthing Problem

Got Syncthing configured between nixos and nixos2. Devices showed up, folders synced... until I rebooted.

After every reboot, Syncthing prompted to confirm the device connection. Auto-discovery worked, but the declarative config wasn't matching exactly between machines. Device IDs, names, settings - something was slightly off.

Eventually got it sorted by making sure both machines had identical device definitions. Now it auto-connects reliably.

Initial Sync: 150GB+

With nixos2 online, I needed to get the Nextcloud data over there. Initial sync of 150GB+ via Syncthing would take days.

Better solution: rsync for the first copy, then let Syncthing handle incremental updates.

First attempt failed - permission denied. rsync needed access to my SSH key to connect to nixos2. Fixed the permissions:

Bash

# Give rsync access to SSH key
chmod 600 ~/.ssh/id_ed25519

# Now rsync works
sudo rsync -avh --progress /var/lib/nextcloud/data/ nixos2:/mnt/nextcloud-data/data/

Took a while, but it worked. Once the initial copy finished, I enabled Syncthing sync for ongoing changes.

Now Syncthing keeps the two in sync. Main server writes new files, Syncthing replicates them to nixos2 within seconds. If the main server dies, nixos2 has everything.

One Small Hiccup

Syncthing couldn't read nextcloud.log due to permissions. Quick fix:

Bash

sudo chmod 644 /mnt/nextcloud-data/data/nextcloud.log

The proper declarative fix is adding the syncthing user to the nextcloud group:

Nix

users.users.syncthing.extraGroups = [ "nextcloud" ];

Haven't gotten around to that yet. The chmod works for now.

Current State

Everything's working:

✅ Nightly database backups to local Restic repo
✅ Nightly system config backups
✅ Syncthing syncing backups to Mac
✅ Mac uploading to iCloud
✅ Tested restores from Mac work
✅ nixos2 online with 6TB drive mounted
✅ Nextcloud data (150GB+) replicating to nixos2 via Syncthing
✅ VM and ISO configs updated and published to GitHub

If the main server dies, I can:

Restore databases from the Mac's backup copy
Restore configs from the Mac's backup copy
Pull Nextcloud data from nixos2
Be back online in under an hour

That's actually bulletproof.

Lessons Learned

NixOS module imports vs. value imports are different. Modules need the function signature ({ config, pkgs, lib, ... }: { ... }), values don't ({ ... }). This distinction matters, and it's not always obvious.

Flakes aren't always worth the pain. Sometimes the "proper" declarative approach fights you more than it helps. If you're spending more time wrestling with the build system than actually building, maybe it's time to simplify. Though to be fair, I would like to revisit it in the future when I'm NOT making changes every few minutes...perhaps I might feel less like throwing the keyboard out the window.

Declarative Syncthing needs exact matching. Device configs between machines must be identical, or you'll get connection prompts on every reboot.

Initial sync strategy matters. For large datasets, don't rely on Syncthing for the first copy. Use rsync (with proper SSH key permissions), then let Syncthing handle incremental updates.

Test your backups. Don't assume they work - actually restore something and verify. I tested restores from the Mac before considering the backup strategy complete.

Deduplication is magic. Restic's efficiency means dozens of snapshots without eating all your storage.

Defense in depth. Local backups + offsite replication + secondary server = multiple layers of protection. If one fails, the others hold.

The Complete Setup

Main Server:

Restic backing up databases and configs nightly
Syncthing replicating backups to Mac
Syncthing replicating Nextcloud data to nixos2

Mac:

Receives encrypted Restic backups via Syncthing
Can perform test restores with --no-lock
Backs up to iCloud for final offsite layer

nixos2:

6TB drive dedicated to backup storage
Receives live copy of Nextcloud data via Syncthing
Ready to take over if main server dies

Three tiers. Multiple redundancies. Tested restores. Finally done.

What's Next?

The infrastructure is solid. Backups are running. Redundancy is in place. Everything important is protected.

But there's one more experiment I've been thinking about: search.

I've been using Kagi for nearly a year now - paying for privacy-focused search. I even built a custom theme for it. It's excellent. Fast, relevant results, no tracking, great interface.

But I'm running a home server specifically to own my services and data. And I keep wondering: Can I self-host search that's actually usable?

Enter SearXNG - a privacy-respecting metasearch engine that aggregates results from multiple sources. No tracking, no ads, completely self-hosted.

The question is: Can it replace a tool I've been happily using and customizing for a year? Or will I be crawling back to Kagi after a week of frustration?

This might be the toughest test yet.

Next up: Part 10 - Can SearXNG Replace Kagi for Me? A Self-Hosted Search Adventure

Main Server (nixos): Codeberg
Second Server (nixos2): Codeberg
ISO can be gotten here.

As always, the complete configuration is available on GitHub, and you can follow along with the series at blog.ppb1701.com.

Questions? Comments? Find me on Mastodon: @ppb1701@ppb.social

← Back to Series