ByteHaven - Where I ramble about bytes Avatar
ByteHaven - Where I ramble about bytes

Gitea, Smarter Monitoring, and the Syncthing Permissions Fight — Part 11 of Building a Resilient Home Server Series

Where We Left Off

Part 10 was about adding new services — SearXNG, LinkWarden, and a failed attempt at self-hosted RSS scraping. The stack was growing, and with it, some cracks were starting to show. My monitoring config was getting brittle (add a service, forget to update Prometheus, wonder why you didn't get an alert). Syncthing was throwing permission errors that made no sense until they made too much sense. And I still didn't have a self-hosted Git solution, which felt like a gap for someone who keeps preaching about not depending on services you don't control.

Time to fix all of that.

Gitea — Self-Hosted Git on nixos2

Why Self-Host Git

UPDATE: I've since pivoted from GitHub for public to Codeberg. For why. A future post in the series will explain more. For the purposes of this the process does hold true more or less as long as it's a git repo, so I will maintain the GitHub in the post.

I've got about fifteen personal repos on GitHub. Course projects, job tests, NixOS configs, apps, personal sites — nothing earth-shattering, but all stuff I'd rather not lose if Microsoft decides to do something weird with GitHub. Which, let's be honest, is not the most unreasonable fear in the world.

Gitea is a lightweight, self-hosted Git service. Think GitHub but running on your own hardware, with a web UI, issue tracking, and — critically — the ability to mirror to and from GitHub. It's written in Go, uses minimal resources, and NixOS has a first-class module for it.

I decided to run Gitea on nixos2 (the second server) rather than the main one. The reasoning: nixos2 was already the backup/failover box with fewer services running. Giving it a primary role with Gitea balanced the load and meant my Git hosting wasn't competing with Nextcloud and everything else on the main server.

The Config

Here's the Gitea service config on nixos2's services.nix:

services.gitea = {
  enable = true;

  database = {
    type = "sqlite3";
    path = "/var/lib/gitea/data/gitea.db";
  };

  settings = {
    server = {
      DOMAIN = "git.home";
      ROOT_URL = "http://git.home";
      HTTP_PORT = 3300;
      HTTP_ADDR = "127.0.0.1";
    };
    security = {
      SECRET_KEY = lib.mkForce secrets.giteaSecret;
      INTERNAL_TOKEN = lib.mkForce secrets.giteaInternalToken;
    };

    service = {
      DISABLE_REGISTRATION = true;
      REQUIRE_SIGNIN_VIEW = true;
    };
  };
};

# Ensure Gitea user/group exists
users.users.gitea = {
  isSystemUser = true;
  group = "gitea";
  home = "/var/lib/gitea";
  createHome = false;  # systemd/tmpfiles handles this
};
users.groups.gitea = {};


NIX

I went with SQLite instead of PostgreSQL. For a single-user Git server with fifteen repos, SQLite is more than enough and means one fewer service to manage. The lib.mkForce on the security keys is necessary because NixOS's Gitea module wants to generate its own — but I need the same keys on both servers for potential failover, so I force my own from secrets.nix.

Registration is disabled, sign-in is required to view anything. This is my personal Git server, not a community forge.

The Virtual Host Headache

This is where things got interesting. When I added the Gitea virtual host to nixos2's nginx config, navigating to git.home just dumped me into AdGuard Home instead.

The problem was layered:

Layer 1: DNS. I'd added a DNS rewrite in AdGuard Home on the main server pointing git.home to nixos2's IP. But nixos2 wasn't using the main server's AdGuard for DNS — it was pointed at Control D (76.76.2.2). So git.home resolved to nothing on nixos2 itself. An nslookup git.home on nixos2 came back NXDOMAIN. Fixed that by pointing nixos2's DNS to the main server.

Layer 2: Default virtual host. On nixos2, adguard2.home had default = true, making it the catch-all for any hostname that didn't match a configured virtual host. So even after DNS resolved correctly, if the nginx virtual host wasn't loading properly, everything fell through to AdGuard.

The fix was straightforward once I understood what was happening. Remove default = true from adguard2.home, make sure the Gitea virtual host was actually being imported, and ensure DNS was resolving through the right nameserver.

Here's the working virtual host on nixos2:

"git.home" = {
  locations."/" = {
    proxyPass = "http://127.0.0.1:3300";
    proxyWebsockets = true;

    extraConfig = ''
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
    '';
  };
};


NIX

The "git push" That Couldn't Find Home

Here's a fun one. After getting the web UI working, I went to push a repo to Gitea from the terminal. git push origin main. Nothing. Timeout. Meanwhile I'm literally staring at the Gitea web UI on my dev machine. What do you mean you can't find it? It's right there.

The browser was resolving git.home through AdGuard Home's DNS rewrite, which works for every device on the network that uses AdGuard as its DNS server. But the servers themselves can't use their own AdGuard for DNS — that would create a resolution loop (AdGuard needs DNS to start, but DNS is AdGuard). Both servers use Control D as their upstream resolver, and Control D has no idea what git.home is.

Same problem showed up on my Mac when I tried to push from there.

The fix is embarrassingly simple. On nixos2, where Gitea actually runs:

networking.extraHosts = ''
  127.0.0.1 git.home
'';


NIX

On the main server, point it to nixos2's IP:

networking.extraHosts = ''
  192.168.50.53 git.home
'';


NIX

And on the Mac, a plain old /etc/hosts entry. The CLI tools use the system resolver, which checks /etc/hosts (or networking.extraHosts on NixOS, which writes to /etc/hosts) before going out to DNS. So the request never leaves the local machine — it goes straight to the right IP without the whole router-to-AdGuard-rewrite-to-nginx dance.

It's the kind of thing that feels obvious in hindsight. Every other service on my network resolves through AdGuard rewrites and I never think about it. But the servers that run AdGuard don't get that luxury, and neither does any machine where you're using the CLI instead of just browsing.

Oh, By the Way — If You Use a Mac

After all of the above was working, my MacBook threw one more curveball. I could ping the Gitea server. I could curl http://git.home and get markup back. I could nc -zv port 80 and it succeeded. Safari loaded the page fine. But Vivaldi? "Address unreachable." Every single time.

I chased this for an embarrassing amount of time. Flushed DNS. Checked HSTS caches. Verified the hosts file was resolving correctly (dscacheutil -q host returned the right IP). Checked for proxy settings, DNS over HTTPS, secure connection upgrades — none of it. Every other machine on the network with Vivaldi accessed Gitea without a problem. Every diagnostic tool on the Mac said the connection should work. But Vivaldi just refused.

The culprit? macOS Sequoia's Local Network permission.

Recent versions of macOS require apps to have explicit permission to access devices on your local network. It's under System Settings → Privacy & Security → Local Network. Safari apparently had this permission already (either by default or from some long-forgotten dev task). Vivaldi did not. And macOS doesn't tell you that's why the connection is failing — it just silently blocks it, and the browser reports "address unreachable" like it's a network problem.

The fix: find Vivaldi (or Chrome, or whatever third-party browser you're using) in that Local Network list and flip the toggle on. Restart the browser. Done.

What makes this especially maddening:

  • Ping works. ICMP isn't subject to the Local Network permission.
  • curl works. Command-line tools aren't restricted by it.
  • Safari works. It already had the permission.
  • Other services on the DNS server machine work. The permission specifically gates access to other devices on the local network.
  • The error message is useless. "Address unreachable" tells you nothing about app permissions.

If you're running a homelab and accessing it from a Mac with a third-party browser, check this setting first before you start tearing apart your DNS and nginx configs. It'll save you a headache — and possibly save you from pulling a me and nuking your entire browser profile trying to fix it. I went down the road of completely uninstalling and reinstalling Vivaldi before finding the real cause. Themes gone. Extensions reset. Cache wiped. Nothing synced back. All my settings, back to factory defaults. Don't be like me. Check the Local Network permission first.

GitHub Mirroring

The whole point of running Gitea isn't to abandon GitHub — it's to not depend on it. I set Gitea as my primary remote and configured push mirroring to GitHub. The workflow:

  1. Push to Gitea (git push origin main)
  2. Gitea automatically mirrors to GitHub
  3. GitHub stays up-to-date as a public portfolio and backup

If GitHub ever "goes stupid" — account suspension, policy changes, acquisition drama — all my code is sitting on hardware I own, with backups being replicated by Syncthing. GitHub becomes the mirror, not the source of truth. (Future self: spoiler alert.)

Backing Up Gitea

Since Gitea uses SQLite, backups follow the same pattern as Vaultwarden: stop the service, snapshot the data, restart.

gitea = lib.mkIf enableBackups.gitea {
  initialize = true;
  repository = "/var/local/backups/restic";

  paths = [
    "/var/lib/gitea/data"           # Repos and database
    "/var/lib/gitea/custom/conf"    # Configuration
  ];

  exclude = [
    "/var/lib/gitea/data/sessions"
    "/var/lib/gitea/data/tmp"
  ];

  passwordFile = "/etc/nixos/private/restic-password";

  timerConfig = {
    OnCalendar = "02:00";
    Persistent = true;
  };

  pruneOpts = [
    "--keep-daily 7"
    "--keep-weekly 4"
    "--keep-monthly 6"
  ];

  backupPrepareCommand = ''
    ${pkgs.systemd}/bin/systemctl stop gitea
  '';

  backupCleanupCommand = ''
    ${pkgs.systemd}/bin/systemctl start gitea
  '';
};


NIX

This runs on nixos2 since that's where Gitea lives. The backup toggle on nixos2 has gitea = true while the main server has gitea = false — mirror image of most other services.

Making Monitoring Actually Smart

The Problem

Every time I added a new service, I had to remember to update the Prometheus scrape configs, add alert rules, and configure exporters. Forget any of those steps and you've got a service running with no monitoring — which defeats the entire purpose of having a monitoring stack.

Worse, if I disabled a service on one of the servers, Prometheus would keep trying to scrape it, fail, and fire false alerts. The monitoring config assumed every service existed, always.

Conditional Scrape Configs with lib.optionals

The fix is one of those things that makes you wonder why you didn't do it from the start. NixOS can check whether a service is enabled in the current configuration and conditionally include monitoring for it.

Here's how the scrape configs work in monitoring.nix:

scrapeConfigs =
  # Core monitoring (always on)
  [
    {
      job_name = "node";
      static_configs = [{
        targets = [ "127.0.0.1:${toString config.services.prometheus.exporters.node.port}" ];
      }];
    }
    {
      job_name = "prometheus";
      static_configs = [{
        targets = [ "127.0.0.1:${toString config.services.prometheus.port}" ];
      }];
    }
  ]
  # Optional: SearxNG
  ++ lib.optionals (config.services.searx.enable or false) [{
    job_name = "searx";
    static_configs = [{
      targets = [ "localhost:8888" ];
    }];
  }]
  # Optional: Nginx
  ++ lib.optionals config.services.nginx.enable [{
    job_name = "nginx";
    static_configs = [{
      targets = [ "127.0.0.1:${toString config.services.prometheus.exporters.nginx.port}" ];
    }];
  }]
  # Optional: Nextcloud
  ++ lib.optionals config.services.nextcloud.enable [
    {
      job_name = "nextcloud";
      static_configs = [{
        targets = [ "localhost:9205" ];
      }];
    }
  ]
  # Optional: Syncthing
  ++ lib.optionals config.services.syncthing.enable [{
    job_name = "syncthing";
    metrics_path = "/metrics";
    static_configs = [{
      targets = [ "127.0.0.1:8384" ];
    }];
    basic_auth = (import /etc/nixos/private/syncthing-secrets.nix).prometheus_auth;
  }]


NIX

The magic is lib.optionals. It takes a boolean condition and a list — if the condition is true, the list gets appended. If false, it returns an empty list. So when SearXNG is enabled on the main server, Prometheus scrapes it. When it's disabled on nixos2, Prometheus doesn't even know it exists.

The or false on some checks (like config.services.searx.enable or false) is a safety net for services where the .enable attribute might not exist in the config at all.

Conditional Exporters

Same pattern for the exporters themselves:

exporters = {
  node = {
    enable = true;
    enabledCollectors = [ "systemd" ];
    port = 9100;
  };

  nginx = lib.mkIf config.services.nginx.enable {
    enable = true;
    port = 9113;
    scrapeUri = "http://127.0.0.1:8080/nginx_status";
  };

  nextcloud = lib.mkIf config.services.nextcloud.enable {
    enable = true;
    url = "http://nextcloud.home:8280";
    username = "root";
    passwordFile = "/etc/nixos/private/nextcloud-admin-pass";
    port = 9205;
  };
};


NIX

lib.mkIf is the single-value version of lib.optionals — if the condition is false, the entire attribute set gets skipped. Node exporter is always on because I always want system metrics. Nginx and Nextcloud exporters only exist when those services do.

Conditional Alert Rules

This was the trickiest part. Alert rules in Prometheus are YAML strings, not Nix attribute sets, so I couldn't just use lib.mkIf on them directly. The solution was building the rule strings as variables with lib.optionalString and then concatenating them:

let
  systemAlertsRules = ''
    - alert: ServiceDown
      expr: up == 0
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: "Service down"
        description: "A service has been down for more than 2 minutes."

    - alert: DiskSpaceWarning
      expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 20
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Low disk space on root filesystem"
        description: "Root filesystem has less than 20 percent space remaining."
  '' + lib.optionalString config.services.nginx.enable ''

    - alert: NginxHighErrorRate
      expr: rate(nginx_http_requests_total{status=~"5.."}[5m]) > 0.05
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High Nginx 5xx error rate"
        description: "Nginx is returning too many 5xx errors."
  '';

  vaultwardenRules = lib.optionalString (config.services.vaultwarden.enable or false) ''
    - name: vaultwarden
      rules:
        - alert: VaultwardenDown
          expr: probe_success{instance="https://${secrets.tailscaleHostname}"} == 0
          for: 20m
          labels:
            severity: critical
          annotations:
            summary: "Vaultwarden password manager is down"
            description: "Vaultwarden has been unreachable for 20 minutes"
  '';

  nextcloudRules = lib.optionalString config.services.nextcloud.enable ''
    - name: nextcloud
      rules:
        - alert: NextcloudDown
          expr: probe_success{job="nextcloud-http"} == 0
          for: 20m
          labels:
            severity: critical
          annotations:
            summary: "Nextcloud is unreachable"
            description: "Nextcloud HTTP check has failed for 15 minutes"
  '';
in


NIX

Then in the Prometheus config:

rules = [
  ''
    groups:
      - name: system_alerts
        interval: 30s
        rules:
    ${systemAlertsRules}
    ${vaultwardenRules}
    ${nextcloudRules}
  ''
];


NIX

lib.optionalString returns the string if the condition is true, empty string if false. So on nixos2 where Vaultwarden and Nextcloud are disabled, those rule groups simply don't exist. No false alerts, no dead scrape targets, no manual cleanup when you add or remove a service.

The Blackbox Prober

The blackbox exporter deserves a special mention because its target list is also conditional:

static_configs = [{
  targets = lib.flatten [
    (lib.optional config.services.syncthing.enable "http://127.0.0.1:8384")
    (lib.optional (config.services.adguardhome.enable or false) "http://127.0.0.1:3000")
    (lib.optional (config.services.vaultwarden.enable or false) "https://${secrets.tailscaleHostname}")
  ];
}];


NIX

lib.optional (singular, not optionals) returns a single-element list or an empty list, and lib.flatten squashes it all into one clean target list. Only services that are actually running get health-checked.

Why This Matters

The same monitoring.nix file now works on both servers without modification. On the main server with everything running, Prometheus monitors everything. On nixos2 with just Gitea, Syncthing, and AdGuard, it monitors just those. Add a service? Monitoring shows up automatically. Remove one? It disappears. No more "I forgot to update the scrape config" incidents.

The Syncthing Permissions Fight

The Problem

Syncthing started throwing permission errors. Specifically, it couldn't write to directories that other services also needed to access. The root cause was NixOS doing exactly what it's supposed to do — isolating services from each other with dedicated system users.

Here's the conflict: Syncthing runs as ppb1701. Restic backups run as root. NoteDiscovery runs as notediscovery. All of them need to touch overlapping directories. When Syncthing syncs a file, it creates it with ppb1701:syncthing ownership. When NoteDiscovery tries to read it, permission denied. When Restic's backup job finishes and sets the backup directory to be readable by the syncthing group, the next service to write there might not have group membership.

The Fix: Groups, Setgid, and Tmpfiles

The solution was a combination of three things.

Shared group membership. Services that need to access Syncthing-managed directories get added to the syncthing group:

users.users.notediscovery = {
  isSystemUser = true;
  group = "notediscovery";
  extraGroups = [ "syncthing" "users" ];
  home = "/var/lib/notediscovery";
  createHome = true;
};


NIX

UMask for group-writable files. Syncthing by default creates files that are only writable by the owner. Setting UMask = "0002" on relevant services means new files are group-writable:

serviceConfig = {
  UMask = "0002";  # Creates files with group write by default
};


NIX

Setgid bit on directories. The setgid bit ensures that new files created in a directory inherit the directory's group, not the creating user's primary group. Without this, Syncthing creates files as ppb1701:syncthing but NoteDiscovery creates files as notediscovery:notediscovery in the same directory, and suddenly nobody can read each other's stuff.

Tmpfiles rules for declarative permissions. Instead of hoping scripts run at the right time, I use systemd tmpfiles to declare what the permissions should be:

systemd.tmpfiles.rules = [
  "d /var/lib/notediscovery 0755 notediscovery notediscovery -"
];


NIX

And for the Restic backup directories that Syncthing needs to replicate:

systemd.tmpfiles.rules = [
  "d /var/local/backups 0755 root root -"
  "d /var/local/backups/restic 0750 root syncthing -"
];


NIX

Each backup job also has a post-backup step that resets permissions:

restic-backups-linkwarden.postStart = ''
  ${pkgs.coreutils}/bin/chmod -R g+rX /var/local/backups/restic/
  ${pkgs.coreutils}/bin/chgrp -R syncthing /var/local/backups/restic/
'';


NIX

The Lesson

NixOS's service isolation is a feature, not a bug. Each service runs as its own user with minimal permissions. That's great for security. But when services need to share directories — which happens a lot when Syncthing is replicating data between machines — you have to explicitly design the permission model.

The key insight: don't fight the isolation model. Work with it. Use groups for shared access, setgid for consistent ownership, umask for sensible defaults, and tmpfiles rules to make it all declarative. The permissions are now part of the NixOS config, not something I set up once and pray doesn't drift.

Lessons Learned

  1. Not everything needs to be self-hosted. The RSS attempt taught me that the goal is self-hosting what makes sense, not self-hosting everything on principle.
  2. Conditional configs are worth the upfront investment. Building monitoring with lib.optionals and lib.mkIf means I'll never forget to update Prometheus when I add or remove a service. The config adapts automatically.
  3. DNS is always the problem. The Gitea virtual host issue was ultimately a DNS resolution problem. When in doubt, nslookup first.
  4. Permission models need to be designed, not discovered. Syncthing + service isolation + shared directories = you need to think about groups, setgid, and umask upfront, not after the permission denied errors start flying.
  5. SQLite is fine. For single-user services like Gitea and Vaultwarden, SQLite is simpler, requires no extra service, and backs up with a simple file copy (well, service stop, file copy, service start). Don't reach for PostgreSQL unless you actually need it.
  6. Back up your Syncthing certificates. Seriously. cert.pem and key.pem in your Syncthing config directory are what determine your Device ID. Lose them and every device on your network needs to be updated with the new ID — and if those devices are NixOS machines, that means config edits and rebuilds, not just clicking a button in the web UI. Ask me how I know. Actually, don't. I'll tell you in a future post.

What's Next

The server stack is in a solid place now. Both machines are monitored, backed up, replicated, and the configs are smart enough to adapt when services change. Next up: I'm adding Homepage as a dashboard — something I can pull up on my phone and get a quick "is everything alive?" glance without opening Grafana. I'm also going to map out the full backup topology across every machine in the fleet, because remember that thing I said about backing up your Syncthing certificates? Yeah. I have a story about that. And macOS is going to be the villain — again. But First...I have some bugs to fix on the app I'm building so I'll circle back to that story soon.

Have you self-hosted your own Git server? Gone down the monitoring rabbit hole? Let me know! Find me at @ppb1701@ppb.social on Mastodon.

← Back to Series

Next: [Part 12 — Homepage, Backup Topology, and Why macOS Can't Always Be Trusted](coming soon)

Main Server (nixos): Codeberg
Second Server (nixos2): Codeberg
ISO can be gotten here.

blog git gitea grafana homelab homeserver monitoring nixos permissions prometheus