Deployment System#

A comprehensive deployment solution with monitoring and automation for your NixOS and nix-darwin fleet.

Features#

🚀 Automated Deployments - Deploy to all hosts or specific hosts with one command
📊 Fleet Monitoring - Real-time health checks and status dashboard
🔄 Auto-rollback - Automatic rollback on deployment failures via magic rollback
🍎 Darwin Support - Full support for macOS hosts with proper builder user handling
🔐 Secure by Default - Uses sops-encrypted deploy keys
🌐 Tailscale Integration - All hosts accessible via Tailscale network
🤖 Scheduled Updates and Releases - Twice-monthly nix flake update → full om ci → auto-merged PR → CalVer tag → fleet convergence (see Automated Flake Updates and Releases)

Quick Start#

Initial Setup#

Ensure deploy key is configured in your sops secrets (secrets.yaml):

nix:
  deploy:
    priv-key: <your-encrypted-ssh-private-key>
    pub-key: ssh-ed25519 AAAA...

Enable deploy node on a host that will perform deployments:
```
# In your host configuration
tsunaminoai.nix.isDeployNode = true;
```
This installs the deploy key to /root/.ssh/id_deploy via sops.
Build and test locally first:
```
nix flake check
```

Deployment Commands#

Command	Description
`nix run .#deploy`	Raw deploy-rs access
`nix run .#deploy-host`	Deploy to a specific host (interactive)
`nix run .#deploy-host <name>`	Deploy to a named host
`nix run .#deploy-all`	Deploy to all configured hosts
`nix run .#deploy-monitor`	Check status of all hosts
`nix run .#deploy-rollback`	Rollback a host to previous generation
`nix run .#update`	Update inputs → full `om ci` → open/auto-merge a PR (see below)
`nix run .#update -- --dry-run`	Update + CI locally, report, then restore `flake.lock` (no push)

Monitor Fleet Status#

nix run .#deploy-monitor

Shows connectivity and system info for all hosts.

Deploy to Specific Host#

# Interactive selection
nix run .#deploy-host

# Or specify directly
nix run .#deploy-host shinobu
nix run .#deploy-host shinobu switch  # or boot, test

Deploy to All Hosts#

nix run .#deploy-all

This will: 1. Run pre-deployment health check 2. Prompt for confirmation 3. Deploy to all reachable hosts 4. Show post-deployment status

Rollback a Host#

# Interactive selection
nix run .#deploy-rollback

# Or specify directly
nix run .#deploy-rollback shinobu

Using deploy-rs Directly#

The system also exposes native deploy-rs configuration:

# Deploy all hosts
nix run .#deploy -- .

# Deploy specific host
nix run .#deploy -- .#shinobu

# Skip checks (faster)
nix run .#deploy -- --skip-checks .#mokou

# Dry run
nix run .#deploy -- --dry-activate .

The battle-tested invocation#

In practice deploy is not on PATH outside a nix run; resolve the locked input once, and always raise the confirm timeout — the per-node default (60s, confirmTimeout in modules/flake/deploy/default.nix) rolled back a large ereshkigal switch that simply took longer than a minute to settle:

DRPATH=$(nix build --no-link --print-out-paths --impure \
  --expr '(builtins.getFlake "'$PWD'").inputs.deploy-rs.packages.x86_64-linux.deploy-rs')
"$DRPATH/bin/deploy" .#<host> -s --confirm-timeout 120

Realities to keep in mind:

deploy-rs deploys the working tree. To deploy committed state from a dirty tree, pin the rev: "$DRPATH/bin/deploy" "git+file://$PWD?rev=<sha>#<host>" -s --confirm-timeout 120. (A working-tree deploy also leaves the host on a -dirty revision, which the auto-upgrade guard then skips by design.)
Magic rollback is on: a failed confirm reverts automatically — no outage, but your deploy silently didn’t land; check before walking away.
mokou rejects SSH-to-self: when working on mokou, use nh os switch .# instead of deploy-rs.

Automated Flake Updates and Releases#

The fleet keeps itself current with a scheduled, CI-gated pipeline: inputs are updated on a cadence, the whole fleet is built with om ci, and only a green build is proposed — and auto-merged — as a pull request. Each merge is tagged, and hosts converge on that tagged release automatically (“eventually”) or on demand (“deliberately”).

[ereshkigal timer — 1st & 15th @ 03:00]
  └─ nix run .#update
       1. nix flake update
       2. push flake.lock → automation/flake-update, open/refresh ONE PR
       3. om ci run                     ← full fleet build (local cache + builders)
       4. CI green? ── yes ─► ✅ comment + enable squash auto-merge
                    └─ no ──► ❌ comment with failing derivations + log tail
                              (PR left open, NOT merged)
                         │
                         ▼  (auto-merge once GitHub's required check is green)
[squash-merge to master]
                         │
                         ▼
[GitHub Actions: release.yml]  on a master push that changes flake.lock
  • provenance gate: HEAD must have landed via a MERGED PR
  • tag  v<YYYY.MM.DD>[.N]   (CalVer)
  • fast-forward the `release` branch to the tagged commit
                         │
         ┌───────────────┴────────────────┐
         ▼                                 ▼
[pull — "eventually"]              [push — "deliberately"]
 tsunaminoai.autoUpgrade →          nix run .#deploy-all / .#deploy-host
 the voile Forgejo mirror's         (unchanged)
 `release` ref, nightly, with a
 fail-closed anti-downgrade guard

Cadence#

The scheduler runs on ereshkigal — the only host that can build the full fleet (it has the binary cache, FlakeHub netrc, and remote builders). It fires twice a month, on the 1st and 15th at 03:00 (plus 30 min of jitter):

tsunaminoai.ci.autoUpdate = {
  enable = true;
  schedule = "*-*-1,15 03:00"; # systemd OnCalendar
};

The `nix run .#update` entrypoint#

The all-in-one command the timer runs (also runnable by hand from a clean checkout):

Step	Action
1	`nix flake update` — re-resolve inputs. Exits early if `flake.lock` is unchanged.
2	Push the lock bump to the fixed `automation/flake-update` branch and open/refresh one PR.
3	`om ci run` — build every host in the fleet.
4	Pass → ✅ comment + enable squash auto-merge. Fail → ❌ comment with the cause.

nix run .#update -- --dry-run performs the update and CI, reports the outcome (printing the failure summary on failure), then restores flake.lock without pushing — the safest way to exercise the pipeline by hand.

Scope the CI command if some remote builders aren’t reliably online (e.g. Darwin):

tsunaminoai.ci.autoUpdate.ciCommand = "om ci run '.#default'"; # or export UPDATE_CI_CMD

Failure reporting#

Because the PR is opened before CI runs, a failing update is never silently dropped — it lands as an open PR with a root-cause comment. The comment does the nix log extraction itself: of the .drv paths named on error lines, only those nix has a build log for actually ran and failed — the rest are scheduler cascade (“Cannot build X” dependents). It posts up to three such root failures (last 30 log lines each), falling back to raw error lines for eval failures, plus the last 80 lines of CI output in a collapsible block. Auto-merge is armed only on a green build, so a broken lock cannot merge. The on-prem om ci is the real gate — it is the only thing that builds host closures; GitHub’s Nix Flake Checks (checks.yml) remains the required status check that auto-merge waits on, but a green GitHub run proves nothing about whether a lock bump builds the fleet. Triage details: fleet-convergence runbook.

Tags and the `release` branch#

.github/workflows/release.yml runs when a push to master changes flake.lock (i.e. when the automation PR merges). It:

checks provenance: the pushed HEAD must have landed via a merged PR — a commit pushed directly to master does not advance the release (the push itself stays allowed; the fleet pointer just doesn’t move),
creates a CalVer tag v<YYYY.MM.DD> (with a .N suffix for same-day re-releases), and
fast-forwards the release branch to that commit.

The provenance gate exists because the om-ci-gated PR flow is the only path that builds host closures — on 2026-08-16 an unbuilt flake.lock pushed straight to master became the fleet-wide release while its closure couldn’t build. The deliberate escape hatch is gh workflow run release.yml (workflow_dispatch skips the check — a human action, taken knowingly).

release is the moving “latest release” pointer that hosts follow; the tag is the human-readable / rollback handle for the same commit.

Host convergence#

Pull — “eventually”. Any NixOS host can converge on the release by itself:

tsunaminoai.autoUpgrade.enable = true;

This wraps system.autoUpgrade (nightly at 04:30, allowReboot = false) pinned to git+http://voile:8418/tsunaminoai/nix-flake-final?ref=release — the internal Forgejo mirror on voile (tsunaminoai.autoUpgrade.flakeRef; the repo is private and hosts have no GitHub fetch token, so a github: ref would 404 every night) — plus a fail-closed anti-downgrade guard: before switching it compares the running host’s configurationRevision against the release tip via the same mirror’s unauthenticated compare API (tsunaminoai.autoUpgrade.mirrorApi). Only a host that is provably behind the release (behind > 0, ahead = 0) switches; ahead, diverged, dirty/unknown revisions, an unresolvable release ref, or an unreachable compare API all skip. Every run writes its verdict to the node-exporter textfile collector (auto_upgrade_guard{result=…}, auto_upgrade_commits_behind_release), and the AutoUpgradeCompareFailed / AutoUpgradeStuckBehindRelease alerts page before “quietly frozen fleet” becomes a surprise — triage in the fleet-convergence runbook. Darwin hosts have no autoUpgrade equivalent and use the push path.

Push — “deliberately”. The existing nix run .#deploy-all / .#deploy-host (above) are unchanged; run them from a checkout at the tag to move hosts on demand.

Prerequisites#

Requirement	Why
`github/automation-token` in `secrets.yaml` (fine-grained PAT: contents + pull-requests)	The scheduler (stage 1, ereshkigal only) uses it for git-over-HTTPS + `gh`. It is the only consumer — the per-host upgrade guard fetches and compares against the Forgejo mirror and needs no GitHub token at all.
Branch protection on `master` requiring Nix Flake Checks, and “Allow auto-merge” enabled on the repo	Gives `gh pr merge --auto` a check to gate on. Without it the PR still opens, but won’t auto-merge.

Monitoring#

CLI Monitor#

The deploy-monitor command provides real-time fleet status:

nix run .#deploy-monitor

Output includes for each host: - Connectivity status (online/offline) - System type (NixOS/Darwin) - Current generation number - NixOS/Darwin version

Manual Status Check#

SSH into individual hosts to check status:

# NixOS
ssh root@<host>.armadillo-banfish.ts.net 'readlink /nix/var/nix/profiles/system'

# Darwin
ssh builder@<host>.armadillo-banfish.ts.net 'readlink /nix/var/nix/profiles/system'

Configuration#

Deploy Node Configuration#

Configure hosts as deploy nodes in your host configuration:

tsunaminoai = {
  nix.isDeployNode = true;
};

This will: - Deploy the sops-managed SSH key to /root/.ssh/id_deploy - Allow this host to run deploy commands

Deploy Target Configuration#

Targets are automatically discovered from your flake’s nixosConfigurations and darwinConfigurations.

NixOS targets need: 1. SSH access for root user 2. Deploy public key in authorized_keys

Darwin targets need: 1. SSH access for builder user 2. builder user must have passwordless sudo 3. Deploy public key in authorized_keys

Architecture#

Module Structure#

modules/flake/deploy/
└── default.nix          # Main deploy-rs flake module

modules/flake/update/
└── default.nix          # `nix run .#update`: update inputs → om ci → auto-merged PR

modules/flake/nix/
└── deploy.nix           # Sops deploy key configuration

modules/flake/checks/
└── deploy.nix           # VM integration tests

modules/nixos/deployment/
├── auto-deploy.nix      # Central git-pull → deploy-all puller (dormant)
├── auto-update.nix      # Scheduled update timer (tsunaminoai.ci.autoUpdate)
└── auto-upgrade.nix     # Per-host pull to the release tag (tsunaminoai.autoUpgrade)

.github/workflows/
└── release.yml          # CalVer tag + advance the `release` branch on merge

Deploy Flake Module (`modules/flake/deploy/default.nix`)#

The main deployment module provides:

Flake Outputs: - deploy.nodes - Auto-generated deploy-rs node configuration from nixosConfigurations and darwinConfigurations

Apps (per-system): - deploy - Raw deploy-rs binary - deploy-host - Interactive host deployment - deploy-all - Fleet-wide deployment - deploy-monitor - Health check dashboard - deploy-rollback - Generation rollback

Packages (per-system): - deploy-system - Unified NixOS/Darwin deployment script - deploy-monitor - Status monitoring script - deploy-all - Batch deployment script - deploy-host - Single-host deployment - deploy-rollback - Rollback script

Node Configuration#

Nodes are automatically generated from your flake’s system configurations:

# Auto-discovered from:
self.nixosConfigurations   # → type: "nixos", user: "root"
self.darwinConfigurations  # → type: "darwin", user: "builder"

Each node is configured with:

Setting	NixOS	Darwin
`sshUser`	`root`	`builder`
`user`	`root`	`builder`
`hostname`	`<name>.armadillo-banfish.ts.net`	same
`remoteBuild`	`true` (see note)	`true`
`autoRollback`	`true`	`true`
`magicRollback`	`true`	`true`
`activationTimeout`	900s (15min)	same
`confirmTimeout`	60s	same

remoteBuild is computed per-node from the host’s capability, not just its architecture: it is true for everything except aarch64-linux hosts (e.g. the octopi Pi) and the hosts listed in lowMemoryHosts (modules/flake/deploy/default.nix) — currently myon, a 1 GB VPS that OOMs even evaluating its own system. Both classes are built locally with their closure pushed.

SSH Key Management#

The deploy key is managed via sops-nix:

# modules/flake/nix/deploy.nix
sops.secrets."nix/deploy/priv-key" = lib.mkIf cfg.isDeployNode {
  mode = "0400";
  path = "/root/.ssh/id_deploy";
};

Enable on a host to make it a deploy node:

tsunaminoai.nix.isDeployNode = true;

Darwin-Specific Handling#

Darwin hosts require special handling because: 1. Root SSH is disabled on macOS 2. Activation requires sudo

The module automatically: - Uses builder user for SSH - Runs activation with sudo - Handles Darwin-specific profile paths

Deployment Flow#

┌─────────────────┐
│   Deploy Node   │
│  (any host with │
│  isDeployNode)  │
└────────┬────────┘
         │
         ├─────► deploy-monitor
         │       └─► SSH to each host
         │           └─► Detect host type (NixOS/Darwin)
         │               └─► Query generation & version
         │
         ├─────► deploy-host / deploy-all
         │       └─► Build closure (NixOS: locally → copy to target;
         │           Darwin: on target, remoteBuild=true)
         │               └─► Activate with rollback
         │
         └─────► deploy-rollback
                 └─► SSH to host
                     └─► nix-env --rollback
                         └─► switch-to-configuration

Network Requirements#

Deploy nodes must be able to SSH to all target hosts on port 22
Hosts must be reachable via <hostname>.armadillo-banfish.ts.net (Tailscale)

Troubleshooting#

Deployment Fails#

Check SSH connectivity:

For NixOS hosts:

ssh -i /root/.ssh/id_deploy root@<host>.armadillo-banfish.ts.net

For Darwin hosts:

ssh -i /root/.ssh/id_deploy builder@<host>.armadillo-banfish.ts.net

Verify build succeeds locally:

# NixOS
nix build .#nixosConfigurations.<host>.config.system.build.toplevel

# Darwin
nix build .#darwinConfigurations.<host>.config.system.build.toplevel

Check deploy-rs directly:

nix run .#deploy -- .#<hostname> --dry-activate

Darwin Deployment Issues#

“Permission denied” / “not privileged to build” errors: - The builder user must be in the staff group — the determinate-nix daemon trusts via @staff, not just by username. - This is handled by the activation script in modules/flake/nix/builder/darwin.nix. If it didn’t run, trigger it manually:
```
sudo dseditgroup -o edit -a builder -t user staff
```
SSH fails as builder (connection closed after key auth): - The builder user must be in the com.apple.access_ssh group (macOS PAM requirement). The activation script handles this, but to check:
```
dscl . -read /Groups/com.apple.access_ssh GroupMembership
```

Verify the authorized key is in place:

cat /etc/ssh/nix_authorized_keys.d/builder

nix: command not found over SSH: - Non-login SSH sessions don’t source any profile. The SetEnv PATH=... in /etc/ssh/sshd_config.d/50-nix-builder.conf handles this. Verify it exists on the Darwin host.

Host Shows Offline in Monitor#

Check Tailscale connectivity:
```
tailscale ping <hostname>
```

Verify host is running:

ping <hostname>.armadillo-banfish.ts.net

Check SSH service:

ssh root@<hostname>.armadillo-banfish.ts.net systemctl status sshd

Rollback Issues#

Manual rollback:

# NixOS
ssh root@<host> 'nix-env -p /nix/var/nix/profiles/system --rollback && /nix/var/nix/profiles/system/bin/switch-to-configuration switch'

# Darwin
ssh builder@<host> 'sudo nix-env -p /nix/var/nix/profiles/system --rollback && sudo /nix/var/nix/profiles/system/activate'

Security Considerations#

Deploy key has root access to all hosts
Store deploy key securely in sops-nix (at nix/deploy/priv-key)
Limit deploy nodes to trusted hosts via isDeployNode
Use Tailscale for secure network access
Monitor deployment activity

Migration from Manual Deployment#

If you were previously deploying manually with nixos-rebuild:

Ensure deploy key is in sops (nix/deploy/priv-key)
Enable tsunaminoai.nix.isDeployNode = true on at least one host
Rebuild that host to deploy the key
Test with nix run .#deploy-host <hostname> on a single host
Once verified, use nix run .#deploy-all for fleet-wide updates

VM Testing#

The deployment infrastructure includes VM integration tests:

# Run the deployment test (also bundled into nix build .#checks-full)
nix build .#checks-vm-deploy-test

# Run interactively
nix build .#checks-vm-deploy-test.driverInteractive
./result/bin/nixos-test-driver

The test verifies: - Deploy-rs node configuration - SSH connectivity between nodes - Key distribution

Future Enhancements#

Planned improvements: - [x] Notification integration (Slack/Discord) — webhook hooks on tsunaminoai.ci.autoUpdate and auto-deploy - [x] Prometheus metrics export — the auto-upgrade guard writes auto_upgrade_guard{result=…} / auto_upgrade_commits_behind_release via the node-exporter textfile collector, alerted on by AutoUpgrade* (see the fleet-convergence runbook) - [ ] Deployment history tracking - [x] CI/CD pipeline integration — see Automated Flake Updates and Releases - [ ] Web dashboard for monitoring

Fleet convergence runbook — triage for the auto-update pipeline and the AutoUpgrade* alerts, plus the manual-intervention playbook.
Telemetry — where the deploy/upgrade guard metrics land and the alert-rule catalogue.
Gitea — the voile Forgejo mirror hosts converge from.
deploy-rs Documentation
NixOS Deployment Options
Maintenance Guide