Fleet convergence — auto-update, release & auto-upgrade triage#

This is the runbook behind the AutoUpgradeCompareFailed and AutoUpgradeStuckBehindRelease alerts — and, more generally, for any “why is this host still on last month’s closure?” question. The pipeline’s failure mode is quiet: the anti-downgrade guard fails closed, so a broken credential, mirror, or release tip doesn’t page by itself — hosts just stop moving. That is exactly what happened 2026-08-12..16 (expired compare PAT, plus an unbuildable release), which is why every guard run now writes a verdict metric and why those two alerts exist.

The three stages#

STAGE 1  ereshkigal: flake-auto-update.timer (1st & 15th, 03:00 + 30min jitter)
         └─ nix run .#update
              scoped input update (public inputs only; git+ssh inputs stay pinned)
              → full `om ci run` (the ONLY thing that builds host closures)
              → PR on automation/flake-update; green ⇒ squash auto-merge
                                   │ merge to master
STAGE 2  GitHub Actions release.yml (push to master touching flake.lock)
         PROVENANCE GATE: HEAD must have landed via a MERGED PR —
         a direct push to master does NOT move the fleet pointer.
         → CalVer tag v<YYYY.MM.DD>[.N] + force-push HEAD:release
         Escape hatch: `gh workflow run release.yml` (deliberate human action)
                                   │
STAGE 3  each NixOS host: nixos-upgrade (default 04:30 nightly)
         fetches git+http://voile:8418/tsunaminoai/nix-flake-final?ref=release
         guard compares current rev vs release tip via the SAME mirror's
         compare API — switches ONLY when provably behind (fail-closed)
  • Stage 1 — tsunaminoai.ci.autoUpdate on ereshkigal. The PR is opened before CI runs so a failure is never silent; a red build leaves the PR open with a root-cause comment and never merges.
  • Stage 2 — release is the fleet’s convergence target, so only om-ci-gated commits may advance it. GitHub’s “Full Flake Check” builds neither host toplevels nor the VM checks — green GitHub CI proves nothing about whether a lock bump builds. The 2026-08-16 unbuildable release shipped precisely by pushing flake.lock straight to master past a red om ci; the provenance gate makes that a no-op now.
  • Stage 3 — tsunaminoai.autoUpgrade wraps system.autoUpgrade, pinned to the voile Forgejo mirror (tsunaminoai.autoUpgrade.flakeRef) for both fetch and compare (tsunaminoai.autoUpgrade.mirrorApi, Forgejo’s unauthenticated compare API). No GitHub token anywhere in the guard — the old GitHub-compare + PAT design is how one expired token froze the whole fleet silently. Comparing against the same source it fetches from is self-consistent by construction.

The guard switches only on the one provable state — behind > 0 && ahead == 0. Ahead, diverged, dirty/unknown current rev, unresolvable release ref, or an unreachable compare API all skip (the previous fail-open behaviour reverted a deliberate deploy on 2026-08-12, rolling ereshkigal’s gen 409 back to 408 and taking the netns work with it).

Guard observability#

Every guard run — every exit path — writes to the node-exporter textfile collector:

  • auto_upgrade_guard{result=…} 1 — one of upgrade, ahead, identical, diverged, compare_failed, dirty, unresolved
  • auto_upgrade_commits_behind_release — commits the release tip has that the host does not

Two vmalert rules watch them (both severity=warning, see telemetry):

AutoUpgradeCompareFailed (result compare_failed/unresolved, 48h)#

The guard cannot resolve the release ref or cannot compare against the mirror — the host has silently stopped converging (fail-closed). First three commands:

ssh <host> journalctl -u nixos-upgrade -e        # the guard says exactly why it skipped
ssh <host> curl -fsS 'http://voile:8418/api/v1/repos/tsunaminoai/nix-flake-final/compare/<current_rev>...<release_rev>'
curl -fsS http://voile:8418/api/v1/version        # is the Forgejo mirror itself up / mirroring?

Usual causes: voile (or its Forgejo container) down, the mirror sync stalled so the host’s rev is unknown to it (404 → empty compare), or the host is running a dirty working-tree rev from a deploy-rs push (expected dirty skip, not a fault — it clears on the next release that supersedes the deploy).

AutoUpgradeStuckBehindRelease (behind > 0, 72h)#

The guard can compare and the host is provably behind, yet never converges — the nightly switch itself is failing. Most likely the release tip does not build on this host, or the run keeps getting skipped. First three commands:

ssh <host> journalctl -u nixos-upgrade -e         # find the failing build/switch
ssh <host> systemctl list-timers nixos-upgrade    # is the timer even firing?
nix log <failing-drv>                             # root-cause the build failure

om ci triage#

  • om ci is the only gate that builds host closures. Never bypass a red om ci by pushing flake.lock to master — and since the provenance gate, doing so no longer moves release anyway.
  • “Cannot build X” lines are scheduler cascade, not causes. Of the .drv paths named on error lines, only those nix has a build log for actually ran and failed. Root-cause with nix log <drv> (works even for remote-builder failures) or rebuild locally with --builders "" --max-jobs 4.
  • The PR failure comment now does this extraction itself: it posts up to three root failures (drvs with build logs, last 30 lines each), skipping the cascade noise, plus the last 80 lines of CI output in a collapsible block. The 2026-08-15 comment predating this was a wall of cascade that led triage to disable a bystander VM check instead of the one real error behind it.

Manual interventions#

Re-run stage 1#

On ereshkigal: trigger-flake-auto-update (starts the systemd unit with its proper environment). Running nix run .#update by hand as root breaks on “No writable cache directories” and other env differences; from your own user in a clean checkout it works, and nix run .#update -- --dry-run exercises the whole pipeline without pushing.

Advance the release deliberately#

gh workflow run release.yml — the workflow_dispatch path skips the merged-PR provenance check. This is a deliberate human action; make sure the commit actually builds fleet-wide first.

Deploy a host now (push path)#

deploy is NOT on PATH; resolve the locked input, and always raise the confirm timeout — the node default (60s) rolled back a large ereshkigal switch mid-activation:

DRPATH=$(nix build --no-link --print-out-paths --impure \
  --expr '(builtins.getFlake "'$PWD'").inputs.deploy-rs.packages.x86_64-linux.deploy-rs')
"$DRPATH/bin/deploy" .#<host> -s --confirm-timeout 120
  • deploy-rs ships the working tree. To deploy committed state from a dirty tree: "$DRPATH/bin/deploy" "git+file://$PWD?rev=<sha>#<host>" -s --confirm-timeout 120. (A working-tree deploy leaves the host on a -dirty rev, which the stage-3 guard then skips by design until a release supersedes it.)
  • Magic rollback is on: a failed confirm reverts automatically, no outage.
  • mokou rejects SSH-to-self — when working on mokou use nh os switch .#.
  • myon and the aarch64 Pis never build on-box (remoteBuild = false via lowMemoryHosts in modules/flake/deploy/default.nix) — closures are built locally and pushed.

Gotchas#

  • The libvirt-from-unstable overlay trap (broke the 2026-08 bump): never pull a library with same-version Python bindings (libvirt being the proven case) from unstable alone — the stable bindings’ generator hard-fails against the newer lib’s API XML. Bump both sides together or neither.
  • The stage-1 github/automation-token PAT (fine-grained: Contents RW, Pull requests RW, Metadata R) is used ONLY by the scheduler on ereshkigal — the stage-3 guard needs no GitHub at all anymore. After any rotation, verify before deploying: curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $(sops --decrypt --extract '["github"]["automation-token"]' secrets.yaml)" https://api.github.com/repos/tsunaminoai/nix-flake-final must print 200 — the 2026-08 rotation shipped a dead token silently.
  • Piping build output (| tee, | tail) eats exit codes without pipefail — gate on the command, not the pipe.
  • nix run .#update refuses dirty trees and (live runs) non-master branches by design; private git+ssh inputs (corp/praxis) are left pinned because the automation host has no SSH creds for them.
  • Deployment — the full deploy system this pipeline is the “eventually” half of (deploy-rs, deploy-* apps, the release workflow).
  • Telemetry — where the guard metrics land and the full alert-rule catalogue.
  • Alerts & Notifications — how the two AutoUpgrade warnings reach you (ntfy; warnings never hit Discord).
  • Gitea — the voile Forgejo mirror that stage 3 both fetches from and compares against.