Fleet convergence — auto-update, release & auto-upgrade triage#

This is the runbook behind the AutoUpgradeCompareFailed and AutoUpgradeStuckBehindRelease alerts — and, more generally, for any “why is this host still on last month’s closure?” question. The pipeline’s failure mode is quiet: the anti-downgrade guard fails closed, so a broken credential, mirror, or release tip doesn’t page by itself — hosts just stop moving. That is exactly what happened 2026-08-12..16 (expired compare PAT, plus an unbuildable release), which is why every guard run now writes a verdict metric and why those two alerts exist.

The three stages#

STAGE 1  ereshkigal: flake-auto-update.timer (1st & 15th, 03:00 + 30min jitter)
         └─ nix run .#update
              scoped input update (public inputs only; git+ssh inputs stay pinned)
              → full `om ci run` (the ONLY pipeline that builds host closures;
                corp is STUBBED here — see Stage 1 below)
              → PR on automation/flake-update; green ⇒ squash auto-merge
                                   │ merge to master
STAGE 2  GitHub Actions release.yml (push to master touching flake.lock)
         PROVENANCE GATE: HEAD must have landed via a MERGED PR —
         a direct push to master does NOT move the fleet pointer.
         → CalVer tag v<YYYY.MM.DD>[.N] + force-push HEAD:release
         Escape hatch: `gh workflow run release.yml` (deliberate human action)
                                   │
STAGE 3  each NixOS host: nixos-upgrade (default 04:30 nightly)
         fetches git+http://voile:8418/tsunaminoai/nix-flake-final?ref=release
         guard compares current rev vs release tip via the SAME mirror's
         compare API — switches ONLY when provably behind (fail-closed)
  • Stage 1 — tsunaminoai.ci.autoUpdate on ereshkigal. The PR is opened before CI runs so a failure is never silent; a red build leaves the PR open with a root-cause comment and never merges. Caveat: ereshkigal has no SSH key for the private sc2in/corp input, so its command is om ci run . -- --override-input flake/corp path:./ci/stub-corp (hosts/x86_64-nixos/ereshkigal/default.nix). ci/stub-corp is a no-op sc2-compliance module, so this om ci proves shinobu builds but not that the real SC2 compliance baseline evaluates; only GitHub’s checks.yml (which holds SC2_CI_SSH_KEY) sees the real corp — and GitHub builds no toplevels. Net effect: no pipeline builds shinobu with the real corp module; do that by hand on a host with the sc2 key when corp changes.
  • Stage 2 — release is the fleet’s convergence target, so only om-ci-gated commits may advance it. GitHub’s “Full Flake Check” builds neither host toplevels nor the VM checks — green GitHub CI proves nothing about whether a lock bump builds. The 2026-08-16 unbuildable release shipped precisely by pushing flake.lock straight to master past a red om ci; the provenance gate makes that a no-op now.
  • Stage 3 — tsunaminoai.autoUpgrade wraps system.autoUpgrade, pinned to the voile Forgejo mirror (tsunaminoai.autoUpgrade.flakeRef) for both fetch and compare (tsunaminoai.autoUpgrade.mirrorApi, Forgejo’s unauthenticated compare API). No GitHub token anywhere in the guard — the old GitHub-compare + PAT design is how one expired token froze the whole fleet silently. Comparing against the same source it fetches from is self-consistent by construction.

The guard switches only on the one provable state — behind > 0 && ahead == 0. Ahead, diverged, dirty/unknown current rev, unresolvable release ref, or an unreachable compare API all skip (the previous fail-open behaviour reverted a deliberate deploy on 2026-08-12, rolling ereshkigal’s gen 409 back to 408 and taking the netns work with it).

Guard observability#

Every guard run — every exit path — writes to the node-exporter textfile collector:

  • auto_upgrade_guard{result=…} 1 — one of upgrade, ahead, identical, diverged, compare_failed, dirty, unresolved
  • auto_upgrade_commits_behind_release — commits the release tip has that the host does not

Two vmalert rules watch them (both severity=warning, see telemetry):

AutoUpgradeCompareFailed (result compare_failed/unresolved, 48h)#

The guard cannot resolve the release ref or cannot compare against the mirror — the host has silently stopped converging (fail-closed). First three commands:

ssh <host> journalctl -u nixos-upgrade -e        # the guard says exactly why it skipped
ssh <host> curl -fsS 'http://voile:8418/api/v1/repos/tsunaminoai/nix-flake-final/compare/<current_rev>...<release_rev>'
curl -fsS http://voile:8418/api/v1/version        # is the Forgejo mirror itself up / mirroring?

Usual causes: voile (or its Forgejo container) down, the mirror sync stalled so the host’s rev is unknown to it (404 → empty compare), or the host is running a dirty working-tree rev from a deploy-rs push (expected dirty skip, not a fault — it clears on the next release that supersedes the deploy).

AutoUpgradeStuckBehindRelease (behind > 0, 72h)#

The guard can compare and the host is provably behind, yet never converges — the nightly switch itself is failing. Most likely the release tip does not build on this host, or the run keeps getting skipped. First three commands:

ssh <host> journalctl -u nixos-upgrade -e         # find the failing build/switch
ssh <host> systemctl list-timers nixos-upgrade    # is the timer even firing?
nix log <failing-drv>                             # root-cause the build failure

om ci triage#

  • om ci is the only gate that builds host closures. Never bypass a red om ci by pushing flake.lock to master — and since the provenance gate, doing so no longer moves release anyway.
  • “Cannot build X” lines are scheduler cascade, not causes. Of the .drv paths named on error lines, only those nix has a build log for actually ran and failed. Root-cause with nix log <drv> (works even for remote-builder failures) or rebuild locally with --builders "" --max-jobs 4.
  • The PR failure comment now does this extraction itself: it posts up to three root failures (drvs with build logs, last 30 lines each), skipping the cascade noise, plus the last 80 lines of CI output in a collapsible block. The 2026-08-15 comment predating this was a wall of cascade that led triage to disable a bystander VM check instead of the one real error behind it.

Manual interventions#

Re-run stage 1#

On ereshkigal: trigger-flake-auto-update (starts the systemd unit with its proper environment). Running nix run .#update by hand as root breaks on “No writable cache directories” and other env differences; from your own user in a clean checkout it works, and nix run .#update -- --dry-run exercises the whole pipeline without pushing.

Advance the release deliberately#

gh workflow run release.yml — the workflow_dispatch path skips the merged-PR provenance check. This is a deliberate human action; make sure the commit actually builds fleet-wide first.

Retract a bad release#

There is no automatic way back. release is a plain branch the bot force-pushes, so move it yourself:

git push origin +<good-tag>:release      # e.g. +v2026.08.16:release

Then understand what does not happen: hosts that already converged onto the bad commit now compare as ahead (or diverged) of the retracted pointer and the fail-closed guard skips — they do not roll themselves back, and auto_upgrade_guard{result="ahead"} is what you will see. Roll each affected host back by hand:

  • nix run .#deploy-rollback <host> (previous generation), or
  • push the good tag with deploy-rs from a checkout at that tag: nix run .#deploy -- "git+file://$PWD?rev=<good-sha>#<host>" -s --confirm-timeout 120.

Manual fallback on the host: nix-env -p /nix/var/nix/profiles/system --rollback && /nix/var/nix/profiles/system/bin/switch-to-configuration switch (Deployment → Rollback Issues). Finally fix master, let a merged PR advance release past the bad commit.

Deploy a host now (push path)#

deploy is NOT on PATH. The one invocation, the confirm-timeout rule (the node default is 60 s and rolled back a large ereshkigal switch mid-activation), the git+file://…?rev= form for a dirty tree, magic rollback, and mokou’s SSH-to-self refusal are all in one place: Deployment → the battle-tested invocation. Short form: nix run .#deploy -- .#<host> -s --confirm-timeout 120.

Convergence-specific addendum:

  • A working-tree deploy leaves the host on a -dirty rev, which the stage-3 guard then skips by design until a release supersedes it.
  • myon and the aarch64 Pis never build on-box (remoteBuild = false via lowMemoryHosts in modules/flake/deploy/default.nix) — closures are built locally and pushed.

Gotchas#

  • The libvirt-from-unstable overlay trap (broke the 2026-08 bump): never pull a library with same-version Python bindings (libvirt being the proven case) from unstable alone — the stable bindings’ generator hard-fails against the newer lib’s API XML. Bump both sides together or neither.
  • The stage-1 github/automation-token PAT (fine-grained: Contents RW, Pull requests RW, Metadata R) is used ONLY by the scheduler on ereshkigal — the stage-3 guard needs no GitHub at all anymore. After any rotation, verify before deploying: curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $(sops --decrypt --extract '["github"]["automation-token"]' secrets.yaml)" https://api.github.com/repos/tsunaminoai/nix-flake-final must print 200 — the 2026-08 rotation shipped a dead token silently.
  • Piping build output (| tee, | tail) eats exit codes without pipefail — gate on the command, not the pipe.
  • nix run .#update refuses dirty trees and (live runs) non-master branches by design; private git+ssh inputs (corp/praxis) are left pinned because the automation host has no SSH creds for them.
  • Stale force-with-lease after a merge (2026-09-01). GitHub deletes automation/flake-update when its PR merges, but the scheduler’s long-lived clone only ever fetched master, so its remote-tracking ref for the branch went stale and every later git push --force-with-lease was rejected with [rejected] (stale info). Fixed in 7f071e88 (modules/flake/update/default.nix): a git fetch --prune origin "+refs/heads/$BRANCH*:refs/remotes/origin/$BRANCH*" right before the push. The refspec must be a glob — an exact refspec for a deleted branch dies with couldn't find remote ref before pruning, while a glob with zero matches prunes and exits 0. journald rate-limits the unit’s output (“Suppressed N messages”), so check /var/log/flake-auto-update.log or gh pr list for ground truth, not just the journal.
  • --override-input target under omnix is flake/corp, not corp. omnix nests the flake under test as input flake; a bare --override-input corp … is silently ignored (“does not match any input”). For a direct nix flake check / nix build the target is plain corp.
  • om ci unit timeout. flake-auto-update.service has a 12 h TimeoutStartSec (00500e1a; was 6 h, which killed a healthy run cross-building haxo’s uncached armv6l toolchain). Completed derivations stay in the store, so a restarted run resumes where it died.
  • Deployment — the full deploy system this pipeline is the “eventually” half of (deploy-rs, deploy-* apps, the release workflow).
  • Telemetry — where the guard metrics land and the full alert-rule catalogue.
  • Alerts & Notifications — how the two AutoUpgrade warnings reach you (ntfy; warnings never hit Discord).
  • Gitea — the voile Forgejo mirror that stage 3 both fetches from and compares against.