Successor Handbook#

This page is for a technical person who has to run or wind down this fleet without Ben. The family-facing version is If Something Happens to Ben. Everything here is in the repository; nothing depends on a chat history or a private note.

Read in this order:

1. This page.
2. Accounts & Credentials — what you need access to and what breaks when each lapses.
3. In-flight work — what is half-done or deliberately parked.
4. SDLC & Policies — how changes are proven and shipped.
5. The machine and service pages for whatever you touch first.

Fleet at a glance#

Source of truth: hosts/default.nix (the easy-hosts registry). Everything not in that file is outside the flake and listed in the second table.

Outside the flake (change them in their own consoles, never in .nix):

You cannot do anything without#

1. An age key that is a recipient of secrets.yaml. .sops.yaml lists the recipients. Every host build decrypts secrets at activation, and just preflight fails at its first check without one. A new person becomes a recipient only when someone who already holds a recipient key adds the new public key to .sops.yaml and runs just sops-updatekeys (Adding a collaborator user). Recipients today are Ben’s personal keys plus the host keys, so the bootstrap paths are: a host you can log into (its /etc/ssh/ssh_host_ed25519_key is an age identity via ssh-to-age), or an offline copy of Ben’s key. TODO(owner): where is the offline copy of a recipient age key (printed, USB, vault item), and who may retrieve it?
2. pki/db is narrower still. The CA database decrypts only to two of Ben’s personal keys (.sops.yaml rule ^pki/db). Holding a secrets.yaml key does not help. See CA escrow and break-glass.
3. Bitwarden. Wi-Fi passwords, the XCA database passphrase (FalseBlue Root CA Key), and the LUKS recovery passphrase for mokou’s /pool live in Ben’s personal vault. TODO(owner): how does a successor reach the vault (emergency-access contact, recovery-key location)?
4. Tailscale admin and Tailnet Lock. Every host is reached over the armadillo-banfish.ts.net tailnet. Tailnet Lock is enabled: a new node joins, shows Online, and has zero peer connectivity until tailscale lock sign <nodekey> <tlpub> is run from a node holding a trusted signing key (mokou, with sudo). TODO(owner): if mokou is gone, which other node or offline key can sign?
5. The YubiKeys. mokou’s root disk unlocks with a YubiKey (FIDO2), SSH to every host uses the Nano 5C’s OpenPGP auth key, and the CA keys are on both keys. What each key gates and what still works without it: YubiKey recovery.
6. GitHub. The repository is private under tsunaminoai. CI needs the sc2ben machine-account key for the private corp input; ereshkigal’s scheduler needs github/automation-token. Details in Accounts.

Build and deploy safely#

Never deploy what you did not build. The local gate is one command:

just gate <host>   # format check → host toplevel → telemetry/gatus checks (when the host has them) → docs

It does not replace om ci on ereshkigal, which is the only pipeline that builds every host closure — GitHub’s nix flake check builds no host toplevels (om ci triage).

Deploy one host, from a clean tree, with a long confirm timeout:

nix run .#deploy -- .#<host> -s --confirm-timeout 120

The traps (working-tree deploys, -dirty revisions, mokou refusing SSH-to-self) are in the battle-tested invocation. Roll a host back with nix run .#deploy-rollback <host>.

Facts that bite first-time operators:

• Dirty-tree builds see git-tracked files only: git add new files before any nix build, or the build silently uses the old tree.
• docs/ is a symlink to pkgs/docs/content/; edit and git add the real path.
• A sops secret must exist in secrets.yaml before you build a host that declares it (sops-nix validates at eval time). Add a placeholder first.
• Evaluating any host fetches the private git+ssh://…/sc2in/corp input (every flake input is in every host’s nix.registry). Without the sc2ben SSH key, stub it: --override-input corp path:./ci/stub-corp (flake/corp under om ci, because omnix nests the target flake as input flake).
• Do not read the four >1000-line files whole (modules/nixos/telemetry/server.nix, modules/nixos/media/weather-channel.nix, modules/nixos/containers/doc-pipeline/default.nix, hosts/x86_64-nixos/ereshkigal/default.nix); grep for a # ── … ── section marker and read that section.

How the fleet updates itself (five lines)#

1. ereshkigal, 1st and 15th at 03:00: nix run .#update bumps the public inputs, runs a full om ci, opens PR automation/flake-update, and auto-merges on green.
2. release.yml tags CalVer and fast-forwards release — only for commits that landed via a merged PR.
3. Each NixOS host’s nixos-upgrade (04:30) fetches release from the voile Forgejo mirror and switches only when provably behind (fail-closed guard).
4. Guard verdicts are metrics; AutoUpgradeCompareFailed / AutoUpgradeStuckBehindRelease page after 48 h / 72 h.
5. Everything else is a deliberate deploy-rs push.

Triage, retracting a bad release, and the gotchas: Fleet convergence runbook.

The builder concentration risk#

All remote builders (ereshkigal, mokou, razer, shinobu, work-laptop-m4 — the builder tag in hosts/default.nix), the binary cache (ereshkigal :11111), and the only closure-building CI (om ci on ereshkigal) are Ben’s own machines. If ereshkigal is off, nothing builds the fleet automatically and the cache is gone. The emergency guide tells the family they may power it down; that is true for the household and false for anyone maintaining the other machines.

Degraded mode, when builders or the cache are unreachable:

• Build locally only: append --builders "" --max-jobs 4 to nix build.
• Stop offloading permanently: remove the builder tag from dead hosts in hosts/default.nix (tag discovery generates nix.buildMachines), then rebuild.
• Shrink the scheduled CI: tsunaminoai.ci.autoUpdate.ciCommand = "om ci run '.#default'" (Automated flake updates).
• Expect long builds. Without the cache a host toplevel takes hours, and the octopi kernel only stays cheap because nixos-raspberrypi.cachix.org serves it: before flashing, confirm nix build --dry-run lists nothing containing linux-rpi under “will be built”.

When a host will not boot#

A hub — each link is the detailed procedure.

Winding down instead#

If the decision is to stop running this fleet, the family guide’s wind-down table is the order of operations. Add these technical steps:

• Cancel myon at BuyVM and remove its node in the Tailscale admin console.
• Export Paperless (paperless-export/ on voile already holds archival copies) and the Jellyfin library before powering ereshkigal off.
• Run the decommissioning runbook per host so disks are cryptographically erased and the log has a row.

Related#

• Accounts & Credentials
• In-flight work
• If Something Happens to Ben
• Maintenance · Deployment · Adding a Host