Maintenance#

Building#

System Configurations#

Each host is exposed as a NixOS configuration and is built via its system.build.toplevel attribute. For example, to build the octopi host, you would run:

nom build .#nixosConfigurations.octopi.config.system.build.toplevel

Before you commit#

Run the gate for every host your change touches. It is one recipe:

just gate <host>

It runs, in order: the formatting check, the host toplevel, the host’s telemetry-rules-<host> and gatus-config-<host> checks when they exist, and the docs build. The same four commands by hand:

nix build --no-link .#nixosConfigurations.<host>.config.system.build.toplevel
nix build --no-link .#checks.x86_64-linux.telemetry-rules-<host>   # vmalert -dryRun
nix build --no-link .#checks.x86_64-linux.gatus-config-<host>      # Gatus YAML render
nix build --no-link .#docs                                          # strict mkdocs

Two rules that are not obvious:

• Dirty-tree builds see git-tracked files only — git add new files before gating, or use path:$PWD#….
• Piping build output (| tail, | tee) eats the exit code. Gate on the command, not the pipe.

To prove a comment-only or docs-only change did not alter a closure, run just drv-invariant (working tree vs master, or pass a ref). Do not compare raw toplevel.drvPath values across two trees: every host’s closure embeds the flake revision and, through nix.registry, the flake’s own source path, so any edit moves every drvPath. The recipe neutralises both before diffing.

GitHub’s nix flake check builds no host toplevels. The only pipeline that builds every closure is om ci on ereshkigal — see om ci triage.

likec4-site hash ritual#

pkgs/likec4-site/default.nix pins a fixed-output hash over bun install --frozen-lockfile. Every change to pkgs/likec4-site/bun.lock invalidates it and breaks both .#docs (the topology site) and .#worldsim-c4-site. Recompute: set the outputHash to lib.fakeHash, run nix build .#likec4-site 2>&1 | grep 'got:', paste the printed sha256-… back. The same file also swaps in bun’s baseline build because ereshkigal (no AVX2) SIGILLs on the default one.

Deploying#

deploy is not on PATH. deploy.nodes is generated automatically from every NixOS and Darwin configuration (except bootable-iso) once a host is registered in hosts/default.nix; deploy with the wrapped app and always raise the confirm timeout:

nix run .#deploy -- .#<host> -s --confirm-timeout 120

The traps (working-tree deploys, -dirty revisions, magic rollback, mokou refusing SSH-to-self) are in Deployment → the battle-tested invocation.

Manual Deployment#

If you are on the remote host, all you need to do is run:

git clone http://voile.armadillo-banfish.ts.net:8418/tsunaminoai/nix-flake-final.git
# or from GitHub:
# git clone https://github.com/tsunaminoai/nix-flake-final.git
cd nix-flake-final
nix develop
nh os switch .#

If on the deployment server:

nh os build -H <host> .# -o result
nix copy --to ssh://<user>@<host> ./result
ssh <user>@<host> 'sudo /nix/store/<hash>/bin/switch-to-configuration switch'

See also NixOS for more information on building and deploying NixOS configurations.

Nix storage#

Currently, this is a local binary store that is service from Ereshkigal and is used for all builds and deployments. It is a Nix binary cache that is available to all hosts.

It may be worth converting this to an S3 bucket on voile¹.

Development Shell#

Currently, the development shell is set up to use the nix develop command. This can be used to enter a development environment with all of the necessary dependencies and tools installed. If direnv is installed, it will automatically load the environment when you enter the directory.

Run just to see the available commands in the development shell.

Useful Commands provided by the Shell#

The shell uses just for all recipes. This is a curated highlights list, not an inventory: just --list (or bare just) is complete and grouped, and every recipe carries a doc comment.

• just preflight - validate the local environment: SOPS age key, secrets.yaml decryptable, an SSH key under modules/users/*/keys/, flakes enabled, git identity. Run this first on a new machine
• just new-host <name> [arch] [class] - scaffold hosts/<arch>-<class>/<name>/default.nix and print the hosts/default.nix entry (Adding a Host)
• just gate <host> - the pre-commit gate: format → host toplevel → telemetry/gatus checks → docs (see Before you commit)
• just docs-build - build this documentation site
• just docs-serve - build and serve this documentation site locally
• just nix-store-fix - repair the Nix store (verify + check contents)
• just clean-nix - clean the Nix store and remove old generations
• just fmt - format all code (Nix, etc.)
• just vulscan - run a vulnerability scan (use the nix develop .#security shell)
• just sbom <host> - generate a Software Bill of Materials for a host
• just borg-generate-secrets <host> - generate borg SSH key + passphrase in secrets.yaml
• just sops-update-hosts - re-encrypt secrets.yaml after updating .sops.yaml
• just get-host-key <host> - fetch and save a host’s SSH host key to modules/nixos/security/pubkeys/
• just plasma-settings-dump-nix - dump current KDE Plasma settings as a plasma-manager nix snippet

Troubleshooting the shell#

Four traps that make a fresh checkout feel broken. All are fixed in the tree; they are listed so the symptom is recognisable if one regresses.

• Tools vanish from the direnv-loaded shell (nh missing, prompt odd): an exec in the devShell shellHook. direnv’s use flake evaluates the hook in a non-interactive bash to capture the environment, and an exec truncates that capture. The hook gates exec fish on [[ $- == *i* ]] (modules/flake/programs/shell.nix); keep it that way.
• direnv.toml settings ignored (disable_stdin, the “…is taking a while, CTRL-C to give up” spam): keys must live under [global]; a bare top-level key is silently ignored. modules/home-manager/direnv.nix sets config.global.warn_timeout = "60s" because a cold use flake of this flake exceeds the 5 s default.
• Literal \[ \] in the bash prompt and shopt: progcomp: invalid shell option name: the devShell exposed stdenv’s minimal bash (no readline, no progcomp). pkgs.bashInteractive is in the devShell packages for this reason.
• Reproduce in the project directory with direnv loaded. ssh host 'bash -ic …' from $HOME uses the system bash and looks clean; two wrong diagnoses came from exactly that.

The macOS-only trap — users.users.<n>.shell silently not applying — is in Mac onboarding.

Other Useful Commands available in the Shell#

• nh os switch .# - switch the system configuration for the current host
• nh os build -H <host> .# - build the system configuration for the specified host
• nh home switch .# - switch the home-manager configuration for the current user
• nh home build -c user@host .# - build the home-manager configuration for the specified user and host
• nix run github:tiiuae/sbomnix#sbomnix ./result -- --buildtime - generate a Software Bill of Materials for the system closure in the result directory, including build time dependencies
• nix run github:tiiuae/sbomnix#nixgraph -- --depth=3 ./result - generate a graph of the closure dependencies using nixgraph
• nix run github:tiiuae/sbomnix#nix_outdated -- ./result - check for outdated packages in the system closure using nix_outdated
• nix run github:tiiuae/sbomnix#provenance (nixpkgs#hello or ./result) --recursive -out ./provenance.json - generate a provenance file for a package using sbomnix