Restoring a Host from Backup#

Quick Reference#

Restoring Files — NixOS (borgmatic)#

After a full wipe and rebuild (see sops bootstrap below), borgmatic is fully configured by activation. The borgmatic config is at /etc/borgmatic.d/voile.yaml with the repo URL, SSH key, and passphrase already wired in.

List borgmatic archives#

sudo borgmatic list

Output looks like:

voile: Listing archives
mokou-2025-01-15T03:00:01.123456  Wed, 2025-01-15 03:00:01 [...]
mokou-2025-01-14T03:00:01.654321  Tue, 2025-01-14 03:00:01 [...]

Extract specific files (borgmatic)#

borgmatic extract always runs in the current working directory and reconstructs the path hierarchy from root. Run from /tmp to avoid collisions:

cd /tmp
sudo borgmatic extract --archive mokou-2025-01-15T03:00:01.123456 \
  --path home/tsunami/.config/sops/age/keys.txt

The file lands at /tmp/home/tsunami/.config/sops/age/keys.txt.

Extract the user sops age key — NixOS#

cd /tmp
sudo borgmatic extract --archive <latest-archive> \
  --path home/tsunami/.config/sops/age/keys.txt

mkdir -p ~/.config/sops/age
sudo mv /tmp/home/tsunami/.config/sops/age/keys.txt ~/.config/sops/age/keys.txt
chmod 600 ~/.config/sops/age/keys.txt

Then re-run activation to decrypt user-level secrets:

./scripts/system-flake-rebuild.sh <hostname>   # runs `nh os switch .#`

Full home restore#

To restore all of /home:

cd /
sudo borgmatic extract --archive <archive-name> --path home

This overwrites in place. Use --dry-run first to preview.

Borgmatic sources#

The borg module’s base source_directories back up:

• /etc
• /home
• /var/log
• /var/backup

On hosts that enable them, service modules append further source directories to the same configurations.voile:

• jellyfin → /var/lib/jellyfin/config, /var/lib/jellyfin/data, /var/lib/jellyfin/cache
• servarr → /var/lib/servarr/
• doc-pipeline → /var/backup/paperless

Restoring Files — Darwin (borg direct)#

Darwin uses raw borg via a launchd-managed shell script. There is no borgmatic. run_backup.sh reads the SSH key from /etc/borgmatic/key and the passphrase from /etc/borgmatic/pass — the same paths as NixOS — but unlike NixOS these files are not provisioned by sops activation. The Darwin sops module declares no sops.secrets, so after a Darwin rebuild you must place /etc/borgmatic/key and /etc/borgmatic/pass manually.

Set up environment#

The repo URL is in the backup script. Pull it from there:

grep 'BORG_REPO=' /etc/borgmatic/run_backup.sh
# → export BORG_REPO="ssh://borgwarehouse@voile.armadillo-banfish.ts.net:22222/./<repo-id>"

export BORG_REPO="ssh://borgwarehouse@voile.armadillo-banfish.ts.net:22222/./<repo-id>"
export BORG_RSH="ssh -i /etc/borgmatic/key"
export BORG_PASSPHRASE="$(cat /etc/borgmatic/pass)"

List borg archives#

borg list

Archive names look like: work-laptop-2025-01-15T03:00:00Z

Extract specific files (borg)#

borg extract also runs relative to the current directory. From /tmp:

cd /tmp
borg extract ::work-laptop-2025-01-15T03:00:00Z \
  Users/bcraton/.config/sops/age/keys.txt

File lands at /tmp/Users/bcraton/.config/sops/age/keys.txt.

Extract the user sops age key — Darwin#

cd /tmp
borg extract ::<hostname>-<timestamp> Users/<user>/.config/sops/age/keys.txt

mkdir -p /Users/<user>/.config/sops/age
mv /tmp/Users/<user>/.config/sops/age/keys.txt /Users/<user>/.config/sops/age/keys.txt
chmod 600 /Users/<user>/.config/sops/age/keys.txt

./scripts/darwin-flake-rebuild.sh <hostname>   # runs `darwin-rebuild --impure --flake .#<hostname> switch`

Darwin backup sources#

The Darwin backup script backs up / with extensive excludes for Nix store, caches, Library folders, derived data, etc. See modules/darwin/borg/default.nix for the full exclude list.

Sops Bootstrap After a Wipe#

Before borg is accessible, the system must activate successfully — which requires secrets.yaml to be encrypted for the new SSH host key. This is the bootstrap procedure that unblocks everything above.

Key architecture#

After a wipe, the SSH host key changes, breaking system-level decryption. You must re-encrypt secrets.yaml for the new key before switching.

Phase 1 — From an authorized host (e.g. mokou)#

1. Get the new host age key#

On the wiped machine, once nix is available:

nix shell nixpkgs#ssh-to-age --command \
  sh -c 'cat /etc/ssh/ssh_host_ed25519_key.pub | ssh-to-age'

2. Update .sops.yaml#

keys:
  - &hosts:
    # ...
    - &myhostname age1<new-key-from-step-1>

3. Re-encrypt secrets.yaml#

sops updatekeys secrets.yaml   # review the key diff, confirm with y
git add .sops.yaml secrets.yaml
git commit -m "Update <hostname> sops key after machine wipe"
git push

Phase 2 — On the wiped machine#

git pull
./scripts/system-flake-rebuild.sh <hostname>   # NixOS  (`nh os switch .#`)
# ./scripts/darwin-flake-rebuild.sh <hostname> # Darwin (`darwin-rebuild --impure --flake .#<hostname> switch`)

On NixOS, sops-nix activates using the SSH host key and places all system secrets, including the borg credentials at /etc/borgmatic/key and /etc/borgmatic/pass. On Darwin no borg sops secrets are declared, so those two files must be placed manually (see the Restoring Files — Darwin (borg direct) section above).

Borg is now accessible — proceed with file extraction above.

Generating a new user sops key (if old key unrecoverable)#

On the wiped machine:

mkdir -p ~/.config/sops/age
age-keygen -o ~/.config/sops/age/keys.txt
# Note the public key: age1...

On an authorized host, update .sops.yaml:

keys:
  - &users:
    - &username-hostname age1<new-public-key>

sops updatekeys secrets.yaml
git add .sops.yaml secrets.yaml
git commit -m "Rotate <user>-<hostname> user sops key"
git push

Back on the wiped machine:

git pull
./scripts/system-flake-rebuild.sh <hostname>   # runs `nh os switch .#`

Troubleshooting#

sops activation fails after switch#

git log --oneline -3                    # confirm updatekeys commit is present
sops --decrypt secrets.yaml | head -5   # test decryption directly

Borg credentials missing after activation#

ls -la /etc/borgmatic/key /etc/borgmatic/pass

If missing, sops activation silently failed. Check that tsunaminoai.borg.enable is set and that borgmatic/<hostname>/key and borgmatic/<hostname>/passphrase exist in secrets.yaml.

borgmatic: no archives found#

The host may not have backed up yet (new repo) or the repo wasn’t initialized. borgmatic runs as a systemd service and logs to the journal (no log file is configured), so check the init-borg-repo unit and the borgmatic journal:

systemctl status init-borg-repo
journalctl -u borgmatic

borg: Permission denied on key#

The SSH key at /etc/borgmatic/key is 0400 root:root. Run borg as root, or copy the key to a temp path with user-readable permissions for one-off access:

sudo cp /etc/borgmatic/key /tmp/borg-key && chmod 400 /tmp/borg-key
export BORG_RSH="ssh -i /tmp/borg-key"