ereshkigal storage reclaim, HAOS disk migration & health fixes#

Background (2026-07-20 health check)#

ereshkigal is an ex-Proxmox Dell PowerEdge (PERC H710). A health check surfaced several issues. The code-level fixes (paperless unit EnvironmentFile, servarr-backup set -e bug, the fan-controller tuning in §4, and the xfs support in §2) ship in the same PR as these docs; the LVM reclaim (§1), HAOS disk migration (§2), and cache-key repair (§3) are live-host steps that need this runbook.

Findings that drive this runbook:

btrfs root / (sdb2) is 90 % full, 1 MiB unallocated — an ENOSPC risk on its own. It hosts both write-heavy workloads: the HAOS qcow2 (/var/lib/libvirt, 31 G) and paperless Postgres (/var/lib/containers/storage/volumes/paperless_pgdata). They contend → bursty “critical iowait”. Smoking gun: 0.5 MB paperless-db checkpoints taking 13–28 s to fsync.
VG kur (sdc1, 10.9 T) and VG tempstorage (sda, 2.7 T) are almost entirely dead Proxmox leftovers. The only live LV is kur/nix-store (mounted /nix). Everything else is reclaimable (owner-confirmed 2026-07-20).
Loud fans are NOT load/thermal — CPUs sit at ~50 °C, iowait ~2 %. The existing dell-idrac-fan-controller kept flipping off its quiet 10 % profile because CPU 1 idles right at its 50 °C safety threshold, plus the third-party PCIe cooling response (P400 + the VFIO-passed NEC uPD720200 USB3 controller, see home-assistant-vm.md) was enabled. Fixed in-repo. See §4.
Every nix cache upload fails (Permission denied (publickey) as nix-ssh) because the sops secret nix/cache-key isn’t a valid SSH public key. See §3.
Root’s 199 G used is mostly garbage, not data (2026-07-20). du -x / only saw 164 G; the rest hid from -x in a subvolume. The real hogs: a 46 G stale /var/log/messages (rsyslog wrote it until 2026-01-25, then was disabled and the file was never removed) and 28 G of leaked /tmp/tmp.* servarr-backup staging dirs (see §0). Clearing both drops root 89 % → ~56 % — so the HAOS migration (§2) is NOT needed for space; it’s now only an iowait/CoW win.

Ordering: §0 (quick reclaim) first — it’s ~74 G back for rm and answers the root-full pressure on its own. Then §1 (Proxmox LVM), §2 (HAOS migration, now optional/iowait-only), §3 and §4 anytime.

§0 — Quick space reclaim (~74 G of garbage on root)#

The ~76 % figure an earlier draft put in §2.4 was wrong: it assumed the 31 G qcow2 was the swing factor. It isn’t — these two are.

46 G stale rsyslog log. rsyslog is disabled (modules/nixos/security/default.nix) but its old catch-all was orphaned. Confirm it’s stale, then delete:

ssh ereshkigal 'sudo stat -c "%y %s" /var/log/messages'   # expect an mtime months old
ssh ereshkigal 'sudo rm -f /var/log/messages'             # ~46 G back instantly

/var/log/messages confirmed stale and removed.

28 G leaked servarr-backup staging in /tmp. Each nightly run left its mktemp -d dir (prowlarr/…/*.db.backup) behind — the script’s cleanup was a RETURN trap (never fires at script scope) and it was also dying on the set -e bug. Both are fixed in this PR (backup-servarr-sqlite.sh now uses an EXIT trap). Delete the existing leaks (they’re old backup temp copies, safe):

# sanity: they should be root-owned dirs full of *.db.backup
ssh ereshkigal 'sudo find /tmp -maxdepth 1 -type d -name "tmp.*" -exec du -sh {} +'
ssh ereshkigal 'sudo find /tmp -maxdepth 1 -type d -name "tmp.*" -mtime +0 -exec rm -rf {} +'

Old /tmp/tmp.* staging dirs removed.
Optional hardening: set boot.tmp.cleanOnBoot = true (or a systemd.tmpfiles age rule) so /tmp — a btrfs subvol here, not tmpfs, so it never clears on its own — can’t silently accumulate again.
df -h / now ~56 %.

§1 — Reclaim the dead Proxmox LVM#

⚠️ lvremove is irreversible. The sequence below deactivates first (reversible), soaks, then removes. kur/nix-store is mounted at /nix — never touch it or the kur VG itself.

Owner disposition (2026-07-20): every VM listed below is either already archived to voile or explicitly disposable. LVs present on disk that were NOT in the owner’s inventory — kur/vm-150/151/152/153-disk-0, kur/vm-156-disk-0 — get an explicit read-only eyeball before removal (step 1.3).

1.1 Snapshot the current state (paste into the PR before destroying anything)#

ssh ereshkigal 'sudo lvs -o lv_name,vg_name,lv_size,lv_attr,lv_path --units g; \
  echo ---; sudo vgs; echo ---; lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT'

Output captured.
Confirm kur/nix-store shows attr ...ao---- (o = open); every LV to be removed shows ...------- (no o, i.e. not open/mounted).

1.2 Deactivate the dead LVs (reversible — `lvchange -ay` undoes it)#

tempstorage (VG on sda — 100 % dead, whole VG can go afterwards):

ssh ereshkigal 'sudo lvchange -an \
  tempstorage/vm-212-disk-0 \
  tempstorage/local-var-tmp-vzdump \
  tempstorage/pve-root-backup20250619'

kur (VG on sdc — dead LVs only; NOT nix-store):

ssh ereshkigal 'sudo lvchange -an \
  kur/nixos \
  kur/vm-100-disk-0 kur/vm-150-disk-0 kur/vm-151-disk-0 kur/vm-152-disk-0 \
  kur/vm-153-disk-0 kur/vm-156-disk-0 \
  kur/vm-9000-disk-0 kur/vm-9000-disk-1 kur/vm-9001-disk-0 kur/vm-9003-disk-0 \
  kur/vm-9004-disk-0 \
  kur/vm-125-cloudinit kur/vm-127-cloudinit kur/vm-144-cloudinit \
  kur/vm-147-cloudinit kur/vm-156-cloudinit kur/vm-230-cloudinit \
  kur/vm-9000-cloudinit kur/vm-9001-cloudinit kur/vm-9003-cloudinit \
  kur/vm-9004-cloudinit'

All deactivated with no error.
Soak ≥ 24 h. If anything on the box breaks, lvchange -ay <lv> to restore. (Nothing should — none were mounted.)

1.3 Eyeball the not-in-inventory LVs before removal#

# For each of 150/151/152/153/156: probe FS type, mount read-only, look, unmount.
ssh ereshkigal 'sudo lvchange -ay kur/vm-156-disk-0; \
  sudo blkid /dev/kur/vm-156-disk-0; \
  sudo mkdir -p /mnt/inspect && sudo mount -o ro /dev/kur/vm-156-disk-0 /mnt/inspect; \
  sudo ls -la /mnt/inspect; sudo umount /mnt/inspect; sudo lvchange -an kur/vm-156-disk-0'

150 / 151 / 152 / 153 / 156 inspected; nothing worth keeping (or archived to voile first).

1.4 Remove (irreversible)#

ssh ereshkigal 'sudo lvremove -y \
  tempstorage/vm-212-disk-0 tempstorage/local-var-tmp-vzdump \
  tempstorage/pve-root-backup20250619'
ssh ereshkigal 'sudo lvremove -y \
  kur/nixos kur/vm-100-disk-0 kur/vm-150-disk-0 kur/vm-151-disk-0 \
  kur/vm-152-disk-0 kur/vm-153-disk-0 kur/vm-156-disk-0 \
  kur/vm-9000-disk-0 kur/vm-9000-disk-1 kur/vm-9001-disk-0 kur/vm-9003-disk-0 \
  kur/vm-9004-disk-0 \
  kur/vm-125-cloudinit kur/vm-127-cloudinit kur/vm-144-cloudinit \
  kur/vm-147-cloudinit kur/vm-156-cloudinit kur/vm-230-cloudinit \
  kur/vm-9000-cloudinit kur/vm-9001-cloudinit kur/vm-9003-cloudinit \
  kur/vm-9004-cloudinit'

sudo lvs now shows only kur/nix-store (+ the new HAOS LV from §2).

1.5 The idle USB disk (sdd) — old Proxmox backup target#

sdd (931 G Toshiba USB, ext4) was the Proxmox backup target; unmounted and unused. Either unplug it, or wipe + repurpose as local scratch:

# Only after confirming nothing on it is needed:
ssh ereshkigal 'sudo wipefs -a /dev/sdd1'   # or leave it and physically remove

sdd dealt with (wiped).

§2 — Move the HAOS qcow2 off the btrfs root onto `kur`#

This is an iowait win, not a space win (do §0 for space): it takes the VM’s write-heavy CoW churn off the btrfs root — where it fights paperless Postgres and fragments against btrfs CoW — and onto the empty 10.9 T kur VG. Moving the 31 G qcow2 frees ~31 G of root, but after §0 the root is already ~56 %, so this is about latency, not capacity.

Approach: a dedicated non-CoW filesystem LV mounted at /var/lib/libvirt/images. Keeps tsunaminoai.haosVm.imagePath at its default; the module doesn’t change. (xfs chosen over btrfs to avoid CoW fragmentation under a growing qcow2; ext4 is equally fine.)

A raw-LV block device passed straight to the domain performs better still, but needs a haos-vm.nix change (block device instead of a qcow2 file). Deferred — the file-on-xfs approach removes the contention with far less risk.

2.0 Prerequisite: XFS support (deploy first)#

ereshkigal only used btrfs/vfat, so xfs support is declared in hosts/x86_64-nixos/ereshkigal/default.nix (boot.supportedFilesystems.xfs = true;) — this ships the xfs kernel module and puts mkfs.xfs/xfs_repair on PATH. Deploy this before 2.1, or mkfs.xfs won’t exist:

nixos-rebuild switch --flake .#ereshkigal --target-host ereshkigal --use-remote-sudo
ssh ereshkigal 'command -v mkfs.xfs'   # must resolve

xfs support deployed; mkfs.xfs resolves.

2.1 Create + format the LV (host)#

ssh ereshkigal 'sudo lvcreate -L 80G -n haos-images kur && \
  sudo mkfs.xfs -L haos-images /dev/kur/haos-images && \
  sudo blkid /dev/kur/haos-images'   # record the UUID for step 2.3

LV created, formatted, UUID recorded: 434354c2-7991-4e02-afcc-6c17d25bd2fd.

2.2 Stop the VM and stage the existing image#

ssh ereshkigal 'sudo virsh shutdown haos'          # graceful; wait for it
ssh ereshkigal 'sudo virsh domstate haos'          # -> "shut off"
ssh ereshkigal 'sudo mv /var/lib/libvirt/images /var/lib/libvirt/images.old'
ssh ereshkigal 'sudo mkdir -p /var/lib/libvirt/images && \
  sudo mount /dev/kur/haos-images /var/lib/libvirt/images && \
  sudo cp -a --sparse=always /var/lib/libvirt/images.old/. /var/lib/libvirt/images/ && \
  sudo ls -lh /var/lib/libvirt/images'

haos.qcow2 present on the new mount, size matches.

2.3 Make the mount declarative#

Add to hosts/x86_64-nixos/ereshkigal/hardware-configuration.nix (use the UUID from 2.1 — do not commit this until the LV exists, or the next host will fail to boot on a missing device):

fileSystems."/var/lib/libvirt/images" = {
  device = "/dev/disk/by-uuid/434354c2-7991-4e02-afcc-6c17d25bd2fd";
  fsType = "xfs";
  options = ["noatime" "nofail"];
};

Also disable CoW belt-and-suspenders isn’t needed on xfs. Then deploy:

# from the repo, targeting ereshkigal's usual deploy path
nixos-rebuild switch --flake .#ereshkigal --target-host ereshkigal --use-remote-sudo

Mount is declarative and survives systemctl daemon-reload + mount -a.

2.4 Start the VM, verify, clean up#

ssh ereshkigal 'sudo virsh start haos && sleep 5 && sudo virsh domstate haos'
# HA reachable at http://192.168.0.55:8123 (see home-assistant-vm.md)

HAOS boots, HA UI reachable, radios present.
Root freed by a further ~31 G (on top of §0’s ~74 G).
After a day of confidence: sudo rm -rf /var/lib/libvirt/images.old.

2.5 (Optional, later) relocate paperless Postgres too#

Even with HAOS moved, paperless pgdata still lives on the btrfs root under /var/lib/containers. If checkpoint latency is still poor, relocate the podman volume (or all of /var/lib/containers) onto a kur LV the same way. Lower priority — removing the qcow2 contention should already fix the fsync stalls.

§3 — Fix the failing nix cache uploads (`nix-ssh` publickey)#

Symptom: nix-post-build-hook-queue signs paths fine, then every upload to ssh://nix-ssh@ereshkigal… fails Permission denied (publickey) (thousands of log lines + failed session-c*.scope units for user nix-ssh).

Root cause: modules/flake/nix/cache.nix authorizes nix-ssh via AuthorizedKeysFile … <sops nix/cache-key>. On the box sshd -T -C user=nix-ssh resolves that path, but:

/run/secrets/nix/cache-key is not a public key file.

i.e. the sops secret nix/cache-key does not contain a valid SSH public key (it’s holding something else — likely a Nix binary-cache signing key in name:base64 form). The hook authenticates with the SSH key whose public half is:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICtfSQTzSDy9N+bErgbp7j/XCXrEZFncMLSIXwuK+EHi tsunami@mokou

…which is the public half of nix/builder/priv-key (modules/flake/nix/builder/nixos.nix, sshPrivateKeyPath).

Fix (sops edit — needs the age/GPG key locally): set nix/cache-key to that exact SSH public key line.

# derive the pubkey authoritatively from the private key if unsure:
#   sudo -n sh -c 'K=$(mktemp); cat /run/secrets/nix/builder/priv-key >"$K"; \
#     chmod 600 "$K"; ssh-keygen -y -f "$K"; rm -f "$K"'   # (run on ereshkigal)
sops secrets/<the-file-with-nix-cache-key>.yaml   # replace nix/cache-key value

Then redeploy ereshkigal and verify:

ssh ereshkigal 'sudo ssh-keygen -lf /run/secrets/nix/cache-key'   # must print a fp, not "not a public key file"
# It must MATCH the hook's key: ssh-ed25519 …tsunami@mokou
#   SHA256:0wztmtN4ARyrIB2iHzUlxpS49bpuxEAuausd77WBod4

Now force one upload. Run the whole test as root (sudo bash) and copy the key to a root-owned 0600 temp first — otherwise ssh refuses the key with “bad permissions … will be ignored” (the credential is 0440 owned by the service user; ssh won’t use a group-readable key you don’t own) and you get a false Permission denied that looks like the bug isn’t fixed:

ssh ereshkigal 'sudo bash -s' <<"SH"
T=$(mktemp /root/.hooktest.XXXXXX); chmod 600 "$T"
cat /run/credentials/nix-post-build-hook-queue.service/sshPrivateKeyPath > "$T"
P=$(ls -d /nix/store/*-check-sshd-config | head -1)
env NIX_SSHOPTS="-i $T -o BatchMode=yes" \
  nix copy --to ssh://nix-ssh@ereshkigal.armadillo-banfish.ts.net "$P"; echo "exit=$?"
rm -f "$T"
SH

exit=0 with no Permission denied = fixed (the real service authenticates fine because it runs as the key’s owner, so it never trips the perms check). “copying 0 paths” is expected — the path is already on the target (same host).

nix/cache-key holds the …tsunami@mokou SSH pubkey (fp matches above).
Root-owned-key nix copy returns exit=0, no Permission denied (verified 2026-07-20). The Child exited with 1 spam stops as the queue next runs.

If uploads still fail after the key is correct, check the egress path — all ereshkigal egress routes via the myon exit node, and the upload target is its own Tailscale FQDN.

§4 — Quiet the fans (fixed declaratively)#

Not load/thermal — CPUs idle ~50 °C, iowait ~2 %. ereshkigal already runs services.dell-idrac-fan-controller (in-band IPMI works: /dev/ipmi0 present, ipmi_si/ipmi_devintf loaded by the module) forcing a quiet 10 % static fan profile. The noise had two causes, both visible in its log:

… User static fan control profile (10%)  … Third-party PCIe card … response: Enabled
… Dell default dynamic fan control profile … CPU 1 temperature is too high, … applied for safety
… User static fan control profile (10%)  … CPU temperature decreased and is now OK (<= 50°C) …

50 °C safety threshold oscillation. CPU 1 idles at 49–51 °C, straddling the controller’s default cpuTemperatureThreshold = 50, so it kept flipping to Dell’s loud dynamic profile “for safety” and back — the intermittent noise.
Third-party PCIe cooling response Enabled. The P400 + VFIO-passed NEC USB3 controller have no temp sensor → iDRAC bumps the fan floor.

Fix (in this PR, hosts/x86_64-nixos/ereshkigal/default.nix):

services.dell-idrac-fan-controller = {
  cpuTemperatureThreshold = 65;              # was default 50 → above idle
  disableThirdPartyPcieCoolingResponse = true;
};

Deploy, then confirm the log stays on the 10 % profile and stops flipping to “Dell default dynamic … for safety”:

ssh ereshkigal 'sudo journalctl -u dell-idrac-fan-controller -n 20 --no-pager'

Deployed; log holds “User static fan control profile (10%)”, no more safety fallbacks; audible fan drop.

§5 — Landmines from the qBittorrent CIFS→LV migration (2026-08-21)#

Condensed to the parts with lasting value. All of these were actually stepped on.

Torrent I/O on a CIFS mount deadlocks the kernel. Random piece writes plus mmap’d reads for hashing, and when the SMB session reconnects, cifs_reopen_file’s writeback flush blocks on a folio held by an in-flight cifs_strict_writev. The threads go to unkillable D state, the thread-group leader becomes an unreapable zombie, and SIGKILL is inert — only a reboot clears it. qBittorrent’s profile and downloads now live on the kur/qbittorrent LV via tsunaminoai.servarr.qbittorrentStateDir. Note a torrent’s save path lives in its .fastresume, not the config: changing Session\DefaultSavePath only affects new torrents, and existing ones need Set Location in the WebUI. Reads alone are survivable — the deadlock needs a CIFS writer racing the reconnect.

Never put SQLite on CIFS. Lidarr’s database was malformed for 2+ weeks (SMB byte-range locking doesn’t survive the round trip) and every nightly backup copied it faithfully, so all 15 in retention were unusable. Repair is sqlite3 <db> ".recover" — but that fixes page-level damage while preserving bad content, so check JSON columns for raw control bytes afterwards (IndexerStatus.LastRssSyncReleaseInfo was the one breaking AlbumSearch). Now guarded: the backup script runs PRAGMA integrity_check on each snapshot and ServarrBackupDbMalformed fires on servarr_backup_db_integrity == 0. .backup succeeds happily on a corrupt database — that is why it went unnoticed.

systemctl mask cannot hold a flake-defined unit. NixOS owns /etc/systemd/system/<unit>.service, and /etc outranks /run in systemd’s unit load path, so plain mask refuses and mask --runtime reports success while being completely inert. Use systemd.services.<name>.enable = false. Verify with LoadState/UnitFileState — not FragmentPath, which reports the unit’s path either way. And never pass --now to a unit wedged in D-state: the stop job can never complete, so the command blocks forever.

kurVolumes provisioners need DefaultDependencies = false. Otherwise each inherits After=sysinit.target while needing to run before local-fs.target, forming a cycle; systemd breaks it by deleting an arbitrary job, and with nofail on the mounts all five LVs can silently land on the btrfs root with a completely clean-looking boot. Guarded by kur-volumes-verify.service (+15 min timer) and KurVolumeNotOnLV.

rsync a torrent tree with --sparse. Preallocated partials are sparse by construction — 168 G of blocks against 715 G logical here — so without -S you materialise every hole as real zeros. Also set Session\Preallocation=false so local usage tracks actual downloaded data.

borgmatic’s hooks restore only what they stopped. They used to start a fixed list unconditionally, so any deliberate downtime was undone by the next nightly run — that is what restarted qBittorrent into its third kernel wedge.