DECIDED: unify fleet logging + metrics into one spot#

Status: DECIDED (phase 1) — research spike completed 2026-07. The open questions from the draft have been researched and answered. The stack is VictoriaMetrics + VictoriaLogs + Grafana + vmalert/Alertmanager, with Grafana Alloy as the fleet agent — all Apache-2.0. Phase 1 (ereshkigal-only) is specified below and scaffolded as tsunaminoai.telemetry. Rejected options are recorded in the appendix.

Why (motivation)#

The 2026-07-15 HA Green outage ran ~10 h before anyone noticed — the classic failure of a homelab with no alerting. Today the observability signal is scattered and mostly “is it up?”, never “what’s wrong / is it about to break”:

• Per-host journald, never shipped anywhere. Correlating an incident means SSHing to each box.
• Gatus (tsunaminoai.monitoring) — a declarative endpoint-contribution module exists, but it is dormant: nothing enables it and nothing contributes endpoints, so even synthetic up/down checks aren’t live. Homer links to https://status.falseblue.com, which nothing serves.
• Homer — a dashboard of links, no live data.
• A single orphaned metric: servarr-backup writes /etc/servarr-backup-metrics in node_exporter textfile format — and it is doubly broken: the content is a static environment.etc string (the $(stat …) command substitutions are never executed), and nothing scrapes it anyway.
• No time-series, no log aggregation, no hardware health (SMART/IPMI/temps/fans — the very things behind this month’s fan + dead-K80 + PERC issues), no alerting.

Goal: one place to query metrics + logs + health for every host, with alerts that actually reach me when something breaks — including when home is down.

Second customer: praxis (control validation + incident determination)#

This stack has a second job beyond the homelab: it is the prototype for a praxis (sc2in/praxis) supporting toolset for validating controls and determining incidents. That constraint shaped the design — keep it light, flexible, available, and secure:

• Every praxis control (and the in-tree modules/compliance PoC it grew from) already carries monitors[] ({name, type, interval, conditions[]}) and alerts[] ({type, severity, description, remediation}) fields in its schema. Nothing consumes them today. The telemetry stack is designed as that consumer: monitors[] → vmalert rule groups; alerts[] → Alertmanager routing labels + annotations (remediation text arrives with the page).
• praxis’s MON family already enforces the collection substrate on managed hosts (MON-02 persistent journald, MON-04 storage capacity, MON-05.1 alert-on-logging-failure, MON-07/07.1 NTS time sync, MON-10 retention). This stack is the centralized aggregation + evaluation layer on top of it.
• All 44 IRO (incident response) controls in praxis are currently disposition organizational (policy-doc only). A pipeline that evaluates monitors[].conditions continuously and records alert history as timestamped evidence is the path to promoting IRO controls toward verified — automated incident determination instead of attestation prose.
• Portability constraint: the rule-generation logic lives in a pure lib (modules/nixos/telemetry/lib.nix, {lib}:-only, zero tsunaminoai.* references, exposed as flake.lib.telemetry) so praxis can pin or vendor it without dragging homelab coupling along. The NixOS modules take plain dataDir/domain/*File options; kurVolumes/step-ca/sops wiring lives only in host config.

What exists today (baseline — corrected)#

Decisions#

Each draft open-question, answered (research spike, 2026-07):

The chatty-log tax (unchanged, now with a mechanism)#

journald here is noisy — paperless emits thousands of INFO lines, and podman tags all container stderr as err so journal priority is unreliable. The Alloy pipeline relabels by unit/container and applies loki.process drop stages contributed via tsunaminoai.telemetry.logFilters — filtering happens at the agent, keyed on unit, never on priority.

Phase-1 architecture (tsunaminoai.telemetry)#

Alloy (per host) ──HTTPS/tailnet──▶ nginx :8427    (step-ca TLS + htpasswd basic auth;
                                       │            the ONLY ingest listener)
                                       ├─▶ victoriametrics 127.0.0.1:8428  (/api/v1/write, 12mo)
                                       └─▶ victorialogs    127.0.0.1:9428  (/insert/loki/…, 3month)
vmalert :8880 ── queries VM (prometheus groups) + VL (type: vlogs groups)
   │   rules = telemetry.alertRules ∪ monitorsToRuleGroups(complianceMonitors) ∪ Watchdog
   └──▶ alertmanager 127.0.0.1:9093 ──▶ ntfy-alertmanager ──▶ ntfy (topic ACLs)
                                    └──▶ webhook ──▶ healthchecks.io ping   (dead-man's-switch)
grafana 127.0.0.1:3000 ◀── nginx vhost :3443 (step-ca pattern: useACMEHost = hostFqdn, onlySSL)
data: kurVolumes.volumes.telemetry (250G) → /var/lib/telemetry/{victoriametrics,victorialogs,grafana}

Security posture: VM/VL single-node have no multi-user auth of their own, so the backends bind loopback-only and the nginx TLS edge is the only ingress — step-ca cert plus an htpasswd file (multi-entry: one user per pushing host, rotatable independently). Scaffold-time refinement over the original vmauth idea: the repo already fronts everything with the step-ca nginx pattern, so htpasswd-at-the-edge buys the same property with one fewer daemon; vmauth returns only if praxis ever needs per-tenant routing. Transport is the tailnet; all credentials via sops (/run/secrets, never /var/run; agent creds via an EnvironmentFile, nothing secret in the store).

Module surface#

Contribution collectors emulate the Gatus idiom (namespace-keyed attrsOf (listOf attrs), folded fleet-wide):

tsunaminoai.telemetry = {
  scrapeTargets.servarr = [ { targets = ["127.0.0.1:9633"]; } ];   # job defaults to namespace
  logFilters.paperless  = [ {
    selector = "{unit=\"podman-paperless.service\"}";
    dropRegex = ".*DEBUG.*";
  } ];
  alertRules.systemd    = [ {
    alert = "UnitFailed";
    expr = ''systemd_unit_state{state="failed"} == 1'';
    "for" = "5m";
    labels.severity = "critical";
  } ];
};

Roles:

• tsunaminoai.telemetry.server (ereshkigal): VM + VL + vmalert + Alertmanager + ntfy + Grafana (provisioned datasources incl. the victoriametrics-logs-datasource plugin, dashboards-as-code), plus the complianceMonitors input — control attrsets whose monitors[]/alerts[] are compiled by flake.lib.telemetry.monitorsToRuleGroups / alertsToRoutes.
• tsunaminoai.telemetry.agent (every capable host): Alloy journald pipeline + local exporters, remote_write/Loki-push to the ingest edge.

Exporters, ranked by “would have caught this month’s incidents”:

1. systemd — failed units (the 3 failed services + nix-ssh scope failures).
2. smartctl (--device megaraid,N for the PERC H710) — disk health.
3. ipmi — fans + temps (off by default until privileges are proven).
4. node — base metrics; its textfile collector finally gives the servarr-backup metric a home (and the writer gets fixed to actually execute).
5. Later: postgres (paperless), nginx, podman/cadvisor, blackbox, HA native /api/prometheus, borgmatic hooks, GPU on mokou.

Scope#

In the phase-1 spike (scaffold PR #243): the modules/nixos/telemetry family + ereshkigal-only enablement; node/systemd/smartctl exporters + journald shipping; one provisioned dashboard; one real alert (failed unit → ntfy); Watchdog → healthchecks.io; monitorsToRuleGroups with unit tests. Eval- and check-verified; live deploy is the acceptance step that follows.

Explicitly deferred (each a named follow-up):

• ~~Fleet agent rollout: mokou, shinobu, razer, octopi.~~ DONE (phase 2, PR #247): per-host ingest users; laptops rely on push-staleness (no absent() rules); octopi runs a memory-capped lean profile.
• voile (Synology), HAOS VM, macOS laptop instrumentation. (voile + HAOS get synthetic coverage via Gatus in the meantime.)
• ~~Gatus revival — independent small PR: enable on ereshkigal, contribute endpoints, actually serve status.falseblue.com.~~ DONE (phase 2, PR #245): status page; alerting rides vmalert (EndpointDown{tier!="intermittent"}), NOT Gatus-native alerting.
• ~~Retiring the SSH-poller deploy dashboard — only after telemetry proves equivalent coverage.~~ DONE (#250, 2026-08): module + host blocks removed after the 2026-08-07 acceptance drill proved end-to-end coverage.
• ~~Retiring Uptime Kuma (HA add-on) — parallel-run until the new stack survives a real incident.~~ DONE (#249, 2026-08): drill-proven (deliberate unit failure → dashboard drill-down → annotated ntfy page on the phone); Homer tile + doc removed. The add-on itself is uninstalled by hand in the HAOS UI (the VM is outside the flake).
• ~~myon NixOS onboarding~~ DONE (#164, 2026-08: hosts/x86_64-nixos/myon, see myon) — the supplemental on-myon heartbeat remains open.
• ipmi exporter enablement if privileges get fiddly.
• praxis integration proper — this phase only keeps lib.nix pure/portable.
• Traces / OTel / anomaly detection.

Acceptance (live validation, post-scaffold)#

1. Deploy to ereshkigal; one dashboard (node + systemd + SMART) renders in Grafana behind the step-ca vhost.
2. One real alert: start a deliberately-failing unit → ntfy notification arrives with the remediation annotation.
3. healthchecks.io shows the Watchdog pinging; stop vmalert briefly → the grace alarm fires (dead-man’s-switch proven end-to-end).
4. Spot-check two or three PromQL-authored rules against MetricsQL evaluation.

Appendix: considered and rejected#

Related#

• home-assistant-vm.md — the outage that motivates alerting.
• ereshkigal-storage-and-health.md — the kur capacity this stack lives on; the fan/SMART/systemd signals it watches.
• decommissioning.md — AST-02/09 evidence flow the compliance tie-in extends.