Telemetry (metrics, logs & alerting)#

tsunaminoai.telemetry is the fleet’s single observability pane: one place metrics and logs land, one alert engine, one notification pipeline (decided in the observability unification plan). Everything backend runs loopback-only on ereshkigal (tsunaminoai.telemetry.server); every capable host runs a Grafana Alloy agent (tsunaminoai.telemetry.agent) that pushes metrics to VictoriaMetrics and journald to VictoriaLogs over one authenticated HTTPS edge. This page is the operations map — what listens where, which rules can page, and how to onboard a host. Delivery (ntfy/Discord, subscribing) is alerts.md; synthetic checks are status-page.md.

Data flow#

every agent host                          ereshkigal (server role)
┌──────────────────────────┐   HTTPS      ┌────────────────────────────────────────┐
│ Alloy: journald ─────────┼─ basic auth ─▶ nginx ingest edge :8427                │
│        node_exporter     │  (step-ca)   │   /api/v1/write        → VM     :8428  │
│        systemd exporter  │              │   /insert/loki/api/v1/ → VLogs  :9428  │
│        smartctl/ipmi     │              │            │                           │
└──────────────────────────┘              │  vmalert :8880 (metrics) :8881 (logs)  │
                                          │            ▼                           │
                                          │  Alertmanager :9093                    │
                                          │   ├─ Watchdog → healthchecks.io (DMS)  │
                                          │   ├─ critical, no control → Discord    │
                                          │   └─ everything → bridge :8111         │
                                          │        → nginx hop :2585 → ntfy :2586  │
                                          │  Grafana :3080 reads VM + VLogs        │
                                          └────────────────────────────────────────┘

Port map (ereshkigal)#

Only three surfaces are meant for humans; everything else is a loopback debug query surface you hit from a shell on ereshkigal (there is no tcpdump on hosts, but curl against these is the fastest triage there is).

Alert rule catalogue#

Alerting is ONE engine: vmalert evaluates every rule, Alertmanager fans out. Rules come from three sources — the server module itself, the Gatus module (modules/nixos/monitoring/default.nix), and any module contributing to tsunaminoai.telemetry.alertRules.<namespace> (one vmalert group per namespace; set datasource = "logs" on a rule for a LogsQL group).

Both AutoUpgrade* rules point at the fleet-convergence runbook; the guard metrics they watch are written by every nixos-upgrade guard run to the node-exporter textfile collector.

Grafana dashboards are Nix attrsets#

There are no dashboard JSON files. Dashboards are built in modules/nixos/telemetry/server.nix from the mkPanel / mkTable / mkLogs / mkDashboard helpers and serialized to JSON at build time — so nix build .#nixosConfigurations.ereshkigal.config.system.build.toplevel catches a malformed dashboard before it ever reaches Grafana. Two helper facts that bite:

• Data links need the literal-${ form. Grafana interpolates ${__data.fields.<column>} itself; inside a Nix indented string that must be written ''${__data.fields.host} (or \${…} in a double-quoted string) or Nix eats it as interpolation.
• The failed-units table deep-links each row to the Logs & Errors dashboard by passing var-host / var-unit — which only resolves because the template variables are sourced from the systemd exporter’s label set (every unit on every host, no allowlist).

Onboarding a new host (agent)#

1. Enable the agent and point it at a per-host credentials file:

tsunaminoai.telemetry.agent = {
  enable = true;
  credentialsEnvFile = config.sops.secrets."telemetry/ingest-env-<host>".path;
};
sops.secrets."telemetry/ingest-env-<host>".mode = "0400";

2. Add the secret to secrets.yaml before building (sops-nix validates at eval time). It must define TELEMETRY_INGEST_USERNAME and TELEMETRY_INGEST_PASSWORD; keep it under /run/secrets, never /var/run.
3. Give the host its own user in the server’s htpasswd (tsunaminoai.telemetry.server.ingestBasicAuthFile) — one entry per host so credentials rotate per-host: echo "user:$(openssl passwd -6 'pass')".
4. Exporters default on: node_exporter (:9100, with the textfile collector) and the standalone systemd exporter (:9558). smartctl and ipmi are opt-in (agent.exporters.smartctl.enable, …). Journald shipping defaults on (agent.journal.enable).

Label plumbing worth knowing:

• Failed units are systemd_unit_state{name=…,state="failed"} from the standalone systemd exporter — NOT node_systemd_* (that node_exporter collector is not enabled).
• Journald log fields host/unit join exactly against metric labels host/name — the agent stamps host as an external label on both streams and relabels __journal__systemd_unit → unit. This is what makes the dashboard’s per-unit log drill-down line up.
• Extra scrape targets and journald drop filters are contributed per-module via tsunaminoai.telemetry.scrapeTargets.<ns> and tsunaminoai.telemetry.logFilters.<ns> (filter on unit/container, never on level — podman corrupts journald priority).
• armv6 hosts (Pi Zero tier) get no agent — Grafana dropped armv6 builds; they stay heartbeat-only.

Retention#

The two grammars differ: VictoriaLogs wants a capital-M month suffix — 3month makes victoria-logs exit with cannot parse duration and crash-loop. Data lives under server.dataDir (/var/lib/telemetry), which on ereshkigal must be the telemetry kur LV, never the 90%-full btrfs root — the units carry RequiresMountsFor so a missing mount stops them instead of silently filling the root disk.

Compliance exceptions (risk acceptances)#

A control whose live verification can’t pass on a host gets a documented risk acceptance instead of a permanently-red critical: declare tsunaminoai.compliance.exceptions.<ID> and wire it through with

tsunaminoai.telemetry.server.complianceExceptions =
  config.tsunaminoai.compliance.exceptions;

The excepted monitors are dropped before vmalert rule compilation (so the alert genuinely stops), while the control’s config still applies and the attestation records risk_acceptance instead of a pass. The two options are deliberately separate so telemetry has no dependency on the compliance module. Live example: ereshkigal waives LGO-02’s auditd monitor (the host boots audit=0). Details in compliance.

Gotchas#

• Alertmanager route order is load-bearing. First matching child route wins; continue resumes at the next sibling, never falling back to the parent receiver. The order is: watchdog → Discord (continue = true) → explicit catch-all ntfy sibling. Delete the catch-all and criticals go to Discord only.
• Alertmanager takes exactly ONE EnvironmentFile and envsubst’s its config at preStart — so the healthchecks ping URL and DISCORD_WEBHOOK_URL are merged into one file by the ereshkigal alertmanager-env sops template. webhook_url_file cannot work: the unit is DynamicUser, the sops secrets are root-owned 0400.
• Never fire unauthenticated curls at ntfy loopback (:2586). ntfy checks its auth-failure limiter (HTTP 429, code 42909, no exemption knob) before credentials, and no-XFF clients all share the 0.0.0.0 visitor bucket — one “test” login failure blocks the publish bridge. Full story in alerts.md.
• Alert acceptance drill (run it after touching the pipeline): systemd-run --unit drill-test sh -c 'exit 1' → Failed-units row on the overview dashboard within ~1 min → SystemdUnitFailed fires after its 5m for → ntfy page. systemctl reset-failed drill-test resolves it.
• Grafana’s loopback port is 3080, not 3000 — Open-WebUI defaults to 3000 and whoever loses the bind race proxies the wrong app.

Related#

• Alerts & Notifications — how pages are delivered (ntfy topic, phone setup, the Discord mirror, the ntfy access model).
• Status page — the Gatus synthetic checks feeding EndpointDown and the tenant-scoped status pages.
• Fleet convergence runbook — what to do when AutoUpgradeCompareFailed / AutoUpgradeStuckBehindRelease fires.
• Observability unification plan — the decision record behind this architecture.
• Compliance — the SCF control monitors compiled into vmalert and the risk-acceptance mechanism.