Telemetry (metrics, logs & alerting)#

tsunaminoai.telemetry is the fleet’s single observability pane: one place metrics and logs land, one alert engine, one notification pipeline (decided in the observability unification plan). Everything backend runs loopback-only on ereshkigal (tsunaminoai.telemetry.server); every capable host runs a Grafana Alloy agent (tsunaminoai.telemetry.agent) that pushes metrics to VictoriaMetrics and journald to VictoriaLogs over one authenticated HTTPS edge. This page is the operations map — what listens where, which rules can page, and how to onboard a host. Delivery (ntfy/Discord, subscribing) is alerts.md; synthetic checks are status-page.md.

Data flow#

every agent host                          ereshkigal (server role)
┌──────────────────────────┐   HTTPS      ┌────────────────────────────────────────┐
│ Alloy: journald ─────────┼─ basic auth ─▶ nginx ingest edge :8427                │
│        node_exporter     │  (step-ca)   │   /api/v1/write        → VM     :8428  │
│        systemd exporter  │              │   /insert/loki/api/v1/ → VLogs  :9428  │
│        smartctl/ipmi     │              │            │                           │
└──────────────────────────┘              │  vmalert :8880 (metrics) :8881 (logs)  │
                                          │            ▼                           │
                                          │  Alertmanager :9093                    │
                                          │   ├─ Watchdog → healthchecks.io (DMS)  │
                                          │   ├─ critical, no control → Discord    │
                                          │   └─ everything → bridge :8111         │
                                          │        → nginx hop :2585 → ntfy :2586  │
                                          │  Grafana :3080 reads VM + VLogs        │
                                          └────────────────────────────────────────┘

Port map (ereshkigal)#

Only three surfaces are meant for humans; everything else is a loopback debug query surface you hit from a shell on ereshkigal (there is no tcpdump on hosts, but curl against these is the fastest triage there is).

Alert rule catalogue#

Alerting is ONE engine: vmalert evaluates every rule, Alertmanager fans out. Rules come from three sources — the server module itself, the Gatus module (modules/nixos/monitoring/default.nix), and any module contributing to tsunaminoai.telemetry.alertRules.<namespace> (one vmalert group per namespace; set datasource = "logs" on a rule for a LogsQL group).

Both AutoUpgrade* rules point at the fleet-convergence runbook; the guard metrics they watch are written by every nixos-upgrade guard run to the node-exporter textfile collector.

Grafana dashboards are Nix attrsets#

There are no dashboard JSON files. Dashboards are built in modules/nixos/telemetry/server.nix from the mkPanel / mkTable / mkLogs / mkDashboard helpers and serialized to JSON at build time — so nix build .#nixosConfigurations.ereshkigal.config.system.build.toplevel catches a malformed dashboard before it ever reaches Grafana. Two helper facts that bite:

• Data links need the literal-${ form. Grafana interpolates ${__data.fields.<column>} itself; inside a Nix indented string that must be written ''${__data.fields.host} (or \${…} in a double-quoted string) or Nix eats it as interpolation.
• The failed-units table deep-links each row to the Logs & Errors dashboard by passing var-host / var-unit — which only resolves because the template variables are sourced from the systemd exporter’s label set (every unit on every host, no allowlist).

Onboarding a new host (agent)#

1. Enable the agent and point it at a per-host credentials file:

tsunaminoai.telemetry.agent = {
  enable = true;
  credentialsEnvFile = config.sops.secrets."telemetry/ingest-env-<host>".path;
};
sops.secrets."telemetry/ingest-env-<host>".mode = "0400";

2. Add the secret to secrets.yaml before building (sops-nix validates at eval time). It must define TELEMETRY_INGEST_USERNAME and TELEMETRY_INGEST_PASSWORD; keep it under /run/secrets, never /var/run.
3. Give the host its own user in the server’s htpasswd (tsunaminoai.telemetry.server.ingestBasicAuthFile) — one entry per host so credentials rotate per-host: echo "user:$(openssl passwd -6 'pass')".
4. Exporters default on: node_exporter (:9100, with the textfile collector) and the standalone systemd exporter (:9558). smartctl and ipmi are opt-in (agent.exporters.smartctl.enable, …). Journald shipping defaults on (agent.journal.enable).

Label plumbing worth knowing:

• Failed units are systemd_unit_state{name=…,state="failed"} from the standalone systemd exporter — NOT node_systemd_* (that node_exporter collector is not enabled).
• Journald log fields host/unit join exactly against metric labels host/name — the agent stamps host as an external label on both streams and relabels __journal__systemd_unit → unit. This is what makes the dashboard’s per-unit log drill-down line up.
• Extra scrape targets and journald drop filters are contributed per-module via tsunaminoai.telemetry.scrapeTargets.<ns> and tsunaminoai.telemetry.logFilters.<ns> (filter on unit/container, never on level — podman corrupts journald priority).
• armv6 hosts (Pi Zero tier) get no agent — Grafana dropped armv6 builds; they stay heartbeat-only.

Grafana admin via Cloudflare Access#

Grafana at grafana.inaba.network signs users in through Cloudflare Access (OIDC, Entra behind it). Admin vs Viewer is decided by group membership, and three things must all be true or everyone lands as Viewer (this cost an afternoon on 2026-08-03):

1. The Cloudflare SaaS-OIDC app must offer the groups scope and Grafana must request it (auth.generic_oauth.scopes = "openid email profile groups" in modules/nixos/telemetry/server.nix; oidc.scopes on the ereshkigal origin). Without the scope Cloudflare omits the claim entirely — group_filter_regex alone does nothing.
2. oidc.groupFilterRegex on the origin (→ saas_app.group_filter_regex in modules/flake/terraform/cloudflare.nix) selects which IdP group names appear; ereshkigal scopes it to ^sg-voile-personal-admins$.
3. Cloudflare emits the claim as plain name strings (groups:["sg-voile-personal-admins"]), not objects — so grafana.oauthRoleAttributePath must match the name, never an Entra GUID: contains(groups[*], 'sg-voile-personal-admins') && 'Admin' || 'Viewer'.

Changing the scope or regex is a Cloudflare change: nix run .#terraform-cloudflare -- plan, then the owner runs apply. To see what Grafana actually received, set services.grafana.settings.log.filters = "oauth.generic_oauth:debug" temporarily — it logs the raw token, granted scopes, and the resolved role. auto_login fires immediately on restart, so a login that predates a just-applied terraform change can look like a failure when it is only timing.

Retention#

The two grammars differ: VictoriaLogs wants a capital-M month suffix — 3month makes victoria-logs exit with cannot parse duration and crash-loop. Data lives under server.dataDir (/var/lib/telemetry), which on ereshkigal must be the telemetry kur LV, never the 90%-full btrfs root — the units carry RequiresMountsFor so a missing mount stops them instead of silently filling the root disk.

Compliance exceptions (risk acceptances)#

A control whose live verification can’t pass on a host gets a documented risk acceptance instead of a permanently-red critical: declare tsunaminoai.compliance.exceptions.<ID> and wire it through with

tsunaminoai.telemetry.server.complianceExceptions =
  config.tsunaminoai.compliance.exceptions;

The excepted monitors are dropped before vmalert rule compilation (so the alert genuinely stops), while the control’s config still applies and the attestation records risk_acceptance instead of a pass. The two options are deliberately separate so telemetry has no dependency on the compliance module. Live example: ereshkigal waives LGO-02’s auditd monitor (the host boots audit=0). Details in compliance.

Gotchas#

• Alertmanager route order is load-bearing. First matching child route wins; continue resumes at the next sibling, never falling back to the parent receiver. The order is: watchdog → Discord (continue = true) → explicit catch-all ntfy sibling. Delete the catch-all and criticals go to Discord only.
• Alertmanager takes exactly ONE EnvironmentFile and envsubst’s its config at preStart — so the healthchecks ping URL and DISCORD_WEBHOOK_URL are merged into one file by the ereshkigal alertmanager-env sops template. webhook_url_file cannot work: the unit is DynamicUser, the sops secrets are root-owned 0400.
• Never fire unauthenticated curls at ntfy loopback (:2586). ntfy checks its auth-failure limiter (HTTP 429, code 42909, no exemption knob) before credentials, and no-XFF clients all share the 0.0.0.0 visitor bucket — one “test” login failure blocks the publish bridge. Full story in alerts.md.
• Alert acceptance drill (run it after touching the pipeline): systemd-run --unit drill-test sh -c 'exit 1' → Failed-units row on the overview dashboard within ~1 min → SystemdUnitFailed fires after its 5m for → ntfy page. systemctl reset-failed drill-test resolves it.
• Grafana’s loopback port is 3080, not 3000 — Open-WebUI defaults to 3000 and whoever loses the bind race proxies the wrong app.

Related#

• Alerts & Notifications — how pages are delivered (ntfy topic, phone setup, the Discord mirror, the ntfy access model).
• Status page — the Gatus synthetic checks feeding EndpointDown and the tenant-scoped status pages.
• Fleet convergence runbook — what to do when AutoUpgradeCompareFailed / AutoUpgradeStuckBehindRelease fires.
• Observability unification plan — the decision record behind this architecture.
• Compliance — the SCF control monitors compiled into vmalert and the risk-acceptance mechanism.