Alerts & Notifications (ntfy + Discord)#

Alert pages are pushed to a self-hosted ntfy on ereshkigal. There is exactly one topic: alerts (option tsunaminoai.telemetry.server.ntfy.topic). ntfy topics aren’t enumerable by design — the topic name acts as a capability — so this page is the discovery mechanism.

What publishes to it#

vmalert (rules) → Alertmanager (grouping, 4h repeat) ─┬→ alertmanager-ntfy bridge → ntfy topic `alerts`   (everything)
                                                      └→ Discord webhook                                 (paging-worthy only, 24h re-nag)

Every notification carries the alert’s summary annotation as its title and description + remediation in the body — e.g. SystemdUnitFailed arrives as “mokou: unit foo.service failed” with the journalctl -u foo.service hint. Resolved notifications are sent too (green circle tag). To see currently-firing alerts with full annotations instead, use Grafana → the Alert groups link on the Telemetry Overview dashboard.

Subscribing — web#

  • https://alerts.inaba.network (Cloudflare Access, observability group) — log in via the top-right account menu as reader, then subscribe to alerts; or open https://alerts.inaba.network/alerts directly and enter the reader credentials in the per-topic auth dialog.
  • The Cloudflare App Launcher “Alerts (ntfy)” tile lands on the bare web UI (tiles can’t deep-link a topic) — hence this doc.

Subscribing — phone (ntfy app)#

The phone app subscribes over the tailnet, not Cloudflare:

  1. Prerequisite: the device trusts the FalseBlue step-ca root (the vhost cert is step-ca-issued).
  2. In the ntfy app, add server https://ereshkigal.armadillo-banfish.ts.net:2587.
  3. Subscribe to topic alerts as user reader with the cleartext password from the comment in sops telemetry/ntfy-auth-env — the app (verified on ntfy iOS 1.7.0) does basic auth with username+password and 401s if you paste the tk_… token in the password field. The reader token (NTFY_AUTH_TOKENS, label phone) is for CLI/scripts via Authorization: Bearer.

The Discord mirror (paging-worthy only)#

ntfy is the complete record; Discord is the “look now” channel, so its scope is deliberately narrow — the predecessor (Uptime Kuma → Discord) was an exercise in alarm fatigue. Enabled with tsunaminoai.telemetry.server.discordMirror.enable.

ntfy Discord
Receives every alert severity="critical" AND no control label
Today that means all of the below SystemdUnitFailed, EndpointDown, SmartDeviceUnhealthy
Excluded — compliance-control alerts (lgo-*, net-*, vpm-*), warnings, Watchdog
Re-notify while firing every 4h every 24h
Resolved messages yes yes

Compliance-control alerts are excluded on purpose: they signal configuration drift (chronic-prone, no 3am action) and stay on ntfy + Grafana. A control whose verification is genuinely not applicable gets a documented risk acceptance instead of a permanently-red critical.

Wiring notes:

  • The webhook URL is the same discord/notify-webhook sops secret the ci.autoUpdate failure notifications use — one channel, deliberately.
  • Alertmanager takes exactly ONE EnvironmentFile and envsubst’s its config at preStart, so the healthchecks ping URL and the Discord URL are merged into one rendered file by the alertmanager-env sops template. webhook_url_file is not usable: the unit is DynamicUser and the sops secrets are root-owned 0400.
  • Route order matters. Alertmanager stops at the first matching child route unless it sets continue, and continue resumes at the next sibling — it does not fall back to the parent receiver. Hence: watchdog route, then the Discord route with continue = true, then an explicit catch-all ntfy sibling last. Delete that catch-all and criticals would reach Discord and nothing else.

Access model (declarative)#

Users and tokens are provisioned declaratively from sops (telemetry/ntfy-auth-env → NTFY_AUTH_USERS/NTFY_AUTH_TOKENS); the ACL lives in Nix (settings.auth-access in the telemetry server module). No imperative ntfy user commands are needed anymore.

Identity Access Purpose
alertmanager write-only on alerts (token) the publish bridge; its token is mirrored in telemetry/ntfy-bridge-config and the two MUST stay in sync
reader read-only on alerts (password + token) humans: web login (password, noted in the sops file) and phone (token)
tsunami@falseblue.com admin (imperative, live host only) break-glass; not managed by Nix, survives provisioning
everyone else deny-all —

Gotchas#

  • ntfy identifies rate-limit visitors by X-Forwarded-For; the nginx edges forward it and the bridge goes through a loopback hop that stamps a synthetic IP. Don’t “test” with unauthenticated curls against 127.0.0.1:2586 — failed auths poison the shared 0.0.0.0 visitor bucket (auth-failure limiter, HTTP 429 code 42909, no exemption knob).
  • Do not hand-edit /var/lib/ntfy-sh/user.db for the provisioned users — startup re-syncs them from the env file, and entries removed from the env file are deleted from the db.
  • ntfy access --reset <user> on a provisioned user also wipes the config-synced grant from the db, silently breaking that identity (the bridge starts getting 403s) until the next systemctl restart ntfy-sh re-syncs it. Learned the hard way during the 2026-08-07 acceptance drill.