Status Page (Gatus)#

status.falseblue.com (LAN/tailnet with the AdGuard rewrite) · status.inaba.network (anywhere, behind Cloudflare Access)

Gatus runs on ereshkigal (tsunaminoai.monitoring, loopback :8425) and probes everything declared in Nix — this replaced the manually-configured Uptime Kuma HA add-on (retired 2026-08, #249). Uptime history lives in sqlite on the telemetry kur LV (/var/lib/telemetry/gatus).

Contributing endpoints#

Any module (or host config) appends to the namespace-keyed collector; the namespace becomes the Gatus group unless overridden:

tsunaminoai.monitoring.endpoints.my-service = [
  {
    name = "my-api";
    url = "https://127.0.0.1:9999/health";
    interval = "5m";
    conditions = ["[STATUS] == 200"];
  }
];

Current sources of truth:

  • tsunaminoai.cloudflare.origins — every public-surface origin is probed automatically at its loopback backend. (Public-edge probing exists behind tsunaminoai.cloudflare.monitorPublicEdge, default off: probes from ereshkigal egress via the myon exit node + Cloudflare, which measures the tunnel rather than the service.)
  • Telemetry self-checks — Grafana / VictoriaMetrics / VictoriaLogs health endpoints, contributed by tsunaminoai.telemetry.server.
  • ereshkigal host config — the full set migrated from Uptime Kuma lives in hosts/x86_64-nixos/ereshkigal/monitors.nix: fleet ICMP, Home Assistant (the 2026-07-15 ~10 h-unnoticed outage this whole stack exists to prevent), FalseBlue web, SC2 sites, client sites, third-party vendor status, and DNS resolvers. Each carries extra-labels.tenant (personal / sc2 / wcw) so Grafana’s Service Availability dashboard can filter per tenant — the Praxis angle.

Availability tiers (why laptops never page)#

Endpoints for hosts that are allowed to be down carry extra-labels.tier = "intermittent" (gatus publishes extra-labels as metric labels); third-party vendor checks carry tier = "informational", and personal-tenant endpoints are excluded wholesale. The paging rule skips all three:

EndpointDown = gatus_results_endpoint_success{job="gatus",tenant!="personal",tier!="intermittent",tier!="informational"} == 0  (for: 5m)

They still show on the status page — red is fine, paging is not. There are deliberately no absent()-per-host rules anywhere: a sleeping laptop’s pushed metrics just go stale and nothing fires.

Tenant-scoped pages (statusPages)#

Gatus has no per-user filtering, so audience scoping is done by instance: each tsunaminoai.monitoring.statusPages.<name> entry runs an extra Gatus process that serves only the endpoints whose extra-labels.tenant matches, and each page is fronted by its own Cloudflare Access origin gated on that audience’s Entra group.

Option Default Meaning
tsunaminoai.monitoring.statusPages.<name>.port — (required) Loopback port for this page’s Gatus instance.
tsunaminoai.monitoring.statusPages.<name>.tenants — (required) Endpoint extra-labels.tenant values this page shows (exact match; endpoints without the label count as untagged).

Mechanics (unit gatus-page-<name>, same static gatus user as the primary):

  • The instance probes independently but publishes no metrics (metrics = false) — the primary instance on :8425 owns the entire alerting path, so the duplicate probes never double-count in EndpointDown.
  • Sqlite history lives beside the primary’s, at <dataDir>-page-<name> (i.e. on the telemetry kur LV on ereshkigal).
  • tsunaminoai.monitoring.environmentFile is passed to every instance, so ${VAR} secret substitution in endpoint definitions keeps working.

Adding a tenant page (live example: family)#

The family page is status.inaba.network — ereshkigal loopback :8430 behind a Cloudflare Access origin (hosts/x86_64-nixos/ereshkigal/default.nix):

tsunaminoai.monitoring.statusPages.family = {
  port = 8430;            # free loopback port
  tenants = ["personal"]; # shows endpoints tagged extra-labels.tenant = "personal"
};

tsunaminoai.cloudflare.origins.status = {
  publicHostname = "status.inaba.network";
  backend = "http://127.0.0.1:8430"; # gatus-page-family
  accessGroup = "family";            # Entra group gate — page content is
  # …                                # already tenant-scoped, so the broad
};                                   # group leaks nothing about other tenants

For a new audience (e.g. a client):

  1. Tag its endpoints with extra-labels.tenant = "<tenant>" (hosts/x86_64-nixos/ereshkigal/monitors.nix).
  2. Add statusPages.<name> with a free loopback port and that tenant list.
  3. Add a tsunaminoai.cloudflare.origins.<name> entry backed by the new port, accessGroup set to the audience’s Entra group (which must exist in groupIds, modules/flake/terraform/default.nix), then nix run .#terraform-cloudflare -- plan (the user runs apply).
  4. Gates: the module’s eval-time assertions reject malformed/duplicate endpoints for all instances, and nix build .#checks.x86_64-linux.gatus-config-ereshkigal renders the primary config (Gatus config errors are startup-fatal).

Alerting path (one engine)#

Gatus does not use its own alerting. Its /metrics are scraped by the local Alloy agent into VictoriaMetrics; vmalert evaluates EndpointDown, GatusScrapeDown and ProbedCertificateExpiringSoon (step-ca renewal watchdog over every HTTPS endpoint probed — see step-ca.md to issue/renew a cert) → Alertmanager → ntfy (topic + subscribe instructions there). Endpoints probed with client.insecure = true carry extra-labels.cert = "external" and are excluded from the cert-expiry alert (an unmanaged appliance cert isn’t our renewal to watch). Whole-host death of ereshkigal is covered by the healthchecks.io Watchdog (dead-man’s-switch), not by Gatus.

See the observability unification plan for the full architecture.

DNS + certificates#

  • The status.falseblue.com cert is issued by step-ca over ACME HTTP-01 with a dedicated lego listener on :8889 (:8888 belongs to the host cert) and a :80 vhost that serves only /.well-known/acme-challenge/. Issuance is self-contained: ereshkigal maps the name to itself via networking.hosts, so no external DNS is involved.
  • Clients resolve the name via the AdGuard rewrite status.falseblue.com → 192.168.0.20 (same precedent as ca.falseblue.com). MagicDNS does not rewrite falseblue.com — off-LAN clients use status.inaba.network (the family-tenant Gatus instance behind Cloudflare Access) instead. The old host-cert fallback vhost on :8426 was retired in #177.

Predecessors (retired 2026-08)#

Uptime Kuma (HA add-on) and the SSH-poller deploy dashboard (tsunaminoai.deploy, :8420) were retired in #249/#250 after the 2026-08-07 acceptance drill proved end-to-end coverage — a drilled unit failure caught, drill-down to logs working, and a correctly-annotated ntfy page received on the phone.