The main stack, all on sn-monitor (10.10.50.104):
| Service | URL | Notes |
|---|---|---|
| Grafana | grafana.house-of-trae.com | Alert rules, dashboards |
| Prometheus | internal only, port 9090 | Scrapes the whole fleet |
| Loki | port 3100 | Log aggregation |
| Uptime Kuma | status.house-of-trae.com (public), monitor.house-of-trae.com (admin) |
Uptime Kuma's two login layers — easy to get confused by: monitor.house-of-trae.com is gated by the SSO wall, but SSO only controls whether you can reach the app at all — it doesn't log you into Uptime Kuma's own separate internal admin account. Once past SSO, if you haven't also logged into Uptime Kuma itself, the app falls back to showing its default public status view — the exact same thing status.house-of-trae.com shows with no gate at all. The two pages will look identical until you log into Uptime Kuma's own account (credential in Vaultwarden, not your Keycloak login) to reach the real dashboard with every monitor, not just the 6 public ones.
There's no visible login button anywhere on the page to get you there — the public status view has no admin link on it at all, by Uptime Kuma's own design. You have to navigate directly to https://monitor.house-of-trae.com/login yourself.
If that link throws a generic "resource is no longer available" error: it's very likely not actually broken — Keycloak's auth flow uses short-lived, single-use codes baked into the redirect URL at each step, so clicking a link that's already partway through the chain (browser back/forward, a stale reload, a delayed second click) will throw exactly this error. Open a fresh tab and type the URL directly rather than reusing/refreshing an old one before assuming it's a real fault.
node-exporter gotcha: Prometheus runs in bridge network 172.18.0.0/16 — UFW must allow that subnet through to port 9100, or scrapes silently fail.
Gatus (/opt/stacks/gatus/) runs on the Gateway, not sn-monitor, deliberately — sn-monitor lives on hot-bm-nl, so it physically can't alert on its own host dying. Gatus exists specifically to catch that.
21 endpoints as of the last widening pass, covering every VM, all 6 sn-web client sites, and sn-infra's admin tools — none of which had independent-of-hot-bm-nl coverage before. Alerts via email only (deliberately not Ntfy/SMS, since those have historically shared a failure domain with what Gatus is meant to catch).
Gotcha: an SSO-gated endpoint check needs client.ignore-redirect: true, or Gatus's HTTP client follows the redirect into the oauth2-proxy/Keycloak dance and 400s against a genuinely healthy target.
3 CRITICAL-severity rules feed the SMS relay directly: Node Down, Disk >95%, TLS cert <7 days. Contact points: email-hot (all critical/high alerts) and ntfy-critical (severity=critical only, webhook to Ntfy). See Backups & Alerting for the full severity/channel matrix.
Gotcha: alert rules default to execErrState: Alerting (fail-open on a query error) — if Grafana's own datasource errors out (e.g. during a restart), every rule fires CRITICAL for every host it last saw, a real false-alarm storm that's happened before. Should be set to execErrState: Error instead.