| Severity | Triggers | SMTP | Ntfy | SMS |
|---|---|---|---|---|
| CRITICAL | Service down, cert <7d, WireGuard down, disk >95% | ✓ | ✓ | ✓ |
| HIGH | Cert <14d, target down >10 min, disk >85% | ✓ | ✓ | — |
| WARNING | Memory >90%, backup failure, CrowdSec spike | ✓ | ✓ | — |
| INFO | Backup completed, update available | — | ✓ | — |
SMS rate limit: max 1 per alert group per 5 minutes, via Twilio (Gateway VPS sms-relay).
Each backup job pushes to its own cloud remote(s) directly, on its own schedule — there is no shared "pull everything" step.
| Tier | Schedule | Destination |
|---|---|---|
| VM snapshots (vzdump) | 02:00 daily | Local ZFS on hot-bm-nl |
| VM snapshots offsite | 04:30 daily | Hetzner Storage Box (rclone crypt) |
| Config sync | 01:00 daily | Forgejo → Codeberg + GitHub mirrors |
| Keycloak DB | 01:30 daily | hot-bm-nl primary, Hetzner+B2 fallback |
| hot-pn PrivateNexus DB | 03:00/03:30 daily | Local → Gateway → Hetzner + B2 (both, unconditionally — the reliable nightly canary) |
| Gateway VPS (Tor keys, PowerDNS DB, mail) | 05:30 daily | hot-bm-nl primary, Hetzner+B2 fallback |
PBS (Proxmox Backup Server) is local hardware (not a rented VPS), being integrated as an additional/primary backup target for hot-bm-nl's VMs — a work in progress, not yet the sole backup path for everything. Its datastore currently has no RAID (single disk).
Daily apt-get update && upgrade via systemd timer, 09:00 (outside the backup window). hot-bm-nl is check-only (never auto-applies — it runs all 4 production VMs, so an unattended bad kernel/qemu-server bump would take the whole fleet down). Reboot policy is per-host: Gateway and hot-bm-nl never auto-reboot; every other host does, 2 minutes after upgrading if required.
Any new recurring systemd timer must use OnCalendar, never OnBootSec/OnUnitActiveSec alone — the latter can silently wedge dead after certain reboot timing and never recompute a next-fire time. This has bitten two separate watchdog timers in this project's history.