Alerts and notifications
Alert definitions are versioned in the portal repository. Installation requires a dedicated Grafana folder, a dedicated email contact point and a matching notification-policy branch. Preserve every existing route and receiver. The recipient is configured privately and never published here.
| Condition | Initial behavior |
|---|---|
| No telemetry | Five minutes without a new report |
| Temperature | Warning above 75°C for 5m; critical above 80°C for 2m |
| Power flags | Active undervoltage/throttling; separate newly observed historical flags |
| Filesystem/inodes | Above 80% warning, 90% critical |
| Memory | Sustained pressure or OOM events |
| Compute/storage | Sustained saturation or disk errors |
| Services | Failed units or excessive restarts |
| Connectivity | Sustained DNS or Internet probe failure |
| Pipeline | Collector failures, remote-write errors, stale data |
| Portal | Independent HTTPS failure or imminent certificate expiry |
Group firing and recovery emails. Gate downstream device alerts on fresh host reports to avoid outage storms; separately alert on absent data. An API process health check is not a collection health check. An independent Grafana synthetic HTTPS probe is optional until availability and account limits are verified.
Do not assume an alert exists because its definition is committed. Validate a controlled firing and recovery notification before marking delivery complete. Inspect current private retention, active-series and log quotas and incremental cost before enabling broad collection; do not purchase capacity or change global retention without agreement.
Deployed checks
Section titled “Deployed checks”The dedicated folder contains 27 rules, including low available memory, private kernel disk-error logs, public ingestion failures/dropped series and the public filesystem quota. The independent HTTPS check runs from one external location every ten minutes and also checks the health response body and certificate. Its alert queries retain the last check for fifteen minutes. Detection can therefore take roughly a check interval plus evaluation/notification delay.
The email integration test returned success. A controlled seven-minute collector pause fired the missing-data alert and automatic recovery cleared it. The recipient confirmed both firing and recovery messages arrived; email delivery acceptance is complete.