How a Silent Heartbeat Monitor Let a System Outage Go Unnoticed for 43 Days

Heartbeat monitoring works by alerting when expected periodic signals stop arriving, rather than waiting for an error to appear — but this approach is harder to implement correctly than it seems. A team discovered their own heartbeat monitor had gone silent for 43 days due to a flaw in how repeated alerts were deduplicated, meaning the system paged once and then went quiet. The core problem lies in alert deduplication: when a monitor fires a single alert and suppresses all subsequent ones, a prolonged outage can persist indefinitely without further notification. Proper heartbeat monitoring requires distinguishing between three states — alive, dead, and unknown — rather than a simple binary, to avoid false alerts on newly created monitors and missed alerts on stale ones. The incident highlights that detecting a failure is only half the challenge; a reliable dead-man's switch must continue escalating until the problem is resolved.