Why a Green Health Check Can Mask a Full Service Outage

Health checks can fail in two ways: false alarms that interrupt engineers unnecessarily, and false all-clears that silently mask real outages — and most monitoring systems are only tuned to prevent the first. A common problem is that health check endpoints return a simple 200 OK status without verifying whether the service's core dependencies are actually functioning. Checks that run inside the same cluster as the service they monitor can fail alongside it, producing no alert at all. A dead man's switch — an external service that expects a regular ping and raises an alert when that ping stops — addresses the blind spot where a failed checker reports nothing. Engineers are advised to treat 'unknown' as a distinct state from 'healthy' and to regularly test failure paths, not just the success path.