How a Dev Team Found Four Monitoring Alerts That Could Never Actually Fire

A development team spent a week auditing their alerting system and discovered four alerts that were structurally incapable of triggering, even when the conditions they were designed to detect occurred. One Prometheus alert for a missing scheduled task used a greater-than-zero filter, making it blind to complete task absence since zero is not greater than zero. A second alert failed silently because an unrouted backend metrics path was intercepted by a reverse proxy, which returned a 200 HTTP response with frontend HTML instead of a 404, causing curl and grep checks to pass without error. On the opposite end, a false-positive alert was paging engineers over failed end-to-end test runs that had actually passed, with the job-level failure caused only by an artifact upload timeout unrelated to test outcomes.