How cutting two-thirds of alerts helped a team catch incidents faster
A dev team was receiving nearly 300 monitoring alerts per day, causing alert fatigue so severe that a critical production incident was missed amid the noise. The root cause was indiscriminate alerting on every available metric, with thresholds set arbitrarily rather than based on real system behavior. The team overhauled their approach by shifting from cause-based alerts to symptom-based ones, tying notifications to SLO error budget burn rates instead of raw resource metrics like CPU or memory usage. They also introduced distributed tracing with a unified trace ID, which reduced incident diagnosis time from hours to minutes across their microservices architecture. After disabling roughly two-thirds of their alerts and retaining only a handful tied to user-facing impact, the team found they were catching incidents more quickly, not less.
