Four Practical Lessons for Writing Alerts That Actually Work

A software engineer writing for DEV Community shares four non-obvious lessons learned from building and maintaining alerting systems in production environments. A key insight arose when a colleague's alert tracked message counts as a throughput proxy, unaware that each message could contain up to ten URLs, causing thresholds and SLOs to be silently denominated in the wrong unit. The author argues that monitoring queue age — rather than simple throughput floors — better distinguishes genuine stalls from idle periods, eliminating false pages during quiet workload windows. Tracking how many times a per-job deadline is exceeded, not just whether it fires, helps differentiate a system-wide failure from a small number of slow outlier jobs. Finally, setting alert thresholds where exceedances are genuinely rare ensures that a page signals something is truly stuck rather than merely slow.