SRE Cuts $150K+ in Annual Datadog Waste by Auditing Observability Configs

A senior site reliability engineer at a payroll-processing company discovered over $150,000 in annual waste hidden within their Datadog observability setup after conducting a thorough configuration audit. The largest single saving came from restricting the Sensitive Data Scanner to only log sources where personally identifiable information could realistically appear, cutting daily costs from $390 to $56 — an 87% reduction worth roughly $120,000 per year. Further investigation uncovered phantom services in the Service Catalog, created by a shared logging library that overrode service name tags, which skewed monitoring and incident response routing. The engineer found that observability spending tends to go unreviewed because teams treat it like insurance, allowing configuration drift and dead settings to accumulate unchecked. The key takeaway is that organizations should audit their observability configurations — not just their bills — regularly, asking whether each component still serves a real, current purpose.