Developer's monitoring stack silently failed for days due to a deduplication bug

A software developer running a self-hosted observability stack discovered on August 13 that Prometheus had stopped writing data to disk due to a full storage volume, yet all health checks and dashboards continued appearing normal. The failure went undetected because Prometheus was not scraping its own internal metrics, leaving no alert rules in place to catch the issue. After adding a watchdog to report errors to Sentry, the developer believed the problem was resolved when alerts went quiet — but the silence was misleading. A module-level deduplication set in the watchdog code meant the alert fired only once per process lifetime, masking dozens of ongoing failures per hour. The root cause was a deduplication window set to 'forever,' making a persistent, recurring fault look identical to a one-time resolved event.