How a Single Mislabeled Metric Crashed Prometheus and What Teams Can Do

A Prometheus monitoring system crashed overnight after memory usage surged from 8GB to 32GB, triggered by a single request_path label added to an HTTP metric that generated 500,000 new time series. The incident, known as the cardinality cliff, highlights how unique label combinations multiply time series exponentially and overwhelm monitoring infrastructure. Engineers recommend enforcing strict label policies that forbid unbounded labels such as user_id, request_id, and IP addresses, while capping per-metric cardinality at 10,000 series. To manage scale, teams can use metric relabeling in Prometheus to drop or aggregate high-cardinality labels before ingestion. For long-term storage and multi-instance querying, a Thanos sidecar architecture paired with pre-computed recording rules is advised to reduce query load on dashboards.