A single order ID label crashed a metrics server and killed payment monitoring

A developer added an order ID as a label to a metrics counter on a Friday to track individual customer orders through a queue. Because order IDs are unique — around 60,000 generated daily across four services — the change caused active time series to explode from 1.1 million to 14 million within three days. The metrics server began crashing in a restart loop by Sunday evening, and by Monday morning the team had no monitoring during a live payments incident. Engineers stopped the bleeding by dropping the label at the scrape layer before redeploying the application, then introduced series budgets, CI checks blocking high-cardinality label names, and a dashboard showing series counts per metric. The incident highlighted that unique identifiers belong on traces or exemplars, not on metric label dimensions where they multiply storage costs across every existing label combination.