How to Instrument a Go API with Prometheus Without Blowing Up Cardinality

A developer has detailed the design decisions behind adding a Prometheus metrics layer to a Go API, focusing on keeping instrumentation accurate and cost-effective at scale. Key metrics include request rate, error ratio, latency histograms, in-flight request gauges, and response size tracking, with route templates used as labels instead of resolved URLs to prevent cardinality explosion. Database connection pool stats are surfaced via a dedicated collector to detect pool exhaustion early, before it appears as user-facing latency. Security events such as auth failures and rate-limit triggers are tracked in parallel with structured logs, enabling real-time alerting on threats like brute-force attempts rather than after-the-fact log analysis. The author emphasizes that every metric is a long-term maintenance commitment, and the goal is a minimal, deliberate set of signals that can answer whether a service is healthy without needing to open a log file.