Detecting Silent Failures: API Error Rates, Cron Success Timestamps, and Alert Testing

A DEV Community guide argues that reliable failure detection depends on signals that distinguish genuine outcomes from mere completed requests. It recommends counting API outcomes by category, recording whether scheduled jobs meet a completion check, and tracking the timestamp of the last verified success to catch jobs that never ran. The author suggests running the poller outside the observed job, with a deadline and a completion predicate taken from the API contract. It also warns against putting identifiers, tokens, or raw error text into metric labels, since these create high cardinality and can expose sensitive data. Diagnostic detail is better kept in access-controlled logs, with metrics, logs, and traces each handling a distinct job.