How a 'Dead-Man's Switch' Catches Silent Cron Job Failures Before Data Rots
A solo developer running multiple production systems discovered a critical blind spot when a scheduled curation job silently stopped running, leaving stale data served undetected for days. Unlike loud failures that trigger error alerts, the job produced no exceptions or log warnings, and all downstream health checks remained green. The developer's fix involved a two-part dead-man's switch: the job writes a timestamp only on verified success, and a separate watcher on an independent runtime alerts if that stamp grows too old. Crucially, the watcher runs in a different failure domain so it can catch the job's silence even if the job itself is completely dead. The threshold for triggering an alert must be tuned to the content's natural update rhythm rather than the job's schedule, to prevent alert fatigue from rendering the safeguard useless.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in