Why Scheduled Jobs Need Two Health Markers, Not One, to Catch Silent Failures

A common flaw in monitoring scheduled jobs is relying solely on a liveness heartbeat, which only confirms that a job ran — not that it succeeded. This distinction matters because a job can execute repeatedly while silently failing, leaving the underlying work undone for days or weeks without triggering any alert. The proposed fix separates job monitoring into two markers: a liveness beat written on every run, and a health marker written only when the job exits successfully. By comparing the age of both files, a monitoring sweep can distinguish between three states — the scheduler not firing, the job running but failing, and the job running and succeeding. The approach also recommends wrapping the alert notification channel itself in the same two-marker protocol to ensure that a broken alert path does not go undetected.