How Silent Cron Job Bugs Survive Every Green Test: A Debugging Case Study

A software team managing 283 scheduled cron jobs discovered that passing tests do not guarantee a job is correctly placed, scheduled, or functioning as intended in production. A subtle read-after-write bug in a node health labeling script caused all real state transitions to be silently suppressed for six weeks, while every monitoring signal falsely reported the job as healthy. A separate race condition between two overlapping cron schedules could corrupt the state file mid-read, producing spurious log entries with missing context. The team found that fixing the logic bug immediately exposed a flaky test caused by a live cron tick firing during a sandboxed test run, revealing further gaps between test coverage and real deployment conditions. The incidents highlight that test coverage, liveness checks, and scheduling verification must each be treated as distinct, independently validated properties.