NTP Clock Step Cost a Batch Job 40 Seconds of Its Budget, Field Notes Show

A developer's 20-minute batch job kept stopping early on a host where every API call was fast. The cause was deadline accounting that read the wall clock, which shifted during the run and made the job think more time had passed than it had. Switching to a monotonic clock would have prevented the early exits, and the author also points to a separate retry bug in parsing the Retry-After header. The author also notes the difficulty of confirming the clock adjustment's cause, since the affected server was not theirs to inspect. The post closes by describing how injecting the clock makes such failures testable on a local machine.