Batch job gave different daily counts on two machines; root causes were timezone, locale sorting, and salted hashing
A developer ran the same batch command on a laptop and a remote host and got different per-day record counts, though both runs exited successfully. The totals matched but one day's records had landed in the wrong bucket, pointing to the bucketing logic rather than the model. Investigation traced the discrepancy to ambient settings the code never pinned, including the process timezone, which shifts calendar-day assignment. Locale-dependent file sorting and per-process salted string hashing for shard assignment were identified as further sources of nondeterminism. The author then rebuilt the pipeline to refuse to start when its environment does not match expectations. The article notes the labeling step ran on a free hosted server, though the author states the bugs originated before any model requests were made.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in