Why LLMs Drift Even at Temperature 0: The Floating-Point Batch Problem

Setting temperature to 0 in large language models does not guarantee deterministic outputs, as repeated identical requests can return different results. The root cause lies in floating-point arithmetic: GPU kernels reorder reduction operations based on batch shape, causing tiny numerical differences in logits that argmax then amplifies through autoregressive decoding. Because shared inference servers batch multiple users' requests together, the batch size and composition change constantly, altering internal computation paths even for byte-identical prompts. Additional factors such as mixture-of-experts routing and prefix caching further shift where compute boundaries fall, compounding the variability. Engineers are advised to test against parsed fields and behavioral invariants rather than exact output strings, since true bitwise reproducibility is only achievable on dedicated hardware running a single request at a time.