Why local LLMs give different answers at temperature 0

Setting temperature to 0 and fixing the seed does not guarantee identical outputs from a local language model. A test with llama.cpp on the gsm8k benchmark changed the generated text on 41 of 100 problems when requests were run one at a time versus eight at once. Other studies cite similar results, including 80 different completions from 1,000 identical temperature-0 requests and accuracy swings of up to 9% when only hardware or batch size changed. The root cause is that floating-point addition is not associative, so differences in GPU kernel reduction order, batching, and prompt caching alter the computed probabilities slightly. Issue trackers for several popular inference engines report the same behavior, making it a broad rather than engine-specific problem.