Inference Backends Silently Affect LLM Output Reproducibility, Study Finds

A research paper highlights that the choice of inference backend — such as Ollama, vLLM, or LMDeploy — can meaningfully alter large language model outputs beyond what random seeds control. Factors like automatically appended system prompts, default generation penalties, and prefix caching mechanisms introduce subtle but measurable differences in model behaviour. Technical variations including accumulation precision during matrix multiplication and kernel-level tie-breaking during greedy decoding can cause divergent text generation across engines. The study found that fixing a random seed alone is insufficient to guarantee reproducibility, as the entire inference stack effectively forms part of the model's runtime environment. Researchers recommend documenting the inference engine, model format, hardware configuration, precision settings, and prompt templates alongside evaluation results to enable fair and reproducible comparisons.