Tiny weight changes swung LLM benchmark scores by 1.4 points, prompting a retracted report

A team comparing 4B-class language models on 400 multiple-choice reasoning questions found that two weight files differing in only about 0.005% of elements produced scores 1.4 percentage points apart. Repeating measurements on the same file showed a much smaller spread than comparing near-identical files, meaning repeated runs understated real uncertainty. The team had earlier reported a candidate model as a 1-point improvement, but after rerunning with a matched baseline the gain disappeared, so they retracted the claim. They also found that output truncation at 8,192 tokens drove much of the ranking, with truncation rates tracking scores closely. The authors attribute run-to-run variation to batch-dependent inference kernels and long reasoning chains, and they now recommend deterministic log-probability scoring and reporting truncation rates.