Why LLM eval score gains may be noise, and how paired bootstrap tests can gate changes

Teams often approve prompt or model changes based on small average eval score gains that turn out to be statistical noise, leading to production regressions. Eval results are affected by sampling noise in small golden sets, judge variability, and task-level factors like temperature and retrieval drift. Comparing baseline and candidate configurations on the same examples, rather than comparing raw averages, cancels out example difficulty and isolates the real change. The article explains that the required sample size depends on how often the two configurations disagree per example, and that small gains are often undetectable with a 200-example set. It proposes a simple stdlib Python paired bootstrap gate that ships a change only when the confidence interval of the difference clears zero.