Why Your AI Eval Set Is Probably Too Small to Detect Real Regressions

Small evaluation sets in AI development can create a false sense of progress, as changes that appear meaningful may fall entirely within statistical noise. To reliably detect a drop in pass rate from 0.90 to 0.85 with 80% statistical power, an eval set needs roughly 253 examples. A 50-example set carries only about 35% power, meaning it will miss that regression approximately two out of three times. Most teams implicitly choose effect sizes and let sample sizes be determined by whatever data is available, rather than sizing the study to detect the smallest meaningful change. Experts recommend reporting confidence intervals instead of bare point deltas and using per-criterion binary labels to make eval results more statistically reliable.