How to Build a Useful AI Eval Set: Size, Statistics, and Clear Decisions

Building an effective evaluation set for AI systems requires anchoring it to a specific decision, such as whether a prompt change causes regressions or whether a cheaper model is sufficient for a task. Developers often make the mistake of building eval sets to broadly 'measure quality' rather than to answer a concrete question, which renders results unactionable. Statistical realities limit what small eval sets can reliably detect: a 50-example set scoring 90% carries a 95% confidence interval of roughly 77–94%, meaning it cannot distinguish a move from 88% to 92% as meaningful. The 'rule of three' further cautions that zero failures across 50 examples still allows for a real failure rate as high as 6%. Small sets remain valuable for catching category-wide breakdowns, and paired comparisons using McNemar's test can make them more informative when evaluating two systems on the same examples.