Your AI eval confidence intervals are likely too narrow due to data clustering

A statistical flaw is quietly undermining many AI model evaluations: standard confidence interval formulas assume each test example is independent, but most eval datasets are not. Examples are often clustered — multiple questions from the same document, turns from the same conversation, or paraphrases of the same base case — meaning they carry overlapping rather than independent information. This clustering inflates the effective sample size, producing confidence intervals that are artificially narrow and leading teams to declare model improvements significant when they may not be. A 200-example eval drawn from 40 source documents, for instance, may be statistically worth only around 91 independent samples, making a reported margin of ±3.5 points closer to ±5.2 in reality. The recommended fix is a cluster bootstrap — resampling whole clusters rather than individual rows — which corrects for correlation without requiring manual estimation of intra-cluster statistics.