Why a 90% vs 92% Robot Policy Win Rate Means Less Than You Think

A technical analysis published on DEV Community warns that comparing robot control policies using raw success-rate percentages alone is statistically unreliable. Without reporting sample sizes, confidence intervals, and statistical power, a difference such as 90% versus 92% cannot be taken as proof that one policy outperforms another. The RoboLab v4 benchmark highlights the problem: with only 10 episodes per task, a 90% success rate carries a 95% confidence interval spanning roughly 19 percentage points. The article recommends reporting raw counts in k/n form and using methods like Clopper-Pearson intervals and McNemar's paired testing for valid comparisons. Resolving a true 2-percentage-point difference near the 90% success level typically requires thousands of roll-out trials, far beyond what most benchmarks currently provide.