Why Sample Size Determines What Your Data Can Actually Prove

A statistical analysis piece published on DEV Community explains the critical difference between two types of data questions: whether a test detects a known failure at all, and whether a measured performance gap is real or just noise. The article argues that some results, such as a fraud filter returning zero hits across confirmed fraud cases, are structural facts requiring no significance testing, while subtler comparisons between two systems demand proper statistical power analysis. Statistical power, defined as the probability of detecting a real effect when one exists, depends on sample size, effect size, and significance threshold, with the field's standard target set at 80%. According to Cohen's 1988 conventions, detecting even a medium-sized effect at 80% power requires roughly 64 observations per group, while small effects demand around 400. The piece cautions that skipping these calculations risks mistaking random variance for genuine performance differences, a problem equally relevant in machine learning evaluation and financial strategy testing.