Why Your A/B Test Results May Be Wrong Before You Even Read Them

A technical guide published on DEV Community outlines three critical validity checks that should precede any interpretation of A/B test results. The first is an A/A test, where two identical experiences are run against each other to expose tracking bugs or randomization errors in the testing pipeline itself — Microsoft's experimentation team once found a false-positive rate of 30% this way. The second check is Sample Ratio Mismatch (SRM), which uses a chi-squared test to verify that traffic was actually split as intended, since even a 4% imbalance can signal bot skew, ID collisions, or broken redirects. The third issue flagged is Simpson's Paradox, where aggregate results can contradict segment-level findings, making the overall metric misleading without deeper breakdown. The article argues these infrastructure checks are not one-time pre-launch steps but should run continuously so pipeline regressions are caught before they corrupt live experiment data.