Simple formula can reveal whether an ad experiment is doomed before it starts

Most inconclusive ad experiments fail not because the idea was bad, but because the test never had enough statistical power to detect a meaningful result. A minimum detectable effect (MDE) formula shows that at a 3% conversion rate, even 20,000 clicks per arm can only reliably detect lifts of 16% or more — far above typical campaign improvements. Real account data compounds the problem, as analysis of 31 advertiser accounts totalling roughly $133 million in spend found that 62% of conversions come from just the top 1% of search terms, shrinking the effective sample size well below raw click counts. Resampling search terms rather than individual clicks provides a more accurate standard error, and the resulting 'design effect' can double the required sample size. Marketers can use these two checks before launching any experiment to determine whether their traffic volume can actually support the conclusion they are hoping to reach.