How Random Forests Cut Variance: The Math Behind Bagging and Tree Averaging

Random forests reduce prediction variance by averaging many deep, unpruned decision trees, each trained on a different bootstrap sample of the data. Random feature selection at every split ensures trees remain diverse, preventing a single dominant feature from making all trees look alike. A mathematical identity shows that the mean squared error of an ensemble always equals the average individual tree error minus the spread among trees, explaining why diversity directly drives accuracy gains. Each tree leaves out roughly 37% of training rows, enabling out-of-bag error estimation as a free, honest validation method without a separate holdout set. Unlike boosting, adding more trees to a random forest converges to an error floor and cannot overfit, making the two methods fundamentally different in how they use depth, data, and sequential dependence.