How Gradient Boosting Works: A Step-by-Step Walkthrough With Real Numbers

Gradient boosting is a machine learning technique that builds an additive model by repeatedly fitting small decision trees to the prediction errors of all previous trees. Each new tree corrects the residual mistakes of the running prediction, and only a small fraction of its output is added at each step. A worked example using six data points shows total squared error dropping from 810 to 40.5 across just three rounds of boosting. Key tuning parameters include learning rate, tree depth, and subsampling, which together control how fast the model learns and how well it generalises. Unlike random forests, where trees independently predict the target and are averaged, gradient boosting builds trees sequentially, each one refining the cumulative sum of all prior corrections.