XGBoost Explained: How It Outperforms Gradient Boosting With Smarter Math

XGBoost is a tree-based machine learning algorithm and an advanced variant of Gradient Boosting, capable of performing calculations up to 10 times faster than its predecessor. Its effectiveness is well-documented in competitive data science, with 17 out of 29 winning Kaggle solutions relying on it. The algorithm minimizes an objective function combining a loss term and a regularization term, using a second-order Taylor expansion to make the optimization tractable across varying tree structures. To find the best splits efficiently, XGBoost employs a greedy exact method for small datasets and an approximate approach using weighted quantiles for larger ones. The Hessian values from the second-order expansion serve as weights in this quantile sketch, allowing the algorithm to handle large-scale sparse data effectively.