Source-linked AI summary
A Comparative Analysis of XGBoost
Candice Bentéjac, Anna Csörgő, Gonzalo Martínez-Muñoz
TL;DR
The paper investigates whether XGBoost is superior to gradient boosting and random forest across classification tasks, focusing on accuracy, training speed, and parameter tuning. It empirically compares tuned and default models and analyzes XGBoost’s tuning process. Gradient boosting achieves the most accurate results by dataset count, while XGBoost trains fastest and requires careful tuning for accuracy.
Problem
The study addresses limited comprehensive comparisons involving gradient-boosting methods and asks how XGBoost compares with gradient boosting and random forest in accuracy, training speed, and tuning.
Method
The paper empirically compares tuned and default XGBoost, gradient boosting, and random-forest models across a wide range of classification tasks, while analyzing XGBoost parameter tuning.
Results
Gradient boosting achieves the best performance on 10 of 28 datasets, while tuned XGBoost leads on 8; XGBoost trains about 3.5 times faster than random forest and 2.4–4.3 times faster than gradient boosting.
Takeaways & Limitations
Accurate gradient-boosting and XGBoost models require meticulous parameter search, whereas random forest performs slightly better on average with default parameters.
Takeaways & Limitations
The conclusions are based on grid-search tuning with within-train 10-fold cross-validation, and tuning contributed over 99.9% of gradient boosting or XGBoost training effort.
Abstract
from arXiv · showhide
XGBoost is a scalable ensemble technique based on gradient boosting that has demonstrated to be a reliable and efficient machine learning challenge solver. This work proposes a practical analysis of how this novel technique works in terms of training speed, generalization performance and parameter setup. In addition, a comprehensive comparison between XGBoost, random forests and gradient boosting has been performed using carefully tuned models as well as using the default settings. The results of this comparison may indicate that XGBoost is not necessarily the best choice under all circumstances. Finally an extensive analysis of XGBoost parametrization tuning process is carried out.
1 Introduction
The study situates XGBoost among ensemble methods motivated by the need to handle complex, large-scale data efficiently. It compares XGBoost with gradient boosting and random forest across accuracy, training speed, and parameter tuning.
- Ensemble methods improve existing models through randomization or adaptive emphasis procedures such as boosting.
- XGBoost is a recent gradient-boosting ensemble method that has achieved competitive results across applications and Kaggle competitions.
- The study addresses a comparison gap by evaluating gradient boosting and XGBoost alongside random forest, a benchmark in prior studies.
- The comparison examines accuracy and training speed using XGBoost, gradient boosting, and random forest.
- The paper also analyzes XGBoost parameter setting to support more effective tuning.
2 Methodology
The methodology describes random forest, gradient boosting, and XGBoost as tree-based ensemble approaches with distinct construction, regularization, randomization, and tuning strategies. It emphasizes parameters affecting model complexity, sampling, feature selection, and optimization.
- 2.1 Random forest: Random forest trains decision trees on bootstrap samples and uses random attribute subsets at each node split, combining classification outputs by majority voting.Bootstrap samples contain approximately 37% duplicated instances, and Breiman’s proposed feature count is floor(log2(#features)+1).
- 2.1 Random forest: Random forest generalization error converges as the number of trees grows, so ensemble size generally need not be tuned.The default setting often performs strongly, although this also limits gains from parameter tuning.
- 2.1 Random forest: The study evaluates random-forest controls for feature sampling, split size, leaf size, and maximum tree depth.These parameters regulate tree size and which split candidates remain available.
- 2.2 Gradient boosting: Gradient boosting builds an additive approximation by iteratively fitting models to pseudo-residuals and selecting function weights through line search.Each model acts as a greedy gradient-descent step toward minimizing the loss.
- 2.3 XGBoost: XGBoost is a scalable decision-tree ensemble based on gradient boosting that controls tree complexity through a modified loss function.Its tuned parameters include learning rate, gamma, maximum depth, feature subsampling, and subsampling rate.
- 2.3 XGBoost: XGBoost reduces over-fitting and training cost through random subsampling, column subsampling, complexity penalties, and shrinkage.Higher gamma values produce simpler trees; reduced complexity also lowers storage requirements.
- 2.3 XGBoost: XGBoost accelerates tree construction by reducing the computational cost of finding split points and using sparsity-aware handling of missing values.
3 Experimental results
Across 28 UCI datasets, the study compares tuned and default random forest, gradient boosting, and XGBoost models, evaluating accuracy, ranks, training time, and parameter settings. Tuned gradient boosting wins on the most datasets, while XGBoost is fastest per fitted ensemble and parameter choices materially affect results.
- Experimental setup: The evaluation uses 28 diverse UCI datasets with stratified 10-fold cross-validation and nested grid search for each method.Default and tuned ensembles are also evaluated on held-out test sets.
- Accuracy comparison: Tuned gradient boosting achieves the highest accuracy on 10 of 28 datasets, followed by tuned XGBoost on 8.Default gradient boosting leads on 5 datasets, while tuned and default random forest lead on 4 each and default XGBoost on 3.
- Accuracy comparison: Random forest changes little with tuning, with tuned-versus-default differences below 0.5% on 18 of 28 datasets, whereas default XGBoost and gradient boosting generally trail tuned versions.Default XGBoost can nevertheless outperform its tuned version on noisy datasets when its default settings are already strong.
- Accuracy comparison: No statistically significant average-rank differences appear among the six methods, although tuned XGBoost has the best average rank.The ranking analysis uses a Nemenyi test across the 28 datasets.
- Training time: 3.5 times faster is the average single-ensemble training advantage of XGBoost over random forest, while its advantage over gradient boosting ranges from 2.4 to 4.3 times.These comparisons exclude grid-search time; XGBoost also takes less than half as long as gradient boosting for grid search despite using twice the grid size.
4 Conclusion
The study finds that gradient boosting achieved the most dataset-level wins, while XGBoost’s main practical advantages arise from careful tuning and a computationally efficient search strategy.
- Gradient boosting achieved the highest accuracy on the largest number of investigated problems.It was the best-performing classifier by dataset-level win count.
- XGBoost and gradient boosting were the least successful when trained with package default parameters.The findings support meticulous parameter search for accurate gradient-boosting models.
- Fixing XGBoost’s subsampling rate at 0.75 and selected features at sqrt reduced the parameter-grid size 16-fold while improving average performance.
- Over 99.9% of gradient boosting and XGBoost training effort was spent in grid-search tuning under within-train 10-fold cross-validation.The proposed smaller XGBoost grid can dramatically reduce grid-search time.
- XGBoost’s competition advantages are linked to fine parameter tuning and computational efficiency, whereas gradient boosting requires longer computational times.The paper contrasts these practical properties with random forest, where tuning yields small gains or none.