PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust XGBoosting for Regression

Iris Aragón Mladosich, Christophe Croux

arXiv:2608.13590v1cs.LGstat.COstat.ML

TL;DR

Classical XGBoost regression can be sensitive to vertical outliers and leverage points. This paper develops robust XGBoost variants through alternative loss functions and finds that MM-XGBoost combines strong robustness with predictive accuracy.

  • Problem

    Classical XGBoost regression can be sensitive to vertical outliers and leverage points.

  • Method

    The paper develops robust XGBoost variants by modifying the loss function used to select successive regression trees.

  • Results

    MM-XGBoost achieves both a high degree of robustness and strong predictive accuracy, while standard XGBoost remains non-robust to outliers.

  • Takeaways & Limitations

    MM-XGBoost is presented as the most attractive overall approach among the studied methods.

  • Takeaways & Limitations

    Developing robust LightGBM methods may require modifications beyond the loss function, including its histogram-based splitting mechanism.

Abstract

from arXiv · show

XGBoost is a very popular and powerful method for prediction. It iteratively fits simple decision trees to the residuals of the previous step. An efficient and scalable implementation is available. The standard loss function for XGBoost is the quadratic loss, but a Huber loss can also be used. In this paper, we study the robustness of XGBoost and show that its performance can be affected by vertical outliers and leverage points. To address this, we explore alternative loss functions, based on M-, S-, and τ -estimators from robust regression. Our results indicate that a two-step procedure, referred to as MM-XGBoost, provides the best trade-off between robustness and prediction accuracy.

1 Introduction

The paper shows that classical XGBoost regression can be sensitive to vertical outliers and leverage points, especially under quadratic loss, and develops robust alternatives by modifying its loss function. It evaluates M-, S-, MM-, and τ-based losses through simulations to assess robustness and predictive performance.

  • Background: XGBoost is an efficient and scalable gradient-boosting implementation that builds strong predictors by iteratively combining simple base learners.At each iteration, a loss function measures the discrepancy between predictions and observed responses.
  • Problem: Classical XGBoost regression is sensitive to vertical response outliers and predictor leverage points, which can reduce prediction accuracy and stability.This sensitivity is mainly observed with the commonly used quadratic loss.
  • Motivation: The smooth Huber-loss alternative does not fully resolve XGBoost’s robustness problem because leverage points can still affect the fitted model.The paper therefore modifies the loss function used in boosting to improve robustness for regression.
  • Contributions: The proposed extensions adapt robust-regression ideas by incorporating loss functions based on M-, S-, and MM-estimators into XGBoost.The authors also include τ-loss, which they state had not been considered previously in this boosting procedure.
  • Evaluation: Simulation studies evaluate whether the alternative losses improve robustness against outlying observations while maintaining good predictive performance.Custom loss functions can be incorporated into XGBoost without modifying its core algorithm.

2 XGBoost for Regression

XGBoost builds regression-tree ensembles by iteratively reducing a specified loss, using second-order approximations and regularization to control tree complexity. Standard quadratic and Pseudo-Huber losses have complementary robustness limitations, motivating alternative robust loss functions such as Tukey’s bisquare.

  • Gradient boosting: Gradient boosting sequentially adds regression trees, with each new tree chosen to reduce a specified loss relative to the current predictions.The resulting tree is added to the existing model, with shrinkage parameter η > 0 used to prevent overfitting.
  • XGBoost: XGBoost efficiently minimizes the loss through a second-order Taylor approximation, explicit regularization, derivative-based leaf weights, and greedy split selection.The regularization parameters γ and λ penalize the number of leaves and leaf weights, respectively.
  • Built-in loss functions: XGB Quadratic is optimal under Gaussian noise but highly sensitive to outliers, including both vertical outliers and leverage points.Its quadratic penalty on large residuals makes it sensitive to extreme observations.
  • Built-in loss functions: XGB Pseudo-Huber penalizes large residuals linearly to reduce their influence and improves robustness to vertical outliers, but remains highly sensitive to leverage points.Its default tuning constant is b = 1.
  • Robust alternative losses: Tukey’s bisquare is introduced as an alternative robust loss whose influence becomes constant beyond cutoff c and whose derivative redescends to zero.The tuning constant balances robustness and efficiency, with the selected constant providing 95% efficiency under Gaussian errors.

3 Robust XGBoost

Section 3 introduces robust XGBoost variants based on M-, S-, τ-, and MM-estimators. MM-XGBoost combines a robust initial S-XGBoost fit with an efficient M-loss refinement step.

  • Robust XGBoost variants: The paper proposes M-XGBoost, S-XGBoost, τ 2-XGBoost, and MM-XGBoost using loss functions inspired by robust regression estimators.M-, S-, and τ 2-XGBoost are based on M-, S-, and τ-estimators, while MM-XGBoost follows the spirit of MM-estimators.
  • Implementation: The custom losses are implemented by deriving their gradients and diagonal Hessian entries for the XGBoost second-order Taylor expansion.The Hessian is approximated by its diagonal when the full matrix becomes intractably large for n big.
  • M- and S-XGBoost: M-XGBoost uses scaled residuals with a fixed robust response scale, while S-XGBoost jointly estimates residual scale through an M-scale equation.For M-XGBoost, σ0 is computed once before boosting; S-XGBoost instead uses a scale σ implicitly defined through the residuals.
  • MM-XGBoost: MM-XGBoost uses a two-stage procedure: S-XGBoost produces robust initial predictions, then an M-loss refinement uses the initial fit and its robust scale estimate.The refinement uses Tukey’s bisquare with c = 4.685 and applies XGBoost for K2 boosting iterations.

4 Simulation Study

The simulation study evaluates XGBoost variants under clean data and multiple contamination scenarios, including vertical outliers and leverage points. MM-XGBoost consistently provides the best overall balance between robustness and predictive accuracy.

  • Simulation results: XGB Quadratic is sensitive to vertical outliers and leverage points, showing increased prediction error and variability relative to clean data.The same instability extends across the simulation configurations.
  • Simulation results: XGB Pseudo-Huber and M-XGBoost improve robustness to vertical outliers but remain sensitive to leverage points.This pattern is reported across the representative settings and the broader simulation configurations.
  • Simulation results: S-XGBoost is robust across contamination scenarios with stable RMSE values, but its clean-data prediction error is higher than that of baseline methods.τ 2-XGBoost improves clean-data efficiency relative to S-XGBoost but shows somewhat reduced robustness under contamination.
  • Simulation results: MM-XGBoost achieves the best overall trade-off, combining low prediction error with strong, stable robustness across contamination scenarios and simulation configurations.Its performance remains stable under vertical outliers, leverage points, strong predictor correlation, high dimensionality, and substantial outlier contamination, while clean-data efficiency loss versus standard XGBoost is limited.

5 Conclusion

The paper proposes robust XGBoost regression methods that preserve XGBoost’s computational efficiency and scalability while improving robustness to outliers. Among the methods evaluated, MM-XGBoost offers the strongest overall combination of robustness and predictive accuracy, with practical computational advantages over related robust gradient boosting methods.

  • Implementation: The proposed robust methods are straightforward to implement in XGBoost by specifying mathematically derived loss gradients and diagonal Hessians.They modify only the loss function while preserving the core XGBoost algorithm.
  • Simulation results: Standard XGBoost is not robust to outliers, including when using its built-in Pseudo-Huber loss, whereas MM-XGBoost combines high robustness with strong predictive accuracy.The methods were evaluated through extensive simulations covering a wide range of outlier scenarios.
  • Comparison with robust gradient boosting: SBoost and RRBoost required up to one hundred times more computation time than S-XGBoost and MM-XGBoost, respectively, despite broadly comparable predictive performance.MM-XGBoost tended to slightly outperform RRBoost in predictive performance.
  • Limitations and future work: The robust loss functions can be incorporated into LightGBM, but exploratory analyses found LightGBM considerably more vulnerable to outliers than XGBoost even with robust losses.The authors suggest robustness may require changes beyond the loss function, potentially involving histogram-based splitting, and leave robust LightGBM methods for future work.
  • Conclusion: Overall, the proposed robust XGBoost extension substantially improves outlier robustness while retaining the computational efficiency and scalability of XGBoost.The authors hope the work will stimulate further research on robust machine learning.

Appendix Proofs · Proof of Result 1

The proof derives the gradient and diagonal Hessian entries for the M-loss by differentiating with respect to ŷ_j. It uses the Kronecker delta to simplify the gradient expression before differentiating again.

  • Appendix Proofs · Proof of Result 1: The proof begins by taking the partial derivative of the M-loss in Equation (10) with respect to ŷ_j.
  • Appendix Proofs · Proof of Result 1: The Kronecker delta δ_ij appears in the derivative expression and is used to simplify the resulting gradient entry g_j.
  • Appendix Proofs · Proof of Result 1: Together, the derivations establish the gradient and diagonal Hessian components required by Result 1.
  • Appendix Proofs · Proof of Result 1: The proof then differentiates the gradient entry g_j with respect to ŷ_j once more.
  • Appendix Proofs · Proof of Result 1: This second differentiation produces the diagonal Hessian entry h_j associated with Equation (11) in Result 1.

Proof of Result 2

The proof of Result 2 derives the gradient and diagonal Hessian entries for the S-loss by implicitly differentiating the M-scale estimator and its standardized residuals. Substitution and simplification recover the stated gradient and Hessian formulas.

  • Gradient: The M-scale estimator σ is defined implicitly, and differentiating its equation with respect to each fitted value yields its first derivative.The derivation uses the standardized residuals ˜r_i = r_i/σ and substitutes their derivatives into the differentiated estimating equation.
  • Gradient: Substituting the first derivative of σ into the S-loss derivative yields gradient entry (14) in Result 2.
  • Hessian: Differentiating the gradient entry once more with respect to each fitted value produces the diagonal Hessian entry.The second derivative of σ is obtained by repeatedly differentiating its implicit identity, evaluating component terms, and simplifying the result.
  • Hessian: Substituting the first and second derivatives of σ into the differentiated gradient yields diagonal Hessian entry (15) in Result 2.The final expression is written using A1 and B1 as defined in the preceding equations.

Proof of Result 3

The proof derives the gradient and diagonal Hessian entries of the τ2-loss by differentiating the τ-scale estimator and substituting the resulting expressions into the loss derivatives.

  • Gradient: The τ2-loss derivative is obtained by applying the chain rule because the loss depends on ŷ_j through τ2.The derivation considers j = 1, . . . , n.
  • Gradient: Differentiating the τ-scale estimator accounts for its direct dependence on ŷ_j and its indirect dependence through the M-scale estimator σ.Substituting derivatives of r̃_i and σ, followed by reordering, produces the first derivative of the τ-scale estimator.
  • Hessian: Substituting the first derivative of the τ-scale estimator into the loss derivative yields gradient entry (18) in Result 3.The proof then differentiates this gradient entry once more with respect to ŷ_j to obtain the diagonal Hessian entry.
  • Hessian: The second derivative of τ2 is derived by differentiating its first-derivative identity and evaluating the resulting terms separately.The proof expands terms, computes ∂C/∂ŷ_j using the derivative of r̃_i, and rearranges the expressions before combining them.
  • Hessian: Inserting the evaluated terms and substituting the relevant derivatives yields diagonal Hessian entry (19) in Result 3.The final substitution uses equations (25) and (29).

Supplementary Material · Results for p = 10 under clean and 5% contamination

The supplementary material reports mean RMSE (SE) for g1, g2, and g3 with p = 10 under clean and 5% contamination across scenarios S1, S2, and S3, based on M = 100 independent runs.

  • Results for p = 10 under clean and 5% contamination: Tables 3–5 report mean RMSE (SE) for g1, g2, and g3 under S1 with p = 10 in clean and 5% contamination scenarios.These results are based on M = 100 independent runs.
  • Supplementary Material: Table 3 specifically covers g1 under S1 with p = 10 across clean and 5% contamination scenarios.It reports mean RMSE (SE) from M = 100 independent runs.
  • Results for p = 10 under clean and 5% contamination: Tables 6–8 report mean RMSE (SE) for g1, g2, and g3 under S2 with p = 10 in clean and 5% contamination scenarios.These results are based on M = 100 independent runs.
  • Supplementary Material: Table 7 specifically covers g2 under S2 with p = 10 across clean and 5% contamination scenarios.It reports mean RMSE (SE) from M = 100 independent runs.
  • Results for p = 10 under clean and 5% contamination: Tables 9–11 report mean RMSE (SE) for g1, g2, and g3 under S3 with p = 10 in clean and 5% contamination scenarios.These results are based on M = 100 independent runs.
  • Supplementary Material: Table 11 specifically covers g3 under S3 with p = 10 across clean and 5% contamination scenarios.It reports mean RMSE (SE) from M = 100 independent runs.

Results for p = 10 under clean and 20% contamination … Results for p = 400 and n = 300 for the representative settings

The reported results comprise mean RMSE (SE) from 100 independent runs across p = 10, p = 100, and representative p = 400 settings under clean and contamination scenarios. The tables cover functions g1, g2, and g3 across S1, S2, and S3 settings, with the p = 400 cases using ntest = 300 and ntrain = 300.

  • Results for p = 10 under clean and 20% contamination: For p = 10, Tables 12–14 report mean RMSE (SE) for g1, g2, and g3 under clean and 20% contamination with S1.Each table uses M = 100 independent runs.
  • Results for p = 10 under clean and 20% contamination: For p = 10 with S2, Tables 15–17 report mean RMSE (SE) for g1, g2, and g3 under clean and 20% contamination.The results are based on M = 100 independent runs.
  • Results for p = 10 under clean and 20% contamination: For p = 10 with S3, Tables 18–20 report mean RMSE (SE) for g1, g2, and g3 under clean and 20% contamination.The tables summarize M = 100 independent runs.
  • Results for p = 100 under clean and 5% contamination: For p = 100 under clean and 5% contamination, Tables 21–29 cover g1, g2, and g3 across S1, S2, and S3.All tables report mean RMSE (SE) for M = 100 independent runs.
  • Results for p = 100 under clean and 5% contamination: For p = 100 under clean and 5% contamination with S3, Tables 27–29 separately report mean RMSE (SE) for g1, g2, and g3.These results also use M = 100 independent runs.
  • Results for p = 400 and n = 300 for the representative settings: For p = 400 and n = 300, Table 39 reports g2 under clean and 5% contamination with S3, while Table 40 reports g3 under clean and 20% contamination with S2.Both tables use M = 100 independent runs with ntest = 300 and ntrain = 300.
Loading 2608.13590v1…