PaperlayerView exact revision on arXiv

Source-linked AI summary

Imbalance-XGBoost: Leveraging Weighted and Focal Losses for Binary Label-Imbalanced Classification with XGBoost

Chen Wang, Chengyuan Deng, Suzhen Wang

arXiv:1908.01672v2cs.LGstat.ML

TL;DR

Binary label imbalance can make accuracy misleading and has produced mixed evidence about XGBoost's effectiveness. The paper introduces Imbalance-XGBoost, integrating weighted and focal losses with XGBoost and deriving the required derivatives. Across the reported experiments, focal-XGBoost generally achieved the highest F1 score, while both loss-based variants improved minority-sensitive performance relative to accuracy-focused baselines.

  • Problem

    Label-imbalanced binary classification is common, but accuracy-driven methods can overlook minority cases, and prior reports on XGBoost for skewed data are mixed.

  • Method

    The paper develops a Python package that integrates weighted cross-entropy and focal losses into XGBoost and derives their first- and second-order prediction derivatives.

  • Results

    Focal-XGBoost generally obtained the highest F1 score; weighted- and focal-XGBoost produced significantly higher F1 scores despite slight accuracy declines, and both outperformed the best classifier in by a large margin.

  • Takeaways & Limitations

    The package combines XGBoost with label-imbalance-robust losses and reports competitive performances across five imbalanced classification datasets.

  • Takeaways & Limitations

    Fair evaluation after parameter selection requires a newly instantiated booster with random initialization rather than direct reuse of the fitted optimal booster.

Abstract

from arXiv · show

The paper presents Imbalance-XGBoost, a Python package that combines the powerful XGBoost software with weighted and focal losses to tackle binary label-imbalanced classification tasks. Though a small-scale program in terms of size, the package is, to the best of the authors' knowledge, the first of its kind which provides an integrated implementation for the two losses on XGBoost and brings a general-purpose extension on XGBoost for label-imbalanced scenarios. In this paper, the design and usage of the package are described with exemplar code listings, and its convenience to be integrated into Python-driven Machine Learning projects is illustrated. Furthermore, as the first- and second-order derivatives of the loss functions are essential for the implementations, the algebraic derivation is discussed and it can be deemed as a separate algorithmic contribution. The performances of the algorithms implemented in the package are empirically evaluated on Parkinson's disease classification data set, and multiple state-of-the-art performances have been observed. Given the scalable nature of XGBoost, the package has great potentials to be applied to real-life binary classification tasks, which are usually of large-scale and label-imbalanced.

1 Introduction

Imbalance-XGBoost extends XGBoost for binary label-imbalanced classification by integrating weighted and focal losses. The paper derives the required loss-function derivatives and motivates the package through inconsistent prior findings and the risks of misleading accuracy.

  • Motivation: XGBoost is widely used for large-scale machine learning, but its performance can become subtle on label-imbalanced classification tasks.The paper notes applications spanning cancer diagnosis, medical records, credit risk, and metagenomics.
  • Motivation: 95% accuracy can result from predicting every patient as cancer-free when only 5% have cancer, while missing cancer cases can be fatal.This example illustrates why accuracy alone can be misleading under label imbalance.
  • Motivation: Prior studies report mixed evidence about XGBoost on skewed datasets, including both superior performance and the need for additional ensembling.The paper identifies label-skewed data as common in practice and treats the resulting performance uncertainty as consequential.
  • Package contribution: Imbalance-XGBoost implements weighted cross-entropy and focal losses on XGBoost for binary label-imbalanced classification.Focal loss reduces the importance of well-classified points through a (1 − y_j)^γ factor, whereas weighted cross-entropy increases penalization for misclassifying a selected class.
  • Algorithmic contribution: The paper derives first- and second-order loss derivatives required to integrate both losses with XGBoost's incremental learning objective.The authors present these derivative derivations and implementations as an algebraic contribution enabling the two losses to run within XGBoost.
  • Implementation: The package is a small Python implementation built on XGBoost, NumPy, and Scikit-learn, with core methods spanning only a few hundred lines of code.The authors distinguish the program's small size from the non-triviality of its derivatives, implementations, and practical significance.

2 Design and Usage of Imbalance-XGBoost

Imbalance-XGBoost is designed as a Scikit-learn-compatible Python estimator with tunable loss classes and evaluation utilities. Its usage supports parameter search, cross-validation, model export, and metrics suited to imbalanced classification.

  • Code Design: The package uses separate Weight Binary Cross Entropy and Focal Binary Loss classes, while users apply the main imbalance xgboost class.The separate loss classes support parameter tuning and are not intended to be called directly by users.
  • Code Design: Figure 1 is identified as the overall structure of the program.The surrounding design description organizes the program around one main estimator and two customized-loss classes.
  • Model Optimization and Evaluation with Scikit-learn: Scikit-learn estimator compatibility enables GridsearchCV() and RandomizedSearchCV() to search parameters for imbalanced XGBoost models.The package is designed to integrate with Scikit-learn model and parameter-selection methods.
  • Model Optimization and Evaluation with Scikit-learn: After fitting, users can retrieve the plain XGBoost booster, enabling a model trained with Imbalance-XGBoost to be saved and used as an XGBoost class.The passage describes this as a way to separate training with the package from later model use.
  • Model Optimization and Evaluation with Scikit-learn: For fair evaluation after parameter selection, the package recommends instantiating a new booster with random initialization rather than directly reusing the fitted GridsearchCV() booster.The recommendation follows from XGBoost's iterative training process and the need for a randomized state.
  • Built-in Evaluation Score: Accuracy alone can be unreliable under label imbalance because predicting every instance as the majority class may produce high accuracy without useful classification.The package therefore provides precision, recall, F1 score, and Matthews Correlation Coefficient through score eval func().

3 Theories and Derivatives

This section develops the second-order framework and derivative formulas needed to integrate weighted cross-entropy and focal losses into XGBoost. It defines the losses, their parameters, and the gradient and hessian calculations used by the boosting procedure.

  • Second-order Approximation of Gradient Boosting Tree: The derivatives are taken with respect to raw predictions z_i, with probabilistic predictions defined as ŷ_i = σ(z_i).The sigmoid derivative is ∂ŷ/∂z = ŷ(1 − ŷ), and both implemented losses use sigmoid activation.
  • Second-order Approximation of Gradient Boosting Tree: XGBoost fits trees additively using a regularized objective and a second-order Taylor approximation.The approximation uses gradient and hessian terms for optimizing the tree added at each training iteration.
  • Weighted Cross-entropy Loss: Weighted cross-entropy introduces an imbalance parameter α that changes the penalty assigned to errors involving the two classes.When α > 1, misclassifying class 1 as class 0 receives extra loss; when α < 1, the weighting emphasizes correctly identifying label 0.
  • Weighted Cross-entropy Loss: The weighted loss requires first- and second-order derivatives so it can be optimized within XGBoost’s customized loss framework.The first derivative resembles ordinary cross-entropy but includes an α^y_i factor, followed by a corresponding second derivative.
  • Focal Loss: Focal loss is parameterized by γ and reduces to ordinary cross-entropy when γ = 0.The section derives both focal-loss gradients and hessians, using shorthand variables to simplify the expressions.
  • Focal Loss: When γ = 0, the focal-loss hessian becomes η1 = ŷ_i(1 − ŷ_i), matching the ordinary cross-entropy hessian.This provides a consistency check for the derived second-order expression.

4 Related Work

Prior work on imbalanced XGBoost classification has primarily used data-level resampling, while algorithm-level loss modifications have been comparatively limited. The section positions the paper’s approach among alternative boosting implementations and related software.

  • Gradient-Boosting Implementations: Related gradient-boosting implementations include pGBRT, LightGBM, and CatBoost, which can outperform XGBoost on some specific problems.Despite these alternatives, the section states that XGBoost remains widely preferred in the data-science community.
  • XGBoost and Imbalanced Classification: Earlier XGBoost applications to label-imbalanced data mainly apply resampling methods before model training.Examples include breast-cancer classification with common resampling techniques and credit scoring with BalanceCascade under-sampling.
  • XGBoost and Imbalanced Classification: Algorithm-level modifications for imbalanced XGBoost have included sigmoid-activation changes and a limited weighted-function implementation.The cited sigmoid approach leaves the loss function unchanged, while details of the weighted XGBoost implementation were not presented.
  • Alternative Implementations: The paper distinguishes its approach from TensorFlow Boosted Trees, which can run these losses through automatic differentiation.That implementation is described as less popular and lacking support for large-scale machine learning and Scikit-learn compatibility.
  • Imbalanced-Classification Software: Imbalanced-learn provides an integrated Python package for data-level resampling, with counterparts such as ROSE in R.These tools address imbalanced classification primarily through data-level methods rather than the loss integration developed here.

5 Experiments

The experiments evaluate Weighted-XGBoost and Focal-XGBoost on Parkinson’s disease data and four increasingly imbalanced UCI datasets. Both losses improve imbalance-sensitive metrics, with focal loss generally producing the strongest F1 performance.

  • Experimental setup: The study compares Weighted-XGBoost and Focal-XGBoost with vanilla XGBoost on Parkinson’s disease data and four imbalanced UCI binary-classification datasets.The UCI imbalance ratios range from 9:1 to 42:1, while the Parkinson’s dataset has a ratio of roughly 3:1.
  • Parkinson’s disease classification: Parkinson’s disease data contain 757 features from 188 patients and 64 healthy individuals, evaluated with leave-one-object-out cross validation.Each individual contributes three records, producing the approximately 3:1 label imbalance.
  • Parkinson’s disease classification: Weighted-XGBoost and Focal-XGBoost generate significantly higher F1 scores than prior classifiers despite slight accuracy declines.The authors interpret the accuracy–F1 trade-off as evidence that earlier high accuracy could result from overlooking the minority class.
  • Feature-group and top-50 comparisons: Focal-XGBoost obtains the highest F1 score for almost all Parkinson’s feature groups and slightly exceeds Weighted-XGBoost on the top-50 feature subset.The paper attributes this pattern to focal loss being more robust to parameters, whereas weighted loss is more affected by sub-optimal parameter choices.
  • Feature-group and top-50 comparisons: On the top-50 features, both weighted- and focal-XGBoost outperform the best classifier reported in by a large margin.The top-50 subset is treated as a master subset for corroborating the package’s performance advantage.
  • Benchmark imbalanced datasets: Across the four benchmark datasets, weighted- and focal-XGBoost remain at least as accurate as vanilla XGBoost while substantially improving F1 score and MCC.The improvements in F1 score and MCC become more significant as the imbalance ratio increases.

6 Conclusion

The paper presents Imbalance-XGBoost, a Python package integrating weighted cross-entropy and focal losses with XGBoost for binary label-imbalanced classification. It also contributes derivative theory and competitive results on five benchmark datasets.

  • Imbalance-XGBoost integrates weighted cross-entropy and focal loss functions with XGBoost for binary label-imbalanced classification.
  • Experiments on five benchmark datasets report competitive performances while emphasizing their imbalanced nature.
  • The paper derives the second-order approximations and essential derivatives required to apply these losses within XGBoost.The merged equations are also described as convenient for vectorization and potentially applicable to other machine-learning fields.

Supplementary Materials

The supplementary materials provide code listings showing basic use of Imbalance-XGBoost and parameter tuning with Scikit-learn tools. The examples cover weighted and focal objectives, cross-validation, and model evaluation.

  • Basic Usage of Imbalance-XGBoost: The basic usage listing initializes weighted and focal Imbalance-XGBoost models and fits them with imbalance-specific parameters.The examples use imbalance alpha=2.0 for weighted XGBoost and focal gamma=2.0 for focal XGBoost.
  • Parameter Tuning and Model Evaluation: The second listing tunes focal gamma with GridSearchCV before retrieving the best estimator and parameters.The candidate focal gamma values are 1.0, 1.5, 2.0, and 3.0.
  • Parameter Tuning and Model Evaluation: The evaluation example applies the tuned focal model with Leave-One-Out cross-validation through Scikit-learn.
Loading 1908.01672v2…