Source-linked AI summary
The harm of class imbalance corrections for risk prediction models: illustration and simulation using logistic regression
Ruben van den Goorbergh, Maarten van Smeden, Dirk Timmerman, Ben Van Calster
TL;DR
Class imbalance corrections may affect the reliability of risk predictions, so the study tested their impact in standard and ridge logistic regression. Using simulations and an ovarian-cancer case study, it found poorer calibration without better AUROC, while threshold shifting gave similar classification changes.
Problem
The study asks how class-imbalance methods affect calibration and prediction performance when developing models for imbalanced outcomes.
Method
The study compared standard and ridge logistic regression trained on unadjusted data or data corrected with RUS, ROS, and SMOTE, using simulations and an ovarian-cancer case study.
Results
Imbalance correction did not improve AUROC but produced strong overestimation of minority-class probabilities.
Takeaways & Limitations
Outcome imbalance is not a problem in itself, and imbalance correction may worsen prediction-model performance.
Takeaways & Limitations
The study did not systematically examine typical imbalance-correction methods, and its simulations were limited.
Abstract
from arXiv · showhide
Methods to correct class imbalance, i.e. imbalance between the frequency of outcome events and non-events, are receiving increasing interest for developing prediction models. We examined the effect of imbalance correction on the performance of standard and penalized (ridge) logistic regression models in terms of discrimination, calibration, and classification. We examined random undersampling, random oversampling and SMOTE using Monte Carlo simulations and a case study on ovarian cancer diagnosis. The results indicated that all imbalance correction methods led to poor calibration (strong overestimation of the probability to belong to the minority class), but not to better discrimination in terms of the area under the receiver operating characteristic curve. Imbalance correction improved classification in terms of sensitivity and specificity, but similar results were obtained by shifting the probability threshold instead. Our study shows that outcome imbalance is not a problem in itself, and that imbalance correction may even worsen model performance.
Introduction
Class imbalance can make accuracy misleading because majority-class performance dominates, but it is not inherently a problem for prediction-model development. Resampling methods may distort probability calibration, whereas changing the classification threshold can address classification decisions while reflecting clinical misclassification costs.
- Proposed corrections: Random undersampling, random oversampling, and SMOTE create artificially balanced training datasets for model development.These are presented as commonly suggested approaches to address class imbalance.
- Motivation: 99% accuracy can result from classifying everyone as low risk when the event fraction is 1%.With a balanced 50% event fraction, the same classifier achieves 50% accuracy.
- Motivation: Class imbalance is argued to be specific to classification accuracy rather than a pervasive problem for prediction-model development.The limitation of accuracy as a predictive-performance measure is well known.
- Alternative to resampling: Adjusting the probability threshold can provide adequate classification performance without requiring a threshold of 0.5.The threshold should reflect clinical considerations and the acceptable costs of false positives and false negatives.
- Study objectives: The study asks how class-imbalance methods affect calibration while investigating standard and penalized logistic-regression models.Its hypotheses concern overestimated probabilities after correction and comparable sensitivity–specificity effects from threshold shifting.
Methods
The study compared unadjusted and imbalance-corrected training data for standard and ridge logistic regression using simulations and an ovarian-cancer diagnosis case study. Performance was assessed out of sample across discrimination, calibration, classification, and clinical utility.
- Model development: The study compared unadjusted data with RUS, ROS, and SMOTE for fitting standard and ridge logistic-regression models.This produced four training datasets and eight model combinations per simulation scenario.
- Case study: The ovarian-cancer case study used 3,369 premenopausal patients, including 658 malignancy events and a 20% prevalence.Patients were recruited across three waves of the IOTA study, and the dataset was split 4:1 into training and test sets.
- Case study: The case-study models used age, lesion diameter, and number of papillary structures, with restricted cubic splines for continuous predictors.Models were evaluated on a held-out test set.
- Simulation design: Simulations varied training-set size, predictor count, and event fraction across 24 scenarios.Training sizes were 2,500 or 5,000; predictor counts were 3, 6, 12, or 24; event fractions were 0.3, 0.1, or 0.01.
- Evaluation: Each simulation scenario used 2,000 runs and a single test set of N = 100,000, with performance assessed by AUROC, calibration, classification, and Net Benefit.The primary metric was the calibration intercept; recalibration was also applied to models trained after imbalance correction.
Results
Across simulations and the ovarian-cancer case study, imbalance correction did not improve discrimination but substantially worsened calibration. It produced more balanced sensitivity and specificity, although threshold shifting achieved similar classification results, while clinical utility could decline markedly.
- Discrimination: There was little variation in discrimination across algorithms and imbalance-correction methods.In simulations, unadjusted models were never lower in median AUROC than corrected models; RUS was often lower, especially with lower event fractions, smaller samples, and more predictors.
- Calibration: Calibration intercepts were 0.06 for SLR and 0.05 for Ridge without correction, versus -1.32 after SMOTE and -1.50 after RUS for SLR.The corrected models therefore produced substantially overestimated minority-class probabilities.
- Clinical utility: Imbalance-corrected models had strongly reduced Net Benefit, with negative Net Benefit at thresholds of 0.3 or higher for treatment selection.The probability overestimation could select too many individuals for treatment or referral.
- Calibration: Simulation median calibration intercepts reached -4.5 or lower at a 1% event fraction, -2.1 or lower at 10%, and -0.7 or lower at 30% after correction.Recalibration improved median intercepts to between -0.07 and 0.03 using the original imbalanced data.
- Classification: At a 1% event fraction, unadjusted models using a 0.5 threshold had median sensitivity of 0% and specificity of 100%.More balanced classification was obtained through imbalance correction or shifting the probability threshold.
Discussion
For standard and ridge logistic regression, imbalance correction produced inaccurate probability estimates without improving AUROC, while threshold shifting offered similar classification benefits. The authors therefore caution that class imbalance is not inherently problematic and that correction can worsen performance.
- Classification: Improved classification from imbalance correction could also be obtained by shifting the probability threshold on models trained with unadjusted data.This makes correction unnecessary when the objective is classification rather than accurate probability estimation.
- Calibration and clinical utility: Correction methods strongly overestimated minority-class probabilities, which can reduce clinical utility and potentially promote overtreatment.For ovarian cancer diagnosis, overestimating malignancy risk could accelerate referral to advanced specialized surgery.
- Discrimination and sample size: At least for logistic regression, RUS, ROS, and SMOTE were unlikely to improve discrimination or separability between the classes.The study contrasted these corrections with models developed from the original imbalanced data.
- Discrimination and sample size: Random undersampling did not improve AUROC and resulted in lower test-set AUROC values, consistent with concerns about reduced training sample size and overfitting.Undersampling deliberately decreases the sample available for model training.
- Future research: Follow-up work should examine imbalance corrections across other algorithms, including Random Forest, Support Vector Machines, and Neural Networks.The authors specifically identify calibration performance in these model classes as unresolved.
- Conclusion: Imbalance correction produced inaccurate probability estimates and did not improve discrimination in terms of AUROC.This conclusion applied to standard and ridge logistic regression models.
illustration and simulation using logistic regression
The supplied material identifies the study authors and their institutional affiliations across Utrecht and Leuven.
- The study lists authors affiliated with Julius Center, KU Leuven, Leiden University Medical Center, and University Hospitals Leuven.
- The listed affiliations span health sciences, primary care, development and regeneration, obstetrics and gynecology, biomedical data sciences, and epidemiology.
- The document includes a supplementary information section.
Supplementary Notes
The supplementary notes describe random undersampling, random oversampling, and SMOTE, alongside logistic-regression model formulation and ridge penalization.
- Class-imbalance methods: Random undersampling reduces the majority class by discarding randomly selected cases until class sizes match.
- Class-imbalance methods: Random oversampling enlarges the minority class by sampling with replacement, producing duplicate minority cases.
- Class-imbalance methods: SMOTE creates synthetic minority cases by interpolating between minority cases and their k nearest neighbors.
- Class-imbalance methods: SMOTE constructs synthetic feature vectors by multiplying neighbor differences by a random number between 0 and 1 and adding them to the minority case.
- Class-imbalance methods: SMOTE is designed for continuous variables, while ordinal or categorical variables require rounding or an adaptation for mixed types.
- Logistic regression models: Logistic regression models event probability from predictors, with standard logistic regression estimated by maximum likelihood and Ridge shrinking coefficients toward zero.
Coefficient estimation
The simulation calibrated regression coefficients numerically so generated datasets matched desired discrimination and event prevalence.
- The simulation numerically optimized intercepts and coefficients to achieve the desired true AUROC and event fraction.
- The Broyden-Fletcher-Goldfarb-Shanno algorithm was used for optimization.
- The optimization minimized the squared differences between observed and desired AUROC and between observed and desired prevalence.
- The procedure was repeated 20 times, median coefficients were selected, and this process was repeated across 20 generated datasets of size N = 105.
Error handling
The simulations monitored and excluded problematic development datasets, including separation and single-class samples, while documenting rare-event generation issues and SMOTE adaptations.
- Separation handling: Development datasets with apparent AUROC equal to 1 were treated as separated and removed from analysis.This reflected the assumption that clinical prediction models should not be developed on separated data.
- Separation handling: Random undersampling was considered especially prone to producing data separation.The paper notes that separation was likely to occur when random undersampling was used.
- Dataset exclusions: Datasets containing cases from only one class were excluded from the analysis.
- Rare-event monitoring: In the most extreme scenario, fewer than 8 events occurred with probability 0.00002, or 1 in 50,000.This scenario combined a 1% event fraction with a sample size of 2,500.
0.5 (“Unc .5”) or a threshold based on the true event fraction (“Unc EF”). For RUS/ROS/SMOTE,
This section compares classification accuracy under the uncorrected model’s default threshold with the corresponding threshold-based evaluation for imbalance-corrected models.
- Classification accuracy: The uncorrected Ridge model used the default threshold of 0.5 for classification accuracy.
0.5 (“Unc .5”) or a threshold based on the true event fraction (“Unc EF”). For RUS/ROS/SMOTE,
The classification analyses evaluated uncorrected models at alternative thresholds and imbalance-corrected models at the default threshold of 0.5.
- Classification accuracy: Uncorrected models were evaluated using either the default threshold of 0.5 or a threshold based on the true event fraction.
- Classification accuracy: Random undersampling, random oversampling, and SMOTE models were evaluated with the default threshold of 0.5.
0.5 (“Unc .5”) or a threshold based on the true event fraction (“Unc EF”). For RUS/ROS/SMOTE,
Across the listed classification analyses, uncorrected models could use a threshold based on the true event fraction, whereas RUS, ROS, and SMOTE models used a default threshold of 0.5.
- Threshold specification: The uncorrected-model comparisons included both the default threshold of 0.5 and a threshold based on the true event fraction.
- Threshold specification: RUS, ROS, and SMOTE models were evaluated using the default threshold of 0.5.
- Specificity: The same threshold setup was applied in the specificity analyses for Ridge and SLR models.