PaperlayerView exact revision on arXiv

Source-linked AI summary

A review of ensemble learning and data augmentation models for class imbalanced problems: combination, implementation and evaluation

Azal Ahmad Khan, Omkar Chaudhari, Rohitash Chandra

arXiv:2304.02858v3cs.LGcs.AIstat.ML

TL;DR

Class imbalance biases classification toward majority classes and makes conventional accuracy unreliable, motivating better evidence on augmentation–ensemble combinations. The paper computationally evaluates these combinations across benchmark imbalanced datasets and finds that established methods, especially SMOTE- and ROS-LightGBM, outperform newer CT-GAN combinations in the study.

  • Problem

    Class imbalance biases models toward majority classes, while prior reviews lacked computational performance analysis across augmentation and ensemble combinations.

  • Method

    The paper evaluates 9 data augmentation and 9 ensemble learning methods across 23 binary datasets with different imbalance ratios and considers multiclass problems.

  • Results

    SMOTE-LightGBM and ROS-LightGBM are the best combinations in the computational study, while CT-GANs are not top performers.

  • Takeaways & Limitations

    Established augmentation methods such as SMOTE and ROS can provide stronger performance than CT-GANs while being simpler and computationally less expensive.

  • Takeaways & Limitations

    CT-GAN performance depends on the proportion of discrete and continuous columns in tabular data and requires hyperparameter tuning.

Abstract

from arXiv · show

Class imbalance (CI) in classification problems arises when the number of observations belonging to one class is lower than the other. Ensemble learning combines multiple models to obtain a robust model and has been prominently used with data augmentation methods to address class imbalance problems. In the last decade, a number of strategies have been added to enhance ensemble learning and data augmentation methods, along with new methods such as generative adversarial networks (GANs). A combination of these has been applied in many studies, and the evaluation of different combinations would enable a better understanding and guidance for different application domains. In this paper, we present a computational study to evaluate data augmentation and ensemble learning methods used to address prominent benchmark CI problems. We present a general framework that evaluates 9 data augmentation and 9 ensemble learning methods for CI problems. Our objective is to identify the most effective combination for improving classification performance on imbalanced datasets. The results indicate that combinations of data augmentation methods with ensemble learning can significantly improve classification performance on imbalanced datasets. We find that traditional data augmentation methods such as the synthetic minority oversampling technique (SMOTE) and random oversampling (ROS) are not only better in performance for selected CI problems, but also computationally less expensive than GANs. Our study is vital for the development of novel models for handling imbalanced datasets.

1. Introduction

Class imbalance creates majority-class bias and unreliable conventional accuracy, motivating specialized evaluation and combinations of data augmentation with ensemble learning. The paper addresses this gap through a computational framework comparing methods across benchmark binary and multiclass problems.

  • The class imbalance problem: Class imbalance occurs when classification datasets contain disproportionate numbers of instances across classes, with minority classes often representing rare events.Examples include fraud detection, rare disease, natural disasters, and software defect detection.
  • The class imbalance problem: Majority-class bias can produce accuracy up to 98% while misclassifying many minority-class samples, making conventional accuracy inappropriate.Sensitivity, specificity, precision-recall curves, and F1 score are emphasized for imbalanced classification.
  • Existing approaches: Ensemble methods combine multiple models and have shown better accuracy and CI metrics across applications including spam detection, fraud detection, and medical prediction.Bagging, boosting, and stacking are prominent ensemble approaches.
  • Existing approaches: Data augmentation reduces class imbalance at the data level and has been applied to domains such as fault and protein classification.Recent methods also target multiclass imbalanced problems.
  • Study contribution: The framework evaluates 23 binary datasets with different imbalance ratios across 9 data augmentation and 9 ensemble learning methods, also demonstrating suitability for multiclass problems.The study compares methods over different metrics.
  • Study contribution: The paper addresses a reported gap in computational performance analysis by combining a literature review with evaluation, open-source code, and data.Earlier computational review work did not cover GAN-based augmentation.

2. Ensemble Learning

Ensemble learning is a meta-learning approach that combines multiple predictive models to improve performance. Its models are built from adapted training data and their predictions are combined through averaging, voting, or related strategies.

  • Core concept: Ensemble learning seeks better predictive performance by combining predictions from multiple classifiers or expert models.It is used for both regression and classification.
  • Main classes: The three main ensemble learning classes are bagging, boosting, and stacking.These classes support applications including intrusion detection, landslide assessment, and short-term prediction.
  • Core concept: Ensemble methods construct diverse models from reweighted or resampled training data and combine their outputs, often through averaging or voting.The models may be adapted versions of the original training data.

2.1. Bagging

Bagging creates diverse training samples and aggregates predictions from multiple models, while random forests specialize this framework with decision trees and feature-wise randomness. The section situates these methods within CI-focused extensions and applications.

  • Bagging: Bagging uses bootstrapping to create diverse samples and aggregation to combine predictions from independently trained models.Decision-tree ensembles commonly use averaging or voting for the final estimate.
  • Bagging for class imbalance: Recent strategies extend conventional bagging to address class imbalance, including roughly balanced, multi-attribute, and neighbourhood-dependent sampling.These approaches alter how training instances are sampled for the ensemble.
  • Random forests: Random forests are bagged ensembles of decision trees that use different data samples and feature-wise splits, producing classification decisions by majority vote.They restrict the base model to decision trees, unlike general bagging.
  • Random forests: Random forests provide transparent decision rules and feature rankings, enabling users to assess feature contributions and reduce datasets.They can also be implemented with parallel computing for large-data problems.
  • Applications: Random forests have been applied to large-scale image data, medical diagnosis, and other domains beyond tabular classification.Their applications include remote sensing and neuroimaging for Alzheimer’s disease.
  • Applications: Class-imbalance extensions of random forests include cost-sensitive forests, cost-sensitive voting, and feature stratification for minority and majority classes.These innovations target software defect prediction and minority-class representation.

2.2. Boosting

Boosting builds a strong ensemble by combining sequentially added weak learners that correct earlier errors. The section describes AdaBoost and Gradient Boosting, their mechanics, applications, and limitations.

  • Boosting overview: Boosting combines sequential weak learners so each new model corrects predictions from earlier models.Popular methods include AdaBoost, Gradient Boost, and XGBoost.
  • AdaBoost: AdaBoost combines decision-tree stumps iteratively, weighting training examples and aggregating the weak classifiers into a strong classifier.Correctly classified samples receive lower weights, while misclassified samples receive higher weights for the next stump.
  • AdaBoost: AdaBoost differs from Random Forests because its sequential stumps receive unequal weights, whereas Random Forest trees are independent and vote equally.Random Forests can use fully grown trees, while AdaBoost uses weak learners whose influence depends on their performance.
  • AdaBoost: AdaBoost has been reported as resistant to overfitting, but it is sensitive to noisy data and outliers and can be slower than XGBoost.Noise-driven misclassifications can produce poor decision boundaries, motivating updated AdaBoost variants.
  • Gradient Boosting: Gradient Boosting minimizes loss by adding models trained on residual errors, but it is slow on large datasets and requires careful hyperparameter tuning.Its sequential structure also makes parallel implementation difficult, and rapid learning can lead to overfitting.

2.3. Stacking and Voting

Stacking and voting combine predictions from multiple models, but they differ in how those predictions are aggregated. Stacking uses a meta-learner, while voting selects or combines class predictions directly.

  • Stacking: Stacking feeds base learners’ predictions into a meta-learner that learns how to combine them.Base learners are level-0 models, and the meta-learner is the level-1 model.
  • Voting: Voting sums classification predictions or averages regression predictions to produce the ensemble output.Classification voting can be hard, using vote counts, or soft, using predicted probabilities.
  • Comparison: Stacking is typically reported to achieve better performance than voting, although ensemble performance varies across datasets.Prior comparisons include stacking outperforming bagging and boosting in network intrusion detection.
  • Voting: Voting is computationally intensive because it requires multiple models and can underperform a single model in some cases.The passage cautions that voting is not a one-size-fits-all approach.

3. Data Augmentation for CI problems

Data augmentation addresses class imbalance by synthetically increasing minority examples or modifying the class distribution. The section covers sampling methods, SMOTE variants, and GAN-based generation for tabular and other data.

  • Overview: Data augmentation reduces class imbalance by generating new data points from existing observations.Random oversampling duplicates minority samples, while random undersampling removes majority samples.
  • SMOTE methods: SMOTE generates minority-class examples in feature space using k-nearest neighbors and is the prominent augmentation approach for imbalanced tabular datasets.The paper focuses on combining augmentation and ensemble learning for tabular class-imbalance problems.
  • SMOTE methods: The study implements prominent SMOTE-based variants, including SMOTE-ENN, which combines minority oversampling with edited-nearest-neighbor undersampling.ENN removes observations whose class differs from the majority class among their nearest neighbors.
  • SMOTE methods: K-means SMOTE clusters the input space, filters clusters rich in minority samples, and allocates more synthetic samples to sparsely distributed minority regions.It performed better than SMOTE and related methods in almost 90% of classification tasks across seven land-cover datasets.
  • Adaptive methods: ADASYN generates more minority samples for difficult-to-learn examples, shifting attention toward challenging decision-boundary regions.The method adaptively follows the minority-class distribution.
  • GANs: GANs use a generator and discriminator to produce realistic synthetic data and have been applied to tabular class-imbalance problems as well as image and fault-data generation.Examples include combining GANs with SMOTE and using GANs for intrusion detection and machine fault diagnosis.

4. Methodology: combination of ensemble and data augmentation methods

The methodology evaluates combinations of data augmentation and ensemble learning methods for binary and multi-class class-imbalance problems. It compares nine augmentation techniques, nine ensemble methods, and multiple evaluation metrics within a general framework.

  • Methodological scope: The framework evaluates selected combinations of data augmentation and ensemble learning methods on benchmark class-imbalance datasets.It covers binary and multi-class classification tasks and compares the techniques over different metrics.
  • Data augmentation: The augmentation methods include SMOTE, SMOTE-ENN, Borderline SMOTE, SVM-SMOTE, ADASYN, RUS, ROS, and CT-GAN.The study also evaluates ensemble methods without augmentation to assess the role of augmentation.
  • Ensemble learning: The ensemble methods span bagging, boosting, stacking, and voting, including LightGBM, AdaBoost, XGBoost, Gradient Boosting, Random Forest, Voting-Soft, Voting-Hard, Stacking-I, and Stacking-II.Stacking-I uses several base learners with Logistic Regression as a meta-learner, while Stacking-II uses Random Forests and XGBoost.
  • Evaluation framework: The evaluation framework organizes datasets, augmentation methods, ensemble methods, and metrics for binary and multi-class classification tasks.This structure is summarized in Figure 4.
  • Evaluation metrics: Performance is assessed using accuracy, F1 score, and ROC-AUC, while the methodology notes that accuracy can misrepresent performance under class imbalance.Precision and recall are also treated as useful imbalance-aware measures, and F1 combines them.
  • Multi-class metrics: For multi-class evaluation, ROC-AUC can use One-Versus-One or One-Versus-Rest strategies, while F1 can be Macro-F1, Micro-F1, or Weighted-F1.These variants differ in whether they emphasize pairwise class comparisons, aggregate instances, equal class weighting, or class frequency.

5. Computational Study

The computational study evaluates combinations of data augmentation and ensemble learning across binary and multiclass imbalanced datasets. Results vary by dataset and metric, but several augmentation–ensemble combinations repeatedly perform strongly, especially SMOTE-based approaches and LightGBM-related combinations.

  • Experimental design: 30 independent runs evaluate every augmentation–ensemble combination using fixed train/test splits and reporting mean, best, and standard deviation results.The study uses a 60:40 train-test ratio and executes augmentation before classification.
  • Binary classification: SMOTE-ENN ranks best on Glass, SMOTE and Borderline-SMOTE on Yeast, and ADASYN on Ecoli, while RUS generally performs worst.These rankings are based on F1 scores across the respective dataset families.
  • Binary classification: LightGBM combinations are superior in most datasets, while SMOTE-LightGBM, ROS-LightGBM, and CT-GAN-LightGBM lead selected Glass, Yeast, and Ecoli AUC comparisons.CT-GAN combinations are particularly noteworthy for Ecoli datasets with lower imbalance ratios.
  • Multiclass classification: For multiclass classification, SMOTE-Stacking-I and SMOTE-LightGBM lead F1-macro, whereas Gradient Boosting without augmentation leads F1-weighted.SMOTE-Gradient Boosting and Stacking-II are top performers across reported AUC settings, and SMOTE-based combinations generally perform better.
  • Multiclass classification: Data augmentation has a greater impact than ensemble learning techniques across four AUC score variants using both OvO and OvR strategies.The comparison averages macro and weighted strategies.

6. Discussion

The discussion compares augmentation–ensemble combinations across benchmark datasets and metrics, highlighting recurring performance patterns and areas for further progress.

  • Application scope: The study evaluates selected combinations on applications including medical imaging, fraud detection, and natural language processing.The discussion frames class imbalance as a common issue across these application areas.
  • Future directions: Future progress requires evaluation and benchmark development that better represent diverse applications and emerging big-data problems.The discussion identifies broader benchmark coverage as necessary for robust class-imbalance models.

6.1. Benchmark datasets

The benchmark suite consists mainly of small tabular datasets focused on binary class imbalance, with some modifications for multiclass settings and limited coverage of modern data domains.

  • Dataset characteristics: The study uses relatively small tabular datasets with at most a few dozen features and fewer than a few thousand instances.The authors describe this as a limitation of the selected benchmark datasets.
  • Class structure: Most benchmark datasets address binary class imbalance, although selected datasets were modified for multiclass scenarios.This limits how broadly the findings represent multiclass problems.
  • Coverage limitations: Existing benchmarks insufficiently cover image, vision, natural-language, and signal-processing applications needed for emerging and big-data models.The authors call for better benchmark datasets spanning these application domains.
  • Future directions: Future work could combine deep ensemble models with data augmentation for spatial, temporal, small-scale, and big-data problems.The paper identifies this combination as an open direction rather than a tested result.

6.2. Natural language processing

The discussion extends class-imbalance considerations to NLP, image augmentation, explainability, climate extremes, finance, pretrained models, and uncertainty quantification.

  • Natural language processing: NLP class imbalance appears in sentiment, spam, fake-news, bullying, abuse, and digital-marketing classification tasks.The paper notes that augmentation methods alone cannot address more complex NLP problems.
  • Natural language processing: Pretrained models such as BERT and Chat-GPT are presented as potential tools for augmenting or addressing complex NLP imbalance problems.The discussion links pretrained models to contextual feature extraction and data augmentation.
  • Image-based augmentation: Image-based augmentation has been widely used in computer vision to improve performance and reduce overfitting.The paper anticipates more sophisticated generative techniques for simulating real-world variation.
  • Explainability: GAN-generated data may face trust barriers in medical diagnosis because related technologies can impersonate individuals and biometrics.The authors connect these concerns to the need for explainable or grey-box models.
  • Extreme forecasting: Extreme forecasting resembles class imbalance because both involve insufficient observations of rare events, including storms, droughts, and floods.The paper states that similar methods can address these data-scarcity challenges in climate science.
  • Uncertainty quantification: Synthetic data for imbalanced forecasting raises a need for uncertainty quantification through Bayesian inference methods such as variational inference and MCMC.The discussion presents uncertainty in model parameters and predictions as important when synthetic data are used.

7. Conclusion

The paper computationally reviews established and novel augmentation–ensemble combinations on selected imbalanced datasets. It identifies effective combinations and finds traditional augmentation methods outperform GANs at lower computational cost, while motivating new ensemble designs.

  • Conclusion: The study implements a computational review of existing and novel data augmentation–ensemble combinations for selected class-imbalance problems.The review covers binary and multiclass imbalanced datasets and also reports ensemble performance without augmentation.
  • Conclusion: SMOTE-LightGBM and ROS-LightGBM are identified as the best combinations in the computational study.The conclusion reports these combinations as more effective than others for the evaluated problems.
  • Conclusion: SMOTE and ROS outperform GANs while requiring less computational expense.The conclusion specifically contrasts these traditional methods with newer GAN-based augmentation.
  • Conclusion: Flexible ensemble architectures offer potential for naturally combining data augmentation with imbalance handling.The paper presents novel ensemble design as a future research direction.

Code and Data

The paper provides a GitHub repository for its code and data.

  • A GitHub repository is provided for the paper’s code.
  • The repository also provides the paper’s data.
  • The repository serves as a resource for accessing the paper’s implementation materials.
Loading 2304.02858v3…