Source-linked AI summary
A systematic study of the class imbalance problem in convolutional neural networks
Mateusz Buda, Atsuto Maki, Maciej A. Mazurowski
TL;DR
Class imbalance is important for CNN classification, but systematic evidence on how it affects deep learning and which remedies work best is limited. The study compares sampling, two-phase training, and thresholding across benchmark datasets and finds that oversampling generally performs best, with complete rebalancing recommended.
Problem
Class imbalance can significantly harm classifier training and generalization, yet its effects and remedies have been less systematically studied for deep-learning classifiers.
Method
The study evaluates imbalance forms, multi-class ROC AUC, and four remedies—oversampling, undersampling, two-phase training, and prior-probability thresholding—across CNN benchmarks.
Results
Class imbalance is detrimental, its impact increases with task scale, and oversampling outperforms other methods in most cases without causing CNN overfitting.
Takeaways & Limitations
Oversampling should fully eliminate imbalance, while undersampling’s optimal ratio depends on imbalance extent and can match oversampling for extreme ratios with many minority classes.
Takeaways & Limitations
ImageNet results were confounded by substantially reduced training-set size and may require a more extensive study of extreme imbalance.
Abstract
from arXiv · showhide
In this study, we systematically investigate the impact of class imbalance on classification performance of convolutional neural networks (CNNs) and compare frequently used methods to address the issue. Class imbalance is a common problem that has been comprehensively studied in classical machine learning, yet very limited systematic research is available in the context of deep learning. In our study, we use three benchmark datasets of increasing complexity, MNIST, CIFAR-10 and ImageNet, to investigate the effects of imbalance on classification and perform an extensive comparison of several methods to address the issue: oversampling, undersampling, two-phase training, and thresholding that compensates for prior class probabilities. Our main evaluation metric is area under the receiver operating characteristic curve (ROC AUC) adjusted to multi-class tasks since overall accuracy metric is associated with notable difficulties in the context of imbalanced data. Based on results from our experiments we conclude that (i) the effect of class imbalance on classification performance is detrimental; (ii) the method of addressing class imbalance that emerged as dominant in almost all analyzed scenarios was oversampling; (iii) oversampling should be applied to the level that completely eliminates the imbalance, whereas the optimal undersampling ratio depends on the extent of imbalance; (iv) as opposed to some classical machine learning models, oversampling does not cause overfitting of CNNs; (v) thresholding should be applied to compensate for prior class probabilities when overall number of properly classified cases is of interest.
1 Introduction
Class imbalance is a widespread and consequential problem in deep-learning classification, yet it has been studied far less systematically for CNNs than for classical machine-learning models. Existing responses include sampling-based approaches and newer neural-network methods such as cost-sensitive learning, specialized loss functions, and two-phase training.
- CNNs and motivation: CNNs have become important across computer vision and other application domains, integrating automatic feature extraction and discriminative representations.Their complex architectures require substantial computational power for training and evaluation, commonly supplied by GPUs.
- The class imbalance problem: Class imbalance occurs when some classes have significantly more training examples than others, creating a common problem across real-world deep-learning applications.The issue is reported in computer vision, medical diagnosis, fraud detection, and other domains.
- Existing approaches: Methods for handling imbalance are well studied in classical machine learning, with sampling methods modifying the data to increase balance.Oversampling increases representation of underrepresented classes, while random majority undersampling removes examples from majority classes.
- Research gap: Deep-learning research has examined cost-sensitive learning, new loss functions, and two-phase CNN training, but systematic analysis of imbalance remains limited.The two-phase method first trains on balanced data and then fine-tunes the output layers.
2 Methods for addressing imbalance
The paper organizes imbalance remedies into data-level, classifier-level, and hybrid methods. It then describes sampling, threshold adjustment, cost-sensitive learning, novelty detection, and combinations such as ensembling.
- Method categories: Imbalance methods either alter the training data distribution, adjust training or inference algorithms, or combine both categories.Data-level methods modify the dataset so standard training algorithms can work, whereas classifier-level methods leave the dataset unchanged.
- Sampling methods: Oversampling replicates randomly selected minority-class samples, while SMOTE creates synthetic examples by interpolating neighboring data points.Oversampling is widely used in deep learning but has been reported to cause overfitting; SMOTE aims to overcome this issue.
- Sampling methods: Undersampling randomly removes majority-class examples until every class has the same number of samples.Its main disadvantage is discarding available data, although it can be preferable to oversampling in some situations.
- Classifier-level methods: Thresholding changes test-time decision thresholds or output class probabilities, with its basic form compensating for prior class probabilities.Corrected class probabilities are obtained by dividing each network output by the estimated prior probability of its class.
- Additional methods: Other approaches include cost-sensitive learning, one-class classification, and hybrid methods such as ensembles trained on undersampled subsets.Cost-sensitive learning can modify inference, backpropagation, learning rates, or the loss function; novelty detection recognizes positive instances through reconstruction error.
3 Experiments
The experiments model class imbalance as either step or linear imbalance and evaluate seven practical CNN-training methods across MNIST, CIFAR-10, and ImageNet. Performance is assessed primarily with multiclass ROC AUC because overall accuracy can be misleading on imbalanced datasets.
- Imbalance models: The study examines step imbalance, where minority and majority classes are internally balanced but differ in class size, and linear imbalance, where class sizes change by constant increments.Step imbalance is parameterized by the minority-class fraction µ and majority-to-minority ratio ρ; linear imbalance uses the maximum-to-minimum class-size ratio ρ.
- Compared methods: Seven methods are compared: minority oversampling, majority undersampling, two variants of two-phase training, thresholding, and sampling combined with thresholding.The two-phase variants pre-train on randomly oversampled or undersampled data, while thresholding uses prior class probabilities and is also combined with both sampling methods.
- Datasets and protocol: Experiments use MNIST, CIFAR-10, and ImageNet, pairing each dataset with a model and training configuration of increasing complexity.Networks for each dataset use the same number of weight updates, start from random initialization, and receive no pretraining.
- MNIST experiments: MNIST experiments span ρ values from 10 to 5 000 and µ values from 0.1 to 0.9 for step imbalance, with each parameter combination repeated 50 times.The number of minority classes ranges from 1 to 9, and the minority-class subsets are randomized across repetitions.
- Evaluation metric: Multiclass ROC AUC is used because overall accuracy can favor overrepresented classes and produce misleading assessments when test sets are imbalanced.ROC AUC is calculated from classifier sensitivities and specificities across prediction thresholds, using trapezoidal-rule AUC calculation.
4 Results
Class imbalance substantially harms CNN classification, with stronger effects as imbalance increases and on more complex datasets. Across MNIST, CIFAR-10, and ImageNet, oversampling was generally the most reliable remedy, while thresholding improved overall accuracy by correcting class-prior effects.
- Impact of class imbalance: Class imbalance substantially deteriorated performance, with larger majority-to-minority ratios and more minority classes causing greater degradation, especially on CIFAR-10 than MNIST.The study evaluated multi-class ROC AUC because overall accuracy is difficult to interpret under imbalance.
- Sampling methods: Oversampling was best in almost all MNIST and CIFAR-10 scenarios, usually improving over the do-nothing baseline without considerable performance decreases.Its default version remained best, and any reduction of imbalance improved scores regardless of the number of minority classes.
- Sampling methods: Undersampling generally performed poorly, although intermediate or reduced levels sometimes matched oversampling or exceeded the baseline when minority classes were numerous.The optimal undersampling level was not known a priori, so oversampling remained the preferred method.
- Sampling methods: For equal imbalance ratios, undersampling’s performance was invariant to minority-class count, while its gap from oversampling narrowed as more classes became minority.The methods become equivalent when all classes are minority.
- Training strategies: Two-phase training generally fell between the baseline and its corresponding sampling method, and fine-tuning reduced scores when oversampling already outperformed the baseline.Fine-tuning provided no gain when the baseline was better.
- ImageNet: On ImageNet, multi-class ROC AUC dropped from 99 to 90 at the largest tested imbalance, while oversampling consistently outperformed undersampling across scenarios.Differences of 1–2 may reflect run variability, and the most imbalanced training set was only about 10% of the original dataset.
- Thresholding: Thresholding improved overall accuracy by offsetting learned class priors, especially when combined with oversampling, without changing class-discrimination ability or ROC AUC.It selects an operating point that yields more correctly classified cases.
- Generalization: Oversampling did not cause CNN overfitting: on CIFAR-10 it improved classifier performance and generalization while making training more stable.The training–test accuracy gap did not increase with iterations for oversampling.
5 Conclusions
Class imbalance detrimentally affects CNN classification, with stronger effects as task scale increases and outcomes depending on class-example distribution. Oversampling generally performs best, while undersampling and thresholding are preferable under specific conditions.
- Class imbalance effects: Class imbalance detrimentally affects CNN classification performance, and its influence increases with task scale.The impact cannot be explained simply by fewer total training cases.
- Class imbalance effects: The impact of imbalance depends on how examples are distributed among classes, not merely on the lower total number of training cases.
- Methods for handling imbalance: Oversampling outperforms other methods in most cases with respect to multi-class ROC AUC.
- Methods for handling imbalance: For extreme imbalance with many minority classes, undersampling performs on a par with oversampling and dramatically reduces training-set size.This makes undersampling preferable when training time is an issue.
- Methods for handling imbalance: Thresholding should compensate for prior class probabilities for best accuracy; combining it with baseline or oversampling is preferred, but not with undersampling.Oversampling does not cause CNN overfitting, unlike in some classical machine learning models.
- Methods for handling imbalance: Oversampling should completely eliminate imbalance, whereas the optimal undersampling ratio depends on imbalance extent and minority-class fraction.Higher minority-class fractions require a greater reduction in the imbalance ratio.