Source-linked AI summary
Survey of resampling techniques for improving classification performance in unbalanced datasets
Ajinkya More
TL;DR
Class imbalance creates a need for classifiers that improve minority-class recall without disregarding majority-class precision. The paper reviews resampling and related approaches, comparing them on a synthetic dataset; SMOTE+ENN with logistic regression and BalanceCascade perform best on the chosen metric, while outcomes depend on data distribution and classifier choice.
Problem
Class imbalance can make standard classifiers suboptimal when high minority-class recall and majority-class precision are desired.
Method
The paper reviews imbalance-handling techniques and compares their effects using a synthetic two-class dataset, cross-validation, and test-set evaluation.
Results
On the synthetic dataset, SMOTE+ENN with logistic regression and BalanceCascade give the best performance with respect to the chosen metric.
Takeaways & Limitations
Other methods may perform better depending on data distribution, within-class imbalance, and the classifier used on resampled datasets.
Takeaways & Limitations
The techniques discussed are not exhaustive, and other proposed methods have also succeeded in handling data imbalance.
Abstract
from arXiv · showhide
A number of classification problems need to deal with data imbalance between classes. Often it is desired to have a high recall on the minority class while maintaining a high precision on the majority class. In this paper, we review a number of resampling techniques proposed in literature to handle unbalanced datasets and study their effect on classification performance.
1. INTRODUCTION
Class imbalance can make standard classifiers perform poorly on minority-class objectives, motivating techniques that improve minority recall while preserving majority-class precision. The paper situates this need in fraud detection, product categorization, and disease diagnosis.
- Imbalanced datasets contain significantly different numbers of examples across classes, which can make out-of-the-box classifiers suboptimal for minority-class performance.
- The paper studies techniques for boosting classification performance under data imbalance and introduces domains where unbalanced datasets are common.
- Fraud detection involves few fraudulent transactions and may prioritize near-perfect fraudulent-class recall when manual review costs less than missed fraud.
- Product categorization can confuse rare iPhone models with far more numerous accessories because their descriptions and images overlap.
- Disease diagnosis is imbalanced because healthy people outnumber affected people, while rare diseases intensify that imbalance.
- Disease classification may require both near-one recall and high minority-class precision because false positives can demand substantial expert analysis.
2. NOTATION AND METRICS
The paper compares imbalance-handling methods using minority-class recall and majority-class precision on a two-class case study. This metric choice reflects settings with large-scale screening, costly missed detections, and cheaper manual review.
- The case study denotes the majority class by L, the minority class by S, their size ratio by r = |S|/|L|, and the training set by T.
- Methods are compared by their effect on recall for minority class S and precision for majority class L.
- The evaluation setting assumes many instances require classification, minority cases are rare, and only flagged minority cases receive manual review.
- Because missed minority detections cost more than manual review, the motivating applications include fraud detection, network failure identification, and product-issue detection.
3. DATASET
The study evaluates imbalance-handling methods on a synthetic two-class dataset generated with scikit-learn and reduced to two principal components for visualization.
- The dataset contains 10000 samples and 2 classes generated with scikit-learn’s make classification function.
- Class weights are [0.1, 0.9], with class separation set to 1.2.
- The dataset has 5 features, including 3 informative and 1 redundant feature, with 1 cluster per class.
- Principal component analysis reduces the data to its first two components to facilitate visualization.
4. METHOD COMPARISON
The method comparison uses a 70/30 train-test split, five-fold cross-validation, and scikit-learn and imbalanced-learn implementations. It includes logistic-regression baselines and weighted-loss handling alongside resampling methods.
- 4. METHOD COMPARISON: The dataset is split into 70% training and 30% test data, with five-fold cross-validation used to select parameters before test evaluation.
- 4.1 Baseline: Logistic regression provides a baseline by searching over regularization parameters and l1 or l2 penalty types using five-fold cross-validation.
- 4.2 Weighted loss function: A weighted loss function increases the penalty for misclassifying minority-class examples to boost minority-class performance.
- 4.2 Weighted loss function: In scikit-learn, setting class weight to balanced applies weights inversely proportional to class sizes to the loss function.
- 4.2 Weighted loss function: The resulting decision boundary and test-set performance are reported for the weighted-loss approach.
4.3 Undersampling methods
Undersampling reduces the majority class using random selection or distance-based NearMiss criteria, with NearMiss variants retaining points according to their proximity to minority examples.
- Random undersampling can lose majority-class information but may work when majority examples are near other same-class examples.
- NearMiss methods undersample majority points using distances to minority-class points.
- NearMiss-1 retains majority points with the lowest mean distance to the k nearest minority points.
- NearMiss-2 instead retains majority points with the lowest mean distance to the k farthest minority points.
4.4 NearMiss-3
NearMiss-3 selects majority neighbors for each minority point, while CNN, ENN, repeated ENN, and Tomek-link removal provide alternative undersampling strategies with distinct selection rules.
- NearMiss-3: NearMiss-3 selects k nearest majority neighbors in L for every minority point in S, directly controlling the undersampling ratio through k.
- CNN: CNN iteratively builds a subset whose nearest-neighbor class assignments match each training point, adding points that are misrepresented by the current subset.
- CNN: CNN may be slower because it requires many passes over training data, and its random point selection can produce substantially different subsets.
- ENN: ENN removes majority points whose labels differ from the majority label among their k nearest neighbors, while repeated ENN continues until no further removals are possible.
- Tomek links: Tomek-link undersampling removes cross-class nearest-neighbor pairs, either entirely or only their majority-class members.
4.5 Oversampling methods
Oversampling expands the minority class through duplication or synthetic generation, including SMOTE and borderline variants that target minority points according to neighborhood structure.
- Random minority oversampling with replacement can cause overfitting.
- SMOTE: SMOTE generates synthetic minority examples along line segments between each minority point and randomly selected minority neighbors.
- Borderline SMOTE: Borderline SMOTE variants are proposed as enhancements that may outperform vanilla SMOTE.
- Borderline-SMOTE1: Borderline-SMOTE1 classifies minority points by their m nearest neighbors in the full training set, ignoring noisy and safe points while applying SMOTE to danger points.
- Borderline-SMOTE2: Borderline-SMOTE2 additionally generates synthetic examples toward nearest minority or majority neighbors, placing majority-directed points closer to the original.
4.6 Combination methods
Combination methods pair oversampling with undersampling, exemplified by SMOTE followed by Tomek-link removal or ENN.
- Combining oversampling and undersampling can often yield better results than either technique alone.
- One combination applies SMOTE with k = 5 and r = 0.5, followed by Tomek-link removal.
- Another applies SMOTE with k = 5 and r = 0.5, followed by ENN with k = 5.
4.7 Ensemble methods
EasyEnsemble repeatedly trains AdaBoost ensembles on randomly undersampled majority-class subsets, then combines them into a meta-ensemble. BalanceCascade follows a similar iterative structure but uses each classifier to guide subsequent sampling and removes correctly classified majority points.
- EasyEnsemble: EasyEnsemble randomly samples majority-class subsets matched in size to the minority class and learns an AdaBoost ensemble on each subset.
- EasyEnsemble: The resulting EasyEnsemble classifiers are combined into a meta-ensemble.
- BalanceCascade: BalanceCascade resembles EasyEnsemble but lets each iteration’s classifier influence the next selection of points.
- BalanceCascade: BalanceCascade tunes each classifier so its false positive rate equals t, then removes correctly classified majority-class points before continuing.
- BalanceCascade: BalanceCascade combines the classifiers produced across iterations into a meta-ensemble.
5. CONCLUSION
The paper reviews resampling methods for improving minority-class classification under imbalance and evaluates several methods on a synthetic dataset. SMOTE+ENN with logistic regression and BalanceCascade perform best under the chosen metric, but the method comparison is not exhaustive and results depend on data and classifier choices.
- The paper evaluates several resampling methods on a synthetic dataset using majority-class precision and minority-class recall.
- The methods reviewed are not exhaustive; other proposed techniques include ADASYN, SVM SMOTE, SMOTEBoost, and kernel-based methods.
- SMOTE+ENN with logistic regression and BalanceCascade give the best performance on the synthetic dataset under the chosen metric.
- Other methods may perform better depending on data distribution, within-class imbalance, and the classifier applied to resampled data.