Source-linked AI summary
Foundations of data imbalance and solutions for a data democracy
Ajay Kulkarni, Deri Chong, Feras A. Batarseh
TL;DR
The chapter addresses how imbalanced classification data can bias decisions and make accuracy misleading. It explains imbalance and concept complexity, implements statistical assessment and data-level methods on an insurance dataset, and reports that oversampling methods performed better than undersampling in this scenario.
Problem
Imbalanced class distributions can bias classification decisions, while high accuracy may coexist with poor minority-class performance.
Method
The chapter examines imbalance factors and statistical metrics, then implements and compares data-level sampling methods on a real-life insurance dataset.
Results
Oversampling methods performed better than undersampling in the analyzed scenario; random undersampling classified positive instances, while Tomek Link and ENN did not.
Takeaways & Limitations
Selecting a balancing method requires comparing multiple preprocessing methods while considering the data and problem scenario; feature engineering may further improve results.
Takeaways & Limitations
No single balancing method suits all problems, so method performance remains dependent on the data and problem scenario.
Abstract
from arXiv · showhide
Dealing with imbalanced data is a prevalent problem while performing classification on the datasets. Many times, this problem contributes to bias while making decisions or implementing policies. Thus, it is vital to understand the factors which cause imbalance in the data (or class imbalance). Such hidden biases and imbalances can lead to data tyranny and a major challenge to a data democracy. In this chapter, two essential statistical elements are resolved: the degree of class imbalance and the complexity of the concept; solving such issues helps in building the foundations of a data democracy. Furthermore, statistical measures which are appropriate in these scenarios are discussed and implemented on a real-life dataset (car insurance claims). In the end, popular data-level methods such as random oversampling, random undersampling, synthetic minority oversampling technique, Tomek link, and others are implemented in Python, and their performance is compared.
1. Motivation & Introduction
Imbalanced class distributions can bias classification and make accuracy misleading, particularly when minority instances are misclassified despite high overall accuracy. The chapter therefore focuses on understanding and addressing imbalance before selecting classification methods.
- Imbalanced data arise when classes have unequal distributions, a widespread issue especially in binary classification.
- Classification algorithms assume maximized accuracy and matching training and test distributions, but imbalanced data can violate one or both assumptions.
- In fraud detection, a classifier may label many fraudulent transactions as non-fraudulent, producing high accuracy but poor performance.
- The chapter implements common approaches, examines misleading traditional metrics, and discusses solutions for imbalanced data.
2. Imbalanced Data Basics
The chapter characterizes imbalance through its degree and the complexity of the represented concept. Class overlap and small disjoints complicate separability, while the Imbalanced Ratio quantifies class-distribution disparity.
- The imbalance problem depends on degree of class imbalance, concept complexity, training-data size, and classifier type.
- Degree of class imbalance: The Imbalanced Ratio represents imbalance by dividing the total negative-class examples by the total positive-class examples.
- Degree of class imbalance: A dataset with 5,000 negative and 1,000 positive examples has a 1:5 class ratio and an Imbalanced Ratio of 5.
- Complexity of the concept: Class overlap mixes examples from both classes in feature space, while small disjoints form the minority concept from subconcepts.
- Complexity of the concept: These complexity factors reduce class separability, require more complex classification rules, and can cause minority-class misclassification.
- Data-level approaches are presented to improve data quality for analysis and overall results.
3. Statistical Assessment Metrics
This section presents statistical metrics for evaluating classifiers on imbalanced data and applies them to the Porto Seguro insurance-claim dataset. The results show that high accuracy can coexist with failure to identify positive cases, motivating class balancing.
- Implementation: The section implements statistical assessment procedures in Python using the Porto Seguro dataset and points readers to precision-recall and cost-sensitive measures for imbalanced-data evaluation.The described implementations use functions from sklearn.metrics and imblearn.metrics where applicable.
- Confusion matrix: Confusion matrices summarize counts of true negatives, true positives, false positives, and false negatives from predicted and actual class values.They support binary and multiclass classification assessment.
- Classification metrics: Accuracy can mislead on imbalanced datasets, so precision, recall, F1, and G-mean provide additional information about positive-class performance.Precision measures correctness among predicted positives, while recall measures the model’s ability to predict positive outcomes.
- ROC curve and AUC: ROC curves plot False Positive Rate against True Positive Rate across classification thresholds, while AUC assigns a score between 0 and 1.AUC values below 0.5 correspond to performance below random guessing, whereas an ideal classifier has AUC 1.
- Insurance dataset: The Porto Seguro dataset contains about 595,000 driver records, with 573,000 negative and 21,000 positive claims, producing an imbalanced ratio of 26.44.Each row represents a policyholder, and the target indicates whether an insurance claim was filed.
- Insurance dataset results: Logistic regression achieved 96% accuracy on the insurance dataset, yet classified no instances as positive and produced precision, recall, F1, and G-mean scores of 0.These results indicate that balancing the classes is critical before further predictions.
4. How to deal with imbalanced data
Sampling methods modify imbalanced datasets to create a more balanced or adequate distribution for learning. The chapter groups these methods into undersampling, oversampling, and hybrid approaches implemented with imbalanced-learn.
- The approaches are divided into undersampling, oversampling, and hybrid methods.
- Sampling methods modify imbalanced datasets to make class distributions more balanced for learning tasks.
4.1 Undersampling
Undersampling reduces majority-class instances through random selection or heuristic neighborhood-based removal. On the evaluated dataset, random undersampling enabled positive-class classification, whereas Tomek Links and ENN retained 96% accuracy but classified no positive instances.
- Undersampling removes majority-class instances to balance the dataset, using Random Undersampling, Tomek Links, or ENN.
- Random Undersampling: Random Undersampling randomly removes majority-class examples until their count matches the minority class.
- Random Undersampling: 59% accuracy followed Random Undersampling, while the model classified positive-class instances and produced nonzero supporting metrics.
- Tomek Links: Tomek Links identify cross-class nearest-neighbor pairs and remove the majority-class member while retaining the minority-class instance.
- Tomek Links: 96% accuracy accompanied Tomek Links, but the model classified no positive instances and precision, recall, F1, and G-mean were 0.
- Edited Nearest Neighbors: ENN removes majority-class instances when most of their K-nearest neighbors belong to the minority class.
- Edited Nearest Neighbors: 96% accuracy accompanied ENN, but the model classified no positive instances, making it unsuitable for balancing in this scenario.
4.2 Oversampling
Oversampling increases minority-class instances through replication or synthetic generation. In the evaluated comparisons, random oversampling enabled positive-class classification, while SMOTE and ADASYN produced 58% accuracy; ADASYN also yielded an AUC of 0.62.
- Oversampling increases minority-class instances to match the majority class and includes Random Oversampling, SMOTE, and ADASYN.
- Random Oversampling: Random Oversampling randomly replicates minority-class instances to produce a balanced class distribution.
- Random Oversampling: 59% accuracy followed Random Oversampling, which classified positive-class instances and produced precision, recall, F1, and G-mean values at or below 0.6.
- SMOTE: SMOTE creates minority-class instances synthetically by interpolating among nearby minority examples, addressing replication-related overfitting concerns.
- ADASYN: ADASYN generates more synthetic minority data from samples that are harder to learn, according to their density distribution.
4.3 Hybrid Methods
Hybrid methods combine undersampling and oversampling to balance majority-class removal with minority-class creation. The chapter identifies SMOTEENN and SMOTETomek as hybrid methods available in Python.
- Hybrid methods combine undersampling and oversampling to balance majority-class removal with minority-class creation.
- Python provides SMOTEENN, combining SMOTE with ENN, and SMOTETomek, combining SMOTE with Tomek Link.
5. Other Methods
Beyond data-level sampling, imbalance can also be addressed through algorithmic-level and ensemble-based approaches that target the classifier or combine models.
- Algorithmic-level approaches modify the classifier rather than the dataset to address imbalance.These methods differ from data-level methods, which primarily resample the dataset.
- Ensemble-based approaches are identified as another family of methods for handling imbalanced data.
- Data-level methods primarily use undersampling, oversampling, or hybrid sampling to alter the class distribution.
Conclusion
The chapter compares data-level methods for imbalanced classification and finds oversampling more effective than undersampling in the evaluated scenario. It also emphasizes method selection, feature engineering, and caution about information loss or overfitting.
- Oversampling methods outperformed undersampling in the evaluated scenario across the reported classification assessment measures.The measures included accuracy, precision, recall, F1 score, G-mean score, and AUC score.
- Only Random under-sampling classified positive-class instances after processing, whereas all discussed oversampling methods produced better results.
- All discussed oversampling methods produced similar accuracy, precision, recall, F1, G-mean, and AUC results.
- No single imbalance method suits every problem, so multiple preprocessing methods should be compared against the data and problem scenario.
- Undersampling may remove significant majority-class records, while oversampling may cause model overfitting or create irrelevant records.
- Feature engineering through feature combinations or selection can improve results before applying imbalance-handling methods.