PaperlayerView exact revision on arXiv

Source-linked AI summary

LoRAS: An oversampling approach for imbalanced datasets

Saptarshi Bej, Narek Davtyan, Markus Wolfien, Mariam Nassar, Olaf Wolkenhauer

arXiv:1908.08346v4cs.LGstat.ML

TL;DR

The paper addresses SMOTE’s tendency to over-generalize minority classes in imbalanced datasets. It introduces LoRAS, which combines localized Gaussian shadowsampling with convex combinations to approximate the minority manifold. Across benchmark datasets, LoRAS improves both F1-Score and Balanced accuracy, while its theory establishes a lower-variance mean estimator than SMOTE under stated assumptions.

  • Problem

    SMOTE can over-generalize the minority class, causing majority-class misclassifications and poorer overall balance.

  • Method

    LoRAS generates localized Gaussian shadowsamples and combines them convexly to oversample from an approximated minority-class data manifold.

  • Results

    LoRAS improves both F1-Score and Balanced accuracy on average, while its estimator has lower variance than SMOTE when |F| > 2.

  • Takeaways & Limitations

    LoRAS produces better classification models for tabular high-dimensional and highly imbalanced datasets within the studied scope.

  • Takeaways & Limitations

    The approach may have limited scope for heterogeneous image-based imbalanced datasets, and its theoretical comparison assumes a local shifted t-distribution.

Abstract

from arXiv · show

The Synthetic Minority Oversampling TEchnique (SMOTE) is widely-used for the analysis of imbalanced datasets. It is known that SMOTE frequently over-generalizes the minority class, leading to misclassifications for the majority class, and effecting the overall balance of the model. In this article, we present an approach that overcomes this limitation of SMOTE, employing Localized Random Affine Shadowsampling (LoRAS) to oversample from an approximated data manifold of the minority class. We benchmarked our algorithm with 14 publicly available imbalanced datasets using three different Machine Learning (ML) algorithms and compared the performance of LoRAS, SMOTE and several SMOTE extensions that share the concept of using convex combinations of minority class data points for oversampling with LoRAS. We observed that LoRAS, on average generates better ML models in terms of F1-Score and Balanced accuracy. Another key observation is that while most of the extensions of SMOTE we have tested, improve the F1-Score with respect to SMOTE on an average, they compromise on the Balanced accuracy of a classification model. LoRAS on the contrary, improves both F1 Score and the Balanced accuracy thus produces better classification models. Moreover, to explain the success of the algorithm, we have constructed a mathematical framework to prove that LoRAS oversampling technique provides a better estimate for the mean of the underlying local data distribution of the minority class data space.

1 Introduction

The paper introduces LoRAS to address SMOTE’s tendency to over-generalize minority data by generating samples from a locally approximated minority-class manifold. It combines localized noise and convex combinations, and supports the approach with benchmarking and mathematical analysis.

  • LoRAS generates minority-class oversamples from a locally approximated data manifold to address SMOTE’s over-generalization and majority-class misclassification limitations.
  • The study compares LoRAS with SMOTE and related extensions across publicly available imbalanced datasets and reports better average performance for LoRAS.
  • SMOTE generates artificial minority points as random convex combinations of nearby minority samples, but does not account for minority-class distributions or latent noise.
  • LoRAS combines localized Gaussian noise around minority samples with convex combinations of multiple noisy samples.
  • A mathematical framework argues that combining multiple shadowsamples provides a better estimate of the local minority-class distribution mean than SMOTE.

2 LoRAS: Localized Randomized Affine Shadowsampling

LoRAS builds local minority-class neighborhoods, adds normally distributed shadow points, and forms synthetic samples through random convex combinations. The method can use manifold learning to choose neighborhoods and treats the resulting samples as local manifold approximations.

  • A random affine combination uses nonnegative coefficients summing to one, producing a convex combination of selected vectors.
  • LoRAS uses |F| shadowsamples to approximate a local (|F|−1)-dimensional manifold plane, while allowing fewer affine-combination points in practice.
  • For high-dimensional data, t-SNE can be used before neighborhood selection because ordinary distance measures may poorly reflect the latent manifold.
  • LoRAS selects a minority-class neighborhood, generates noisy shadowsamples around its parent points, and forms synthetic points through positive-weight affine combinations.The method assumes sufficiently low noise keeps shadowsamples near the latent manifold and uses their affine span as a local approximation.
  • The algorithm exposes neighborhood, noise, shadow-sample, affine-combination, generation-count, and embedding parameters for controlling synthetic-sample construction.

3 Case studies

The validation uses diverse imbalanced datasets, including highly imbalanced, high-dimensional, small, and fraud-detection data. Models are evaluated with stratified cross-validation under shared neighborhood settings, while LoRAS-specific manifold parameters are tuned separately.

  • 3.1 Datasets used for validation: The benchmark covers 14 datasets selected for extreme imbalance, high dimensionality, small minority classes, or credit-card fraud detection.The datasets include 11 imbalanced or high-dimensional cases, two small datasets, and one fraud-detection dataset.
  • 3.2 Methodology: The study evaluates classifiers with stratified cross-validation, using 5×10-fold validation generally and 5×3-fold validation for the two smallest datasets.
  • 3.2 Methodology: All oversampling methods use the same neighborhood size per dataset, while LoRAS receives additional tuning for parameters intended to model the minority-class manifold.
  • 3.2 Methodology: LoRAS is tuned with dataset- and model-specific manifold settings, including Naff, embedding, and perplexity, while generated samples are balanced toward majority-class counts.
  • 3.2 Methodology: The final classifier set comprises k-nearest neighbors, linear-kernel SVM, and logistic regression after a pilot comparison with additional models.

4 Results

Across 14 datasets, LoRAS achieved the strongest average performance across the two reported measures, while subgroup analyses showed strengths and trade-offs by dataset characteristics. Statistical tests further supported differences in F1-Score, with more mixed evidence for Balanced accuracy.

  • Overall results: LoRAS achieved the best average Balanced accuracy and F1-Score across all 14 datasets.SMOTE extensions often improved one metric while compromising the other, whereas LoRAS improved both relative to SMOTE.
  • Datasets with high imbalance ratio: For highly imbalanced datasets, LoRAS produced the highest F1-Score and Balanced accuracy among the evaluated oversampling models.Its advantage was especially pronounced for the credit fraud dataset, while webpage and letter image showed improved Balanced accuracy with minimal F1-Score compromise.
  • High dimensional datasets: On high-dimensional datasets, LoRAS had the best average F1-Score but the second-best Balanced accuracy, narrowly behind Borderline-2 SMOTE.Even excluding webpage, LoRAS retained the best average F1-Score and a marginally lower Balanced accuracy than the leading methods.
  • Small datasets: For the two small datasets, LoRAS ranked first on F1-Score for ar1 and Balanced accuracy for ar2, while remaining competitive on the other metric.It also performed well on abalone and arrhythmia, which have few minority-class data points.
  • Statistical analysis: Wilcoxon tests found LoRAS F1-Scores significantly different from every compared oversampling method, while Borderline-2 SMOTE and ADASYN did not differ significantly for Balanced accuracy.The authors therefore conclude that LoRAS’s overall results differ significantly from the compared algorithms, emphasizing F1-Score as a reliable imbalanced-classification metric.
  • Statistical analysis: Effect-size analysis gave LoRAS higher W+ values for both metrics against every compared convex-combination oversampler, with high F1-Score improvement and mostly high Balanced accuracy improvement.The Balanced accuracy improvement was medium against Borderline-2 SMOTE.

5 Discussion

The discussion frames LoRAS as a local-manifold oversampling estimator that is unbiased like SMOTE but has lower variance when the feature count exceeds two. Its outlier-sparing sampling behavior is associated with better local-manifold approximation and higher F1 performance.

  • Mathematical comparison: LoRAS and SMOTE are both unbiased estimators of the local minority-class mean, but LoRAS has lower estimator variance when |F| > 2.The analysis models local minority-class data with a shifted t-distribution and represents LoRAS using affine combinations of |F| shadowsamples, versus two samples for SMOTE.
  • Mathematical comparison: LoRAS generates samples from multiple shadowsamples, whereas SMOTE forms affine combinations of two minority-class samples.The comparison treats SMOTE as |F| = 2 independently of the number of features.
  • Ozone visualization: In the ozone PCA visualization, SMOTE and ADASYN oversample heavily near outliers, while LoRAS samples sparsely and more uniformly in that neighborhood.Borderline-SMOTE and SVM-SMOTE reduce but do not eliminate sampling near the outliers.
  • Ozone visualization: The ozone comparison links oversampling behavior near outliers with model performance: LoRAS has the highest average F1-Score, while SMOTE and ADASYN have the lowest.The passage states that Borderline-SMOTE and SVM-SMOTE improve F1-Score relative to SMOTE and ADASYN while sampling less near the outlier neighborhood.

6 Conclusions

LoRAS produces comparatively balanced classification performance across the evaluated convex-combination oversampling methods. The authors attribute this to fewer majority-class misclassifications and identify a scope boundary for heterogeneous image datasets.

  • Conclusion: LoRAS produces comparatively balanced average Balanced Accuracy and F1-Score among the evaluated convex-combination oversampling techniques.The conclusion reports improved Balanced accuracy and F1-Score for machine-learning classification models on tabular high-dimensional, highly imbalanced datasets.
  • Conclusion: LoRAS usually reduces majority-class misclassifications while accepting a reasonably small compromise in minority-class misclassifications.
  • Scope: The authors state that convex-combination strategies including LoRAS may have limited scope for heterogeneous image-based imbalanced datasets.
  • Comparison: LoRAS and SMOTE both address imbalance by generating new data points independently of the majority-class distribution, enabling a direct comparison between them.

Supplementary data

The supplementary data provides per-dataset, per-model F1-Score and Balanced accuracy results, with selected model results based on the highest average F1-Score criterion.

  • Supplementary results: The supplementary results report F1-Scores and Balanced accuracies for logistic regression, SVM, and k-nearest-neighbor models across named datasets.The supplied tables cover abalone 19 and arrhythmia, with separate metric tables for each model.
  • Selection criterion: For each dataset, the analysis marks the machine-learning model with the highest average F1-Score for inclusion in further analysis.

Dataset: isolet

The isolet and letter image supplementary tables report F1-Scores and Balanced accuracies for logistic regression, SVM, and k-nearest-neighbor models.

  • Isolet: The isolet tables provide F1-Scores and Balanced accuracies for logistic regression, SVM, and k-nearest-neighbor models.
  • Letter image: The letter image tables provide F1-Scores and Balanced accuracies for logistic regression, SVM, and k-nearest-neighbor models.

Dataset: mammography

The supplied tables report F1-Scores and Balanced accuracies for the mammography and scene datasets across repeated cross-validation runs. Results are organized by logistic regression, SVM, and KNN models.

  • Mammography: Mammography results are reported for logistic regression, SVM, and KNN using F1-Scores and Balanced accuracies.Each model has separate tables for the two metrics under 5 runs of 10-fold cross validation.
  • Scene: Scene results are reported for logistic regression, SVM, and KNN using F1-Scores and Balanced accuracies.Each model has separate tables for the two metrics under 5 runs of 10-fold cross validation.

Dataset: ozone level

The supplied tables report F1-Scores and Balanced accuracies for the ozone level and webpage datasets across repeated cross-validation runs. Results are organized by logistic regression, SVM, and KNN models.

  • Ozone level: Ozone level results are reported for logistic regression, SVM, and KNN using F1-Scores and Balanced accuracies.Each model has separate tables for the two metrics under 5 runs of 10-fold cross validation.
  • Webpage: Webpage results are reported for logistic regression, SVM, and KNN using F1-Scores and Balanced accuracies.Each model has separate tables for the two metrics under 5 runs of 10-fold cross validation.

Dataset: wine quiality

The supplied tables report F1-Scores and Balanced accuracies for the wine quiality and yeast ml8 datasets across repeated cross-validation runs. Results are organized by logistic regression, SVM, and KNN models.

  • Wine quiality: Wine quiality results are reported for logistic regression, SVM, and KNN using F1-Scores and Balanced accuracies.Each model has separate tables for the two metrics under 5 runs of 10-fold cross validation.
  • Yeast ml8: Yeast ml8 results are reported for logistic regression, SVM, and KNN using F1-Scores and Balanced accuracies.Each model has separate tables for the two metrics under 5 runs of 10-fold cross validation.

Dataset: yeast me2

The supplied tables report repeated cross-validation metrics for yeast me2 and credit fraud. Yeast me2 uses logistic regression, SVM, and KNN, while credit fraud uses logistic regression and random forest.

  • Yeast me2: Yeast me2 results are reported for logistic regression, SVM, and KNN using F1-Scores and Balanced accuracies.Each model has separate tables for the two metrics under 5 runs of 10-fold cross validation.
  • Credit fraud: Credit fraud results are reported for logistic regression and random forest using F1-Scores and Balanced accuracies.The supplied tables identify separate metric tables for each model under 5 runs of 10-fold cross validation.
Loading 1908.08346v4…