PaperlayerView exact revision on arXiv

Source-linked AI summary

Effective Class-Imbalance learning based on SMOTE and Convolutional Neural Networks

Javad Hassannataj Joloudari, Abdolreza Marefat, Mohammad Ali Nematollahi, Solomon Sunday Oyelere, Sadiq Hussain

arXiv:2209.00653v2cs.LG

TL;DR

Imbalanced datasets can bias binary classifiers toward majority classes, complicating reliable classification. This paper combines resampling with DNN and CNN models, finding that SMOTE-Normalization-CNN delivers superior performance, including 99.08% accuracy.

  • Problem

    Imbalanced datasets create a long-standing classification challenge because majority-class samples can substantially outnumber minority-class samples.

  • Method

    The study evaluates DNN and CNN classifiers combined with oversampling and undersampling methods, including SMOTE-generated minority samples.

  • Results

    The mixed SMOTE-Normalization-CNN model outperformed related methodologies across several metrics, achieving 99.08% accuracy in Table 11.

  • Takeaways & Limitations

    The authors conclude that the proposed methodology is more robust for imbalanced binary classification.

  • Takeaways & Limitations

    Applying the proposed model to other applications and real datasets remains future work.

Abstract

from arXiv · show

Imbalanced Data (ID) is a problem that deters Machine Learning (ML) models for achieving satisfactory results. ID is the occurrence of a situation where the quantity of the samples belonging to one class outnumbers that of the other by a wide margin, making such models learning process biased towards the majority class. In recent years, to address this issue, several solutions have been put forward, which opt for either synthetically generating new data for the minority class or reducing the number of majority classes for balancing the data. Hence, in this paper, we investigate the effectiveness of methods based on Deep Neural Networks (DNNs) and Convolutional Neural Networks (CNNs), mixed with a variety of well-known imbalanced data solutions meaning oversampling and undersampling. To evaluate our methods, we have used KEEL, breast cancer, and Z-Alizadeh Sani datasets. In order to achieve reliable results, we conducted our experiments 100 times with randomly shuffled data distributions. The classification results demonstrate that the mixed Synthetic Minority Oversampling Technique (SMOTE)-Normalization-CNN outperforms different methodologies achieving 99.08% accuracy on the 24 imbalanced datasets. Therefore, the proposed mixed model can be applied to imbalanced binary classification problems on other real datasets.

1. Introduction

The introduction frames class imbalance as a persistent classification challenge that can bias models toward majority classes. It motivates combining resampling with DNNs/CNNs and presents SMOTE-Normalization-CNN as the leading proposed methodology across extensive evaluations.

  • Problem setting: Class imbalance occurs when majority-class samples greatly outnumber minority-class samples, creating a longstanding challenge in supervised classification.The introduction notes that such datasets occur across diverse real-world research domains.
  • Existing approaches: Classical classifiers may maximize overall accuracy while neglecting the minority class, motivating established oversampling and undersampling solutions.The introduction surveys random undersampling, Tomek Links, One-Sided Selection, Near Miss, random oversampling, and SMOTE.
  • Motivation: The paper proposes two deep-learning methods mixed with resampling techniques to address imbalanced datasets, whose CNN applications remain insufficiently investigated.The motivation follows limitations of existing oversampling and undersampling approaches and the limited investigation of CNNs on imbalanced data.
  • Contributions: SMOTE-Normalization-CNN reportedly outperforms related methods across accuracy, precision, recall, G-Mean, specificity, AUC-ROC, and Kappa.The paper identifies this mixed methodology as superior in its comparison with related research works.

2. Related work

Related work addresses imbalanced classification through ensemble learning, resampling, deep generative models, optimization, and CNN-specific strategies. Studies emphasize handling class overlap, noise, majority-class bias, and the detrimental effects of imbalance on classification performance.

  • Learning and optimization approaches: Ensemble, optimization, and generative approaches addressed imbalance through AUC-aware AdaBoost, genetic sampling-ratio selection, discriminator-generator-classifier frameworks, and bagging with undersampling.The genetic-algorithm methods achieved the best AUC against random sampling methods on 14 imbalanced datasets.
  • Comparative analyses: Comparative studies examined how resampling affects multiclass neural networks, evaluation metrics, training duration, class overlap, and imbalance severity across diverse datasets.One study compared class overlap and imbalance across broad ranges, while another reported four patterns linking resampling to metrics and training time.
  • Resampling methods: Researchers developed resampling methods spanning undersampling, oversampling, hybrid sampling, class decomposition, and density-based approaches for reducing imbalance and class overlap.Examples include Naïve Bayes-guided undersampling, CDSMOTE, density-based undersampling, and density-based hybrid sampling.
  • SMOTE-based methods: Several studies refined SMOTE by removing noise or adapting synthetic-sample generation through local outlier factors, Kalman filtering, clustering, and kernel density estimation.DeepSMOTE additionally combines a SMOTE-based encoder/decoder structure with an improved loss function.
  • CNNs and imbalanced data: CNN experiments on MNIST, CIFAR-10, and ImageNet found that imbalanced data harms classification performance and indicated that oversampling should reduce the imbalance sufficiently.The study also considered undersampling, two-phase training, and thresholding.

3. Methodology

The methodology addresses class imbalance using datasets from KEEL, breast cancer, and Z-Alizadeh Sani benchmark repositories. It comprises preprocessing, classification, and model analysis, alongside resampling through oversampling and undersampling.

  • 3. Methodology: The study applies its methods to KEEL, breast cancer, and Z-Alizadeh Sani datasets to address class imbalance.
  • 3. Methodology: Figure 2 provides an overview of the proposed methodology.
  • 3. Methodology: The methodology consists of preprocessing, classification, and analysis of the resulting models.
  • 3. Methodology: Resampling balances imbalanced data by adding minority samples through oversampling or removing majority samples through undersampling.

1) Random Over-Sampling (ROS)

Random over-sampling (ROS) balances imbalanced datasets by regenerating minority-class samples until class distributions tend toward balance. It is a simple, non-heuristic method without intelligent decision boundaries.

  • Random Over-Sampling (ROS): ROS increases minority-class samples until the dataset’s class distributions tend toward balance.The method addresses imbalance by regenerating samples from the minority class.
  • Random Over-Sampling (ROS): ROS is non-heuristic and does not construct intelligent decision boundaries.It is described as the first and simplest method in this field.
  • Random Over-Sampling (ROS): ROS is typically applied until the imbalance is excluded from the dataset.

2) Synthetic Minority Oversampling Technique

SMOTE balances highly imbalanced datasets by increasing minority-class samples through synthetic-data generation rather than simple duplication. Random Under-Sampling instead removes randomly selected majority-class samples, which can discard valuable decision-boundary information.

  • Synthetic Minority Oversampling Technique: SMOTE balances highly imbalanced datasets by generating synthetic samples for the minority class through interpolation.Its synthetic generation differs from simple multiplication to help avoid overfitting.
  • Random Under-Sampling: Random Under-Sampling balances class distributions by randomly deleting samples from the majority class.Because informative samples may be removed, RUS can lose decision-boundary information.

2) Tomek Links

Tomek Links (TL) is an undersampling technique that identifies closely paired samples from different classes near class boundaries. Removing majority-class samples in these pairs increases class separation and improves dataset balance.

  • 2) Tomek Links: Tomek Links identify pairs of nearby samples belonging to different classes and contiguous to the class boundary.A Tomek Link is defined using a distance-based condition between two samples.
  • 2) Tomek Links: Removing majority-class samples in Tomek Links increases the distance between classes and balances the dataset.One sample may be noisy, or the pair may represent a borderline region.
  • 2) Tomek Links: Figure 6 illustrates how Tomek Links reduce the number of majority-class samples.

3) One-Sided Selection (OSS)

One-Sided Selection (OSS) is an undersampling technique that combines TL with the Condensed Nearest Neighbor Rule. It preserves all minority samples while removing redundant majority samples to address class imbalance.

  • 3) One-Sided Selection (OSS): OSS combines TL with the Condensed Nearest Neighbor Rule as an undersampling strategy for imbalanced datasets.Its distance function δ(x, y) is required to satisfy the conditions for being a TL.
  • 3) One-Sided Selection (OSS): OSS leaves minority-class samples completely intact.
  • 3) One-Sided Selection (OSS): OSS filters redundant majority-class samples through a modification of the Condensed Nearest Neighbor Rule.The method’s operation is illustrated in Fig. 7.

4) NearMiss

The study introduces NearMiss as an informed undersampling method and evaluates normalized ANN-based classifiers under repeated randomized training. Its proposed DNN and CNN models use layered architectures, with the CNN incorporating Focal Loss to address class imbalance.

  • 4) NearMiss: NearMiss uses K-nearest neighbors and mean distances to remove majority-class samples intelligently rather than randomly.The method includes NearMiss-1, NearMiss-2, and NearMiss-3 variants.
  • 4) NearMiss: The input data are normalized to produce feature distributions between zero and one.
  • 4) NearMiss: Models are trained and evaluated for 100 runs using randomly shuffled train-test splits and 2000 training epochs per run.
  • 4) NearMiss: The proposed DNN classifier combines fully connected layers with activation, batch normalization, dropout, and a single-neuron output layer.
  • 4) NearMiss: The proposed CNN has two one-dimensional convolutional and two fully connected layers with ReLU activations, using Focal Loss to address class imbalance.Focal Loss adds a modulating factor α(1 − p_t)^γ, where γ > 0 is tunable.

4. Experimental Results

The experiments evaluated six sampling-model combinations across three imbalanced classification datasets using eight performance metrics. The mixed SMOTE-NORM.-CNN model achieved the best AUC value.

  • Implementation details: Model parameters were optimized with Adam at a learning rate of 0.001, while focal loss used alpha 0.25 and gamma 2.DNN and CNN hyperparameters are described in Tables 2 and 3.
  • Datasets and protocol: The experiments used KEEL, breast cancer, and Z-Alizadeh Sani datasets containing different imbalanced classification tasks.Table 4 provides the dataset descriptions and imbalance ratios.
  • Datasets and protocol: Datasets were randomly shuffled, split into training and testing sets, and used to train models for 2000 epochs across 100 runs.The repeated runs were used for model training and evaluation.
  • Classification results: Average accuracy, precision, recall, F1-score, G-Mean, specificity, AUC, and kappa were reported for six models across three imbalanced datasets.The datasets were drawn from the KEEL repository, breast cancer, and Z-Alizadeh Sani classification tasks.
  • Classification results: The mixed SMOTE-NORM.-CNN model achieved the best AUC value across the evaluated models and datasets.ROC plots based on the best AUC scores are shown in Figures 18(a)-(z).

5. Discussion

The discussion frames imbalanced binary classification as a challenge because performance can decline and models may fail when minority and majority classes are unevenly represented. It presents DNN- and CNN-based methods combining data resampling, normalization, and focal loss to address these challenges.

  • Imbalanced datasets can reduce classification performance and produce failures, while minority samples often represent more significant interests than majority samples.
  • The proposed methods combine DNNs and CNNs with undersampling or oversampling techniques, including RUS, Tomek Links, OSS, Near Miss, ROS, and SMOTE.These methods are applied during data preprocessing.
  • The approach also normalizes datasets and uses focal loss when training the neural-network models.Normalization is intended to improve performance, while focal loss is incorporated into model training.

SMOTE + NORM. + CNN

SMOTE+NORM+CNN was the strongest reported model, achieving near-perfect performance across the evaluated imbalanced datasets and outperforming related methods on the Z-Alizadeh Sani dataset.

  • SMOTE + NORM. + CNN: 99.08% accuracy, 99.09% precision, 99.08% sensitivity, 99.09% F1-score, 99.08% G-Mean, 99.03% specificity, 99.08% AUC, and 98.92% kappa were achieved by SMOTE+NORM+CNN.The study identifies SMOTE+NORM+CNN as the most robust model.
  • SMOTE + NORM. + CNN: 99.00% recall, 99.00% G-Mean, and 98.98% F1-score were attained by SMOTE+NORM+CNN on 16 imbalanced datasets.Table 12 compares these results with DB_HS and DB_US methods using SVM or RF.
  • SMOTE + NORM. + CNN: 98.57% accuracy, 98.58% recall, 98.57% F1-score, 98.58% precision, 98.42% specificity, and 99.14% AUC established SMOTE+NORM+CNN’s dominance over other studies.This comparison was conducted on the imbalanced Z-Alizadeh Sani dataset.

6. Conclusion and future work

The paper concludes that its proposed methodology is more robust for addressing imbalanced binary-classification data. Future work will apply it to intrusion and fraud detection and combine other deep-learning architectures with metaheuristic algorithms on additional real datasets.

  • Conclusion: The proposed methodology is reported as more robust for addressing imbalanced datasets in binary classification.The study considered oversampling and undersampling techniques including SMOTE, TL, OSS, NearMiss, ROS, and RUS.
  • Future work: Future work will apply the proposed model to intrusion detection and fraud detection.
  • Future work: Future work will also evaluate different deep-learning architectures combined with metaheuristic algorithms on other real datasets.
Loading 2209.00653v2…