PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Over-sampling Framework for Classifying Imbalanced Data

Shin Ando, Chun-Yuan Huang

arXiv:1704.07515v3cs.LGstat.ML

TL;DR

Class imbalance remains difficult for deep models because conventional synthetic over-sampling is limited on complex structured data. DOS extends over-sampling into CNN deep feature space through supervised synthetic embedding targets and iterative updates, and empirical benchmarks show stronger imbalance handling than existing countermeasures, with additional gains in balanced settings.

  • Problem

    Class imbalance harms classification, while conventional synthetic over-sampling has limited success for complex structured data handled by deep learning models.

  • Method

    DOS jointly trains a CNN representation and classifier using synthetic embedding targets sampled from in-class neighbors, then iteratively updates those targets.

  • Results

    DOS addresses class imbalance more effectively than existing deep-learning countermeasures, with larger improvements under stronger imbalance, and also improves representation learning in balanced settings.

  • Takeaways & Limitations

    Synthetic instances can supervise CNN representation learning as well as classifier learning without substantial architectural modification.

  • Takeaways & Limitations

    Synthetic over-sampling assumes interpolations remain within the original distribution, an assumption that may fail for complex structured data.

Abstract

from arXiv · show

Class imbalance is a challenging issue in practical classification problems for deep learning models as well as traditional models. Traditionally successful countermeasures such as synthetic over-sampling have had limited success with complex, structured data handled by deep learning models. In this paper, we propose Deep Over-sampling (DOS), a framework for extending the synthetic over-sampling method to exploit the deep feature space acquired by a convolutional neural network (CNN). Its key feature is an explicit, supervised representation learning, for which the training data presents each raw input sample with a synthetic embedding target in the deep feature space, which is sampled from the linear subspace of in-class neighbors. We implement an iterative process of training the CNN and updating the targets, which induces smaller in-class variance among the embeddings, to increase the discriminative power of the deep representation. We present an empirical study using public benchmarks, which shows that the DOS framework not only counteracts class imbalance better than the existing method, but also improves the performance of the CNN in the standard, balanced settings.

1 Introduction

Class imbalance harms minority-class performance and can also degrade CNN representation learning, while existing deep-learning countermeasures are often complex and synthetic sampling is difficult for structured data. DOS addresses this by supervising CNN embeddings with synthetic in-class targets and iteratively updating them to improve class distinction without substantial architectural changes.

  • Class imbalance can cause poor minority-class performance and deteriorate CNN representation learning, affecting majority-class performance as well.
  • Synthetic over-sampling methods such as SMOTE interpolate in-class neighbors, but generating synthetic samples for complex structured inputs is not straightforward.
  • DOS extends synthetic over-sampling into the CNN’s acquired deep feature space, using synthetic instances as supervised targets for representation learning.
  • DOS assigns each raw input a class label and a synthetic target sampled from a linear subspace of its in-class embedded neighbors.
  • Multiple synthetic targets augment training examples, while iterative CNN learning and target updates enhance the discriminative power of deep features.
  • The framework jointly learns deep representations and classifiers under class imbalance without substantial architectural modification, supporting applicability across deep learning models.

2 Background

Class imbalance harms minority-class performance and can also degrade CNN representation learning, while existing sampling methods face limitations on complex structured data. Preliminary results motivate deep-representation over-sampling as a broadly applicable alternative.

  • 2.1 Class Imbalance: Class imbalance causes learning algorithms to over-generalize toward majority classes, making minority-class retrieval difficult in practice.The paper frames imbalance as a difference in training-sample counts between classes and notes that minority retrieval is often the primary practical interest.
  • 2.1 Class Imbalance: Synthetic over-sampling methods such as SMOTE interpolate minority-class neighbors, but assume locally linear features and vector-form inputs.These assumptions are generally suitable for traditional models but can fail for complex structured data handled by deep neural networks.
  • 2.1 Class Imbalance: Under-sampling can discard information and require computationally intensive, task- and model-specific analyses and configurations.The discussed prior approach also uses a specialized loss and architecture that do not clearly generalize across classifiers.
  • 2.1 Class Imbalance: The paper explores over-sampling in deep representation space with a basic CNN to retain wide applicability and joint representation-classifier learning.This approach is intended to avoid computationally intensive analyses while preserving cohesive CNN learning.
  • 2.2 Preliminary Results: Preliminary MNIST-back-rotation experiments remove 90 % of four randomly selected digit classes and evaluate minority and majority retrieval separately.The CNN experiments repeat training ten times and report Precision, Recall, and F1-score; deep representations are additionally evaluated with kNN.
  • 2.2 Preliminary Results: Imbalanced training reduces minority precision and affects many majority-class precision or recall measures for both CNN classification and deep-representation kNN.Average results over ten trials support a negative effect on CNN representation learning and classifier training, with influence varying by classifier.

3 Deep Over-sampling Framework

Deep Over-sampling (DOS) re-samples in the CNN’s deep feature space, pairing raw inputs with synthetic in-class targets to supervise both representation and classification. Iterative target updates and weighted local-neighbor losses address imbalance while reducing in-class variance.

  • Deep feature overloading: DOS re-samples in the nonlinear deep feature space acquired by a CNN rather than the raw-input space used by methods such as SMOTE.The framework targets complex, structured data by exploiting learned deep feature instances.
  • CNN architecture: The CNN separates embedding function f in lower layers from classification function g in top layers, with both functions receiving explicit supervision.Training uses class labels together with deep-feature targets for each raw input sample.
  • Deep feature overloading: Each input is paired with its own embedding and k in-class neighbors, forming an overloaded training set with multiple deep-feature instances per raw sample.The overloading parameter k may differ between minority and majority classes because minority classes receive synthetic supplementation.
  • Micro-cluster loss: The embedding micro-cluster loss uses squared errors whose minimum maps f(x) to the mean of the local neighbor set, a synthetic deep-feature point.Neighbor weights can emphasize closer points, with the original embedding receiving the largest weight in the described weighting scheme.
  • Iterative optimization: Local-mean targets overload minority samples and induce smaller in-class variance, which the paper links to better class distinction; iterating with updated embeddings strengthens this process.After CNN training, DOS recomputes targets using the updated representation and repeats the procedure.
  • Parameter selection: Different overloading values can weight minority-class errors more heavily; with kmjr = 0, minority samples account for (kmnr + 1) times more squared errors.This provides an explicit mechanism for placing additional emphasis on the minority class.

4 Empirical Results

DOS was evaluated on five public image datasets across imbalanced and balanced settings, comparing baseline methods, imbalance levels, and overloading parameters. It generally improved class-wise performance, degraded more slowly under imbalance, and remained effective in balanced data.

  • Experimental Settings: The study evaluated DOS on MNIST, MNISTrb, SVHN, CIFAR-10, and STL-10 using CNN-based architectures.Experiments covered baseline comparison, sensitivity to imbalance and k, and balanced-data performance.
  • Baseline Comparison: DOS showed the slowest decline in class-wise recall as the overall reduction rate increased, compared with TL-RS-CSL and LMLE.The baseline comparison used MNIST-back-rotation images and average per-class accuracy.
  • Sensitivity Analysis on Imbalanced Data: DOS exhibited greater advantages over the basic CNN and CNN-CL as imbalance increased across MNIST, MNISTrb, and SVHN.The sensitivity analysis reported precision, recall, F1-score, and AUPRC for minority and majority classes.
  • Sensitivity Analysis on Imbalanced Data: DOS was generally insensitive to k values of 3, 5, and 10, although smaller k slightly favored minority classes and larger k slightly favored majority classes.The analysis used a reduction rate of p = 0.01 across SVHN, CIFAR-10, and STL-10.
  • Evaluation on Balanced Data: DOS consistently improved performance across all balanced datasets, with gains of 0.01 to 0.03 over the basic CNN.The balanced experiment set the over-sampling rate r to 1 and evaluated k = 3, 5, and 10.

5 Conclusion

The paper concludes that DOS jointly trains CNN representations and classifiers in imbalanced settings without substantial architectural modification. Its empirical results indicate stronger performance under class imbalance and improved representation learning in balanced settings.

  • 5 Conclusion: DOS jointly learns the deep representation and classifier without substantially modifying the CNN architecture.Synthetic instances supervise representation learning as well as classifier learning.
  • 5 Conclusion: DOS addresses class imbalance more effectively than existing deep-learning countermeasures, with larger improvements under stronger imbalance.The conclusion also reports improved performance in balanced settings, supporting benefits for representation learning.
  • 5 Conclusion: Improved balanced-setting performance supports DOS’s ability to enhance representation learning and class distinction beyond imbalance correction.The framework uses synthetic instances to supervise the learned deep representation.
Loading 1704.07515v3…