PaperlayerView exact revision on arXiv

Source-linked AI summary

Identifying and Compensating for Feature Deviation in Imbalanced Deep Learning

Han-Jia Ye, Hong-You Chen, De-Chuan Zhan, Wei-Lun Chao

arXiv:2001.01385v4cs.LGcs.CVstat.ML

TL;DR

Class-imbalanced ConvNets perform poorly on minor classes, and the paper investigates why. It identifies feature deviation between training and test examples and proposes class-dependent temperatures to compensate; CDT achieves promising results on benchmark datasets.

  • Problem

    ConvNets trained on class-imbalanced data perform poorly on minor classes, where training examples are relatively scarce.

    Source: §1 · pp. 1–2

  • Method

    The paper analyzes feature deviation between minor-class training and test features and uses class-dependent temperatures (CDT) during training to compensate for its effect.

    Source: §1 · pp. 2–3

  • Results

    CDT achieves promising results on benchmark datasets.

    Source: §9 · p. 13

  • Takeaways & Limitations

    The paper concludes that feature deviation plays a crucial role in ConvNet over-fitting to minor-class data, while it is rarely observed with traditional methods using pre-defined features.

    Source: §9 · p. 13

    The evaluation assumes class-balanced test data or uses average per-class accuracy.

    Source: §2 · p. 3

Abstract

from arXiv · show

Classifiers trained with class-imbalanced data are known to perform poorly on test data of the "minor" classes, of which we have insufficient training data. In this paper, we investigate learning a ConvNet classifier under such a scenario. We found that a ConvNet significantly over-fits the minor classes, which is quite opposite to traditional machine learning algorithms that often under-fit minor classes. We conducted a series of analysis and discovered the feature deviation phenomenon -- the learned ConvNet generates deviated features between the training and test data of minor classes -- which explains how over-fitting happens. To compensate for the effect of feature deviation which pushes test data toward low decision value regions, we propose to incorporate class-dependent temperatures (CDT) in training a ConvNet. CDT simulates feature deviation in the training phase, forcing the ConvNet to enlarge the decision values for minor-class data so that it can overcome real feature deviation in the test phase. We validate our approach on benchmark datasets and achieve promising performance. We hope that our insights can inspire new ways of thinking in resolving class-imbalanced deep learning.

1 INTRODUCTION

The paper argues that class-imbalanced ConvNets over-fit minor classes because training and test features deviate, pushing test examples toward low decision values. It proposes class-dependent temperatures (CDT) to compensate for this effect and reports promising benchmark performance.

1 INTRODUCTION

  • The paper identifies a distinct explanation for poor minor-class ConvNet performance: over-fitting, rather than the major-class dominance commonly invoked to motivate re-weighting and re-sampling.It reports that those approaches may be ineffective for ConvNets and may aggravate minor-class over-fitting.Source: §1 · pp. 1–3
  • Feature deviation between training and test features is more pronounced for minor classes and makes learned linear classifiers poorly suited to separating their test instances.The deviation grows as the number of training instances decreases.Source: §1 · p. 2
  • Minor-class test features move toward lower decision values, helping explain ConvNet over-fitting; simply increasing minor-class influence may worsen the problem.Unlike the under-fitting described for traditional methods using pre-defined features, ConvNets can fit limited minor-class training data but generalize poorly.Source: §1 · p. 2
  • CDT simulates feature deviation during training by dividing class decision values by temperatures set inversely to class training counts, encouraging larger minor-class decision values.CDT compensates for feature deviation; it does not reduce it.Source: §1 · pp. 2–3
  • CDT achieves superior performance compared with existing methods on benchmark datasets.The paper also reports that CDT outperforms or is on a par with state-of-the-art approaches in most cases.Source: §1 · p. 3

2 RELATED WORK

The section surveys re-sampling and cost-sensitive approaches to class imbalance, then distinguishes deep-learning over-fitting from traditional under-fitting and relates feature deviation to prior work.

2 RELATED WORK

  • Re-sampling changes the training distribution, while cost-sensitive methods adjust misclassification costs through instance weights or class-dependent losses.Common re-sampling strategies over-sample minor classes, under-sample major classes, or synthesize minor-class examples; weighting schemes include reciprocal-count and effective-number approaches.Source: §2 · p. 3
  • Prior research also studies imbalanced feature embeddings, two-stage training, and decoupled feature-extractor and classifier training.The authors say their section 3 analysis supports observations that re-weighting and re-sampling may hurt feature quality.Source: §2 · p. 3
  • Prior work reports that under-fitting is common in traditional methods, whereas over-fitting appears prevalent in end-to-end deep learning on imbalanced data.The authors describe the latter as a distinct empirical pattern, contrasting it with earlier work on under-fitting and over-scaling minor-class influence.Source: §2 · p. 3
  • The authors identify feature deviation between training and test instances of the same class as an explanation for minor-class over-fitting, distinguishing their analysis and CDT approach from related observations.They note that also observed a training/test feature-distribution discrepancy, but from its effect on classification boundaries.Source: §2 · p. 3
  • Unlike domain adaptation, this work assumes training and test data come from the same domain and attributes feature deviation to end-to-end ConvNet training on imbalanced data.The text reports no deviation trend with balanced-data ERM or pre-defined features.Source: §2 · p. 3

3 THE FEATURE DEVIATION PHENOMENON

The section finds that end-to-end ConvNets over-fit minor classes and links this pattern to training/test feature deviation. It also reports that naively increasing minor-class influence does not resolve the problem, unlike traditional classifiers’ under-fitting pattern.

3.1 Background and notation

  • The section defines the ConvNet classifier using a feature extractor and class-specific linear classifiers, and evaluates long-tailed CIFAR-10 with 5,000 versus 50 examples at the class extremes.The experiment uses ResNet-32 and uniformly sampled instances for SGD.Source: §3.1 · p. 4, §3.2 · p. 4

3.2 Learning with class-imbalanced data

  • With imbalanced CIFAR-10, ConvNet training accuracy is about 100% across classes, but test accuracy drops sharply for minor classes.The authors interpret this gap as over-fitting to minor-class data.Source: §3.2 · p. 4

3.3 Analysis on the effect of feature deviation

  • Feature-mean distance grows as class training data decrease, and test features shift toward lower diagonal decision values and often toward major-class regions.The diagonal decision-value drop is strongest for minor classes; some off-diagonal values increase, corresponding to misclassification into major classes.Source: §3.3 · pp. 4–5
  • The authors explain minor-class over-fitting by the mismatch between training and test feature regions, which makes classifiers learned on training features poorly suited to minor-class test features.The section measures class feature-mean distance using sub-sampled training and test feature sets.Source: §3.3 · p. 4

3.4 Scaling up minor classes increases deviation

  • Re-weighting and re-sampling neither reduce feature deviation nor improve test accuracy, and may aggravate deviation or reduce feature quality.With nearest-center classification, re-sampling remains behind other methods even when test feature means are used as centers.Source: §3.4 · p. 5, Table 1 · p. 5

3.5 Traditional vs. end-to-end training

  • With pre-defined features, logistic regression shows no feature-deviation trend but under-fits minor classes in both training and testing.The authors contrast this with end-to-end ConvNets, for which re-weighting alleviates traditional-model under-fitting but not ConvNet over-fitting.Source: §3.5 · pp. 5–6

4 CLASS-DEPENDENT TEMPERATURES (CDT)

The paper introduces class-dependent temperatures (CDT) to compensate for feature deviation by simulating its effect during training. In the reported analysis, CDT balances test decision values and improves average test accuracy.

4 CLASS-DEPENDENT TEMPERATURES (CDT)

  • CDT reduces training decision values by class-dependent temperatures, with larger temperatures for classes having fewer training instances, to encourage larger learned decision values.The method simulates the effect of test-time feature deviation during training; it does not directly prevent feature deviation.Source: §4 · p. 6
  • The test phase keeps the usual classification rule; CDT is not applied again because test data already undergo real feature deviation.The method seeks to compensate for reduced test diagonal decision values rather than prevent feature deviation.Source: §4 · p. 6

4.1 Analysis on CDT

  • On test data, CDT yields more balanced diagonal decision values across classes despite their drop for minor classes, and the authors report higher accuracy for minor classes and all classes on average.The analysis uses CDT with γ = 0.3 and points to Table 1 and Figure 4 for accuracy results.Source: §4.1 · p. 6

4.2 Discussion and comparison to related work

  • The authors describe CDT as complementary to related methods because its motivation is feature-deviation analysis and its decision-value adjustment differs from theirs.They contrast its motivation with margin-based generalization bounds, label-distribution shift, and dominant-label bias.Source: §4.2 · p. 6

5 EXPERIMENTAL SETUPS

The experiments evaluate class-imbalanced learning across four image datasets, with implementation choices, CDT hyperparameter selection, and comparisons against established baselines.

5.1 Datasets

  • Experiments use CIFAR-10, CIFAR-100, Tiny-ImageNet, and iNaturalist, constructing long-tailed or step-imbalanced training sets while leaving test or validation sets unchanged and balanced.The imbalance ratio is defined using the largest and smallest class counts; iNaturalist is already long-tailed.Source: §5.1 · pp. 6–7

5.2 Implementation Details

  • CIFAR experiments use ResNet-32 and default to 200 training epochs, while Tiny-ImageNet and iNaturalist use ResNet-18 and ResNet-50, respectively.Some step-imbalanced CIFAR experiments run for 300 epochs; Tiny-ImageNet runs for 400 epochs and iNaturalist for 180.Source: §5.2 · p. 7

5.3 Hyperparameter γ

  • CDT’s γ is selected from dataset-specific ranges using a held-out training split with three instances per class, excluding classes with fewer than three instances.The search ranges are [0.0, 0.4] for CIFAR-10, [0.0, 0.2] for CIFAR-100 and Tiny-ImageNet, and [0.0, 0.1] for iNaturalist.Source: §5.3 · p. 7

5.4 Baselines

  • Comparisons include ERM, re-sampling, re-weighting, and methods including LDAM with DRW, BBN, ReMix, BALMS, and LogitAdjust.Table 2 reports test accuracy on long-tailed CIFAR-10 and CIFAR-100 and distinguishes two training-strategy blocks.Source: §5.4 · pp. 7–8, Table 2 · p. 8

6 EXPERIMENTAL RESULTS

Across CIFAR, Tiny-ImageNet, and iNaturalist experiments, CDT is competitive with or outperforms baselines in many settings; its γ trades improved minor-class test accuracy against reduced major-class accuracy.

6.1 Results on CIFAR-10 and CIFAR-100

  • CDT outperforms baselines in many CIFAR-10/-100 settings and is best or competitive under a stronger training strategy; combining it with DRW improves accuracy in some cases.Against methods with similar algorithm designs, CDT attains higher accuracy in most settings.Source: §6.1 · p. 8

6.2 Results on Tiny-ImageNet

  • On Tiny-ImageNet long-tailed and step-imbalanced settings with ρ = 10 or 100, CDT is on a par with the state of the art.The reported results are validation accuracy; the tables cover long-tailed and step imbalance separately.Source: §6.2 · p. 8, Table 4 · p. 8, Table 5 · p. 9

6.3 Results on iNaturalist

  • On iNaturalist, which has 8,142 classes and many scarce-instance classes, CDT outperforms many baselines and is on a par with the state of the art.The authors note that methods using batch sizes larger than 128 may contribute to performance differences.Source: §6.3 · p. 8, Table 6 · p. 9

6.4 Ablation studies

  • In CDT’s γ ablation, increasing γ improves test accuracy on minor classes but gradually decreases major-class accuracy in both training and testing, as shown in Figure 7.Source: §6.4 · p. 8

6.5 Different training objectives and classifiers

  • With linear classifiers, CDT outperforms the other training objectives by a notable margin, while nearest-center classification mostly beats linear classifiers except when paired with CDT.These comparisons cover ERM, re-sampling, re-weighting, and CDT with linear or nearest-center classifiers.Source: §6.5 · pp. 8–9

7 EXPERIMENTAL ANALYSES

The analyses track class-wise accuracy and feature deviation, compare the effects of imbalanced data on features and classifiers, and examine how CDT behaves across datasets and visualizations.

7.1 Training and testing along epochs

  • CDT does not remove the biased trends in classifier norms or feature deviation, but it makes training and test accuracy more balanced across classes.With ERM, major-class training accuracy quickly approaches 100%, while minor classes show greater feature deviation; balanced CIFAR-10 displays similar trends in statistics.Source: §7.1 · p. 9, Figure 8 · p. 10, Figure 9 · p. 10

7.2 Features vs. classifiers

  • Table 8 indicates that imbalanced data harms ConvNet features more than linear classifiers: imbalanced features with balanced classifiers perform worse than balanced features with imbalanced classifiers.The authors identify feature deviation from imbalanced training as a fundamental issue to resolve.Source: §7.2 · p. 10, Table 8 · p. 10

7.3 Results using traditional machine learning

  • On long-tailed CIFAR-10 and CIFAR-100, traditional logistic regression with pre-defined features under-fits minor classes under ERM, while re-weighting alleviates the problem and improves overall accuracy.The comparison uses Figure 10’s training and test accuracy results.Source: §7.3 · p. 10, Figure 10 · p. 11

7.4 Effects of learning with class-imbalanced data

  • On long-tailed CIFAR-100 with ρ = 100, minor classes show larger feature deviation and low test accuracy despite near-100% training accuracy; CDT compensates for deviation and improves their test accuracy.The figure caption says CDT compensates for feature deviation but does not directly reduce it.Source: §7.4 · p. 11, Figure 11 · p. 11
  • On Tiny-ImageNet and the re-split iNaturalist set, CDT likewise compensates for feature deviation and leads to higher test accuracy.The iNaturalist analysis uses 1,462 classes after retaining classes with at least 47 training instances and moving 22 instances per retained class to validation.Source: §7.4 · p. 11, Figure 12 · p. 12, Figure 13 · p. 12

7.5 Feature visualization

  • t-SNE visualizations show feature deviation for minor classes across training objectives, while CDT separates classes better and keeps same-class training and test instances closer than different classes.The plots sample 25 training and 25 test instances per class.Source: §7.5 · p. 12, Figure 14 · p. 12

8 DISCUSSIONS

The discussion distinguishes under-fitting in traditional machine learning from over-fitting in imbalanced deep learning, explains how CDT differs from re-weighting and post-calibration, and attributes feature deviation to end-to-end training with few minor-class examples.

8.1 Imbalanced learning for deep learning and traditional machine learning

  • Traditional methods with pre-defined features tend to under-fit minor classes, whereas ERM-trained ConvNets already over-fit them, so re-weighting or over-sampling may further fit minor-class data.The discussion cautions that the two learning settings have different reasons for poor minor-class accuracy.Source: §8.1 · pp. 12–13

8.2 CDT vs. re-weighting and post-calibration

  • CDT divides each class classifier weight by its temperature during training, enlarging minor-class decision values to compensate for test-time drops from feature deviation; re-weighting instead scales the loss.Post-calibration such as τ-normalization adjusts decision values at testing rather than training.Source: §8.2 · p. 13

8.3 The reason for feature deviation

  • The studied datasets have similar training and test image distributions, and feature deviation is mainly attributed to end-to-end ConvNet training with few minor-class images.The authors report no feature-deviation trend when features are trained with balanced, sufficient data or pre-defined from other datasets.Source: §8.3 · p. 13

9 CONCLUSION

Feature deviation plays a crucial role in ConvNet overfitting on minor-class data, pushing test examples toward low-decision-value regions. The paper proposes class-dependent temperatures (CDT) to compensate and reports promising benchmark results.

9 CONCLUSION

  • Feature deviation plays a crucial role in ConvNet overfitting on minor-class data by pushing test examples toward regions of low decision values.The paper notes that this phenomenon is rarely observed in traditional machine-learning approaches using pre-defined features.Source: §9 · p. 13
  • Class-dependent temperatures (CDT) are proposed in ConvNet learning to compensate for the effect of feature deviation.Source: §9 · p. 13
  • CDT is simple to implement and achieves promising results on benchmark datasets.Source: §9 · p. 13

Related papers

Loading 2001.01385v4…