PaperlayerView exact revision on arXiv

Source-linked AI summary

Feature Space Augmentation for Long-Tailed Data

Peng Chu, Xiao Bian, Shaopeng Liu, Haibin Ling

arXiv:2008.03673v1cs.CV

TL;DR

Long-tailed datasets leave rare classes with insufficient coverage, limiting methods that only rebalance existing samples. The paper decomposes features into class-generic and class-specific components, then fuses head-class generic features with tail-class specific features to generate samples online. Across long-tailed CIFAR, ImageNet-LT, Places-LT, and iNaturalist datasets, the method shows effectiveness, with accuracy increasing about 7% in one long-tailed CIFAR-10 setting.

  • Problem

    Rare classes may be too under-represented to recover their distributions, while balanced-loss methods deteriorate when tail samples are insufficient.

  • Method

    The method uses class activation maps to separate class-generic and class-specific features, then fuses head-class generic features with tail-class specific features during online feature-space augmentation.

  • Results

    The method shows effectiveness on long-tailed CIFAR-10/100, ImageNet-LT, Places-LT, and iNaturalist 2017/2018, including about a 7% validation-accuracy increase in one CIFAR-10 setting.

  • Takeaways & Limitations

    Feature-space augmentation provides a way to enhance tail classes by using class-generic information from head classes while preserving tail-class-specific information.

  • Takeaways & Limitations

    The approach assumes deeper features are more linearly separable and that lower-level augmentation can introduce spatial artifacts.

Abstract

from arXiv · show

Real-world data often follow a long-tailed distribution as the frequency of each class is typically different. For example, a dataset can have a large number of under-represented classes and a few classes with more than sufficient data. However, a model to represent the dataset is usually expected to have reasonably homogeneous performances across classes. Introducing class-balanced loss and advanced methods on data re-sampling and augmentation are among the best practices to alleviate the data imbalance problem. However, the other part of the problem about the under-represented classes will have to rely on additional knowledge to recover the missing information. In this work, we present a novel approach to address the long-tailed problem by augmenting the under-represented classes in the feature space with the features learned from the classes with ample samples. In particular, we decompose the features of each class into a class-generic component and a class-specific component using class activation maps. Novel samples of under-represented classes are then generated on the fly during training stages by fusing the class-specific features from the under-represented classes with the class-generic features from confusing classes. Our results on different datasets such as iNaturalist, ImageNet-LT, Places-LT and a long-tailed version of CIFAR have shown the state of the art performances.

1 Introduction

Long-tailed, fine-grained visual datasets contain many rare classes whose limited samples fail to capture within-class variation, weakening standard reweighting and augmentation methods. The paper addresses this missing coverage by generating tail-class samples in feature space from head-class information.

  • Motivation: Fine-grained real-world datasets often have long-tailed class frequencies, with iNaturalist 2017 spanning 5,089 classes from fewer than 10 to more than 1,000 samples.This combination makes rare-category recognition especially challenging.
  • Motivation: Deep neural network performance suffers when rare classes are severely undersampled and their within-class variation is not fully represented.
  • Existing methods: Reweighting, sampling, and balanced-loss methods can improve imbalance but deteriorate when classes are too under-represented to reveal their complete distributions.
  • Approach: The proposed method augments tail classes in feature space using information from head classes rather than relying only on reweighting or resampling.
  • Approach: Class activation maps separate class-specific from class-generic features, which are fused online to create novel high-level samples for tail classes.

2 Related Work

Related approaches address long-tailed learning through balanced optimization, input or feature augmentation, and transfer from head classes. This paper instead decomposes feature space and transfers only class-generic information for online tail-class synthesis.

  • Learning with Balanced Loss: Balanced-loss methods counter skew by resampling data or assigning larger loss weights to tail-class examples.
  • Learning with Balanced Loss: These optimization-based methods are clean to implement but deteriorate when tail samples are insufficient to recover the true distribution.
  • Relation to Balanced Loss: The proposed augmentation can be combined with focal loss, allowing feature synthesis for tail classes alongside higher weights for hard examples.
  • Data Synthesis and Augmentation: Existing augmentation methods generate samples from similar examples, dataset information, transformations, or generative models to compensate for small tail-class sample sizes.
  • Data Synthesis and Augmentation: Unlike methods that train a heavily parameterized function across classes, this approach models feature-space decomposition and formulates samples online.
  • Transfer and Few-Shot Learning: Unlike broad transfer assumptions, the method transfers only class-generic features from head classes while retaining class-specific features for each tail class.

3 The Problem of Long Tail

Long-tailed learning combines an easier imbalance problem with a harder missing-coverage problem: scarce tail data can make the optimal decision boundary ill-defined. The paper therefore separates transferable generic features from class-specific information and uses them to reconstruct tail distributions.

  • Two Reasons of Model Performance Drop: Long-tailed data reduce classification performance through data imbalance and limited coverage of the underlying data distributions.
  • Two Reasons of Model Performance Drop: Sampling and loss weighting can compensate for bias from unequal class sizes and move the decision boundary in the right direction.
  • Two Reasons of Model Performance Drop: When tail data cannot recover their underlying distribution, finding an optimal decision boundary becomes ill-defined.
  • Two Reasons of Model Performance Drop: In CIFAR-10 feature visualizations, an under-represented ship class produces a different boundary from the boundary obtained with sufficient samples.
  • Feature Decomposition: Head-class knowledge may recover missing tail information, but class-generic features must be distinguished from class-specific features that could mislead reconstruction.
  • Class Activation Map and Feature Decomposition: Class activation maps use classifier weights and spatial feature vectors to identify locations important to a target class, enabling separation of generic and specific information.
  • Training Scheme: The proposed training scheme uses Phase-I representation learning and Phase-II online feature-space augmentation for tail classes.

4 Method

The method uses two-phase training and online feature-space augmentation to recover tail-class distributions by combining class-specific tail features with class-generic features from confusing classes. CAMs separate these feature components, while deeper representations are preferred for fusion.

  • Phase-I learns the feature representation and base classifier from all classes; Phase-II generates novel tail-class samples online.
  • Class activation maps separate each image's class-specific and class-generic information using feature maps and adapted classifier weights.
  • Augmented tail samples mix class-specific features from a tail class with class-generic features from confusing classes in feature space.Feature-space fusion avoids artificial edges and potential bias associated with direct pixel-level blending.
  • Online training samples Nt tail images, creates Na augmented samples per real tail sample, and balances them with an equal number of head-class samples.The resulting online batch contains 2Nt(1 + Na) samples.
  • Feature augmentation is applied after feature extraction, with deeper features favored because they are more linearly separable and less prone to spatial artifacts.The implementation selects features immediately before the last average-pooling layer.

5 Experiments

Experiments evaluate the method across long-tailed CIFAR, ImageNet-LT, Places-LT, and iNaturalist datasets. The method improves tail-class and overall performance, with results reported as outperforming competing methods on CIFAR and achieving comparable state-of-the-art performance on ImageNet-LT and Places-LT.

  • Experiments cover simulated long-tailed CIFAR datasets and real or constructed long-tailed ImageNet-LT, Places-LT, and iNaturalist datasets.CIFAR uses imbalance factors {10, 20, 50, 100, 200}; iNaturalist includes 5,089 and 8,142 classes across its 2017 and 2018 versions.
  • About 7% validation accuracy is gained during Phase-II fine-tuning on long-tailed CIFAR-10 with imbalance factor 200 using ResNet-18, while the baseline shows no noticeable change.
  • Tail-class diagonal accuracy improves substantially after Phase-II, and reduced off-diagonal errors outweigh a slight decrease in head-class accuracy.Overall accuracy still increases after fine-tuning.
  • The method outperforms Class-balanced loss, Focal Loss, and an augmentation-based method on both long-tailed CIFAR datasets.
  • Both ordinary cross-entropy and Focal Loss variants achieve performance comparable to the state-of-the-art on ImageNet-LT and Places-LT.The stronger variant differs by dataset: Ours+FL performs better on ImageNet-LT, whereas Ours performs better on Places-LT.
  • The proposed method achieves the best performance in all evaluated iNaturalist settings against class-balanced cross-entropy and class-balanced focal-loss comparisons.
  • Phase-II accuracy peaks around h_r = 0.95 across datasets, while the CIFAR experiments choose the minimum h satisfying h_r ≥ 0.9.Too few head classes limit generic-feature extraction; too few tail classes leave insufficiently represented classes without augmented fine-tuning.
  • Applying augmentation after Block4 yields the best performance across datasets and imbalance factors, whereas shallower features introduce more spatial artifacts.

6 Conclusion

The paper proposes an online feature-space augmentation scheme for long-tailed classification, combining class-generic head-class features with class-specific tail-class features. Results across several long-tailed datasets show the method’s effectiveness.

  • The method combines class-generic features from head classes with class-specific features from tail classes to generate augmented samples online.The scheme targets deep convolutional neural network classifiers trained on long-tailed datasets.

7 Supplementary

Supplementary results report improved learning curves across network architectures and datasets, while feature visualizations show greater separation of class-specific than class-generic features. This supports using head-class generic features to recover tail-class information.

  • Learning curves show significant improvement over conventional data augmentation across network architectures and CIFAR-10 and CIFAR-100.The comparison uses a baseline model trained with conventional ImageNet ResNet data augmentation.
  • 7.1 Feature Space Visualization: Feature decompositions show class-specific features are significantly more separated than class-generic features within the same class.The visualization covers different network architectures on CIFAR-10 and CIFAR-100.
  • 7.1 Feature Space Visualization: The remaining class-generic features from head classes can help recover lost information for tail classes.This conclusion follows the reported feature-space decomposition analysis.
Loading 2008.03673v1…