Source-linked AI summary
MetaSAug: Meta Semantic Augmentation for Long-Tailed Visual Recognition
Shuang Li, Kaixiong Gong, Chi Harold Liu, Yulin Wang, Feng Qiao, Xinjing Cheng
TL;DR
Long-tailed data leaves minority classes with too few examples for reliable semantic covariance estimation, limiting augmentation diversity. MetaSAug learns class-wise covariance through meta-updates on a balanced validation set, and experiments validate its effectiveness across long-tailed benchmarks. It also improves compatibility with established long-tailed losses and backbones.
Problem
Minority classes provide insufficient data for ISDA to estimate reasonable covariance matrices, while standard training degrades on long-tailed distributions.
Method
MetaSAug uses meta-learning to update class-wise covariance by minimizing loss on a small balanced validation set, producing semantic augmentation for minority classes.
Results
MetaSAug consistently outperforms competing methods across long-tailed benchmarks and backbone networks, with learned covariance exhibiting more balanced singular-value distributions.
Takeaways & Limitations
MetaSAug provides a versatile augmentation module that complements re-weighting and can be combined with focal loss, LDAM, and other long-tailed methods.
Takeaways & Limitations
ISDA performs unsatisfactorily in long-tailed settings because scarce minority-class data cannot produce reasonable covariance matrices.
Abstract
from arXiv · showhide
Real-world training data usually exhibits long-tailed distribution, where several majority classes have a significantly larger number of samples than the remaining minority classes. This imbalance degrades the performance of typical supervised learning algorithms designed for balanced training sets. In this paper, we address this issue by augmenting minority classes with a recently proposed implicit semantic data augmentation (ISDA) algorithm, which produces diversified augmented samples by translating deep features along many semantically meaningful directions. Importantly, given that ISDA estimates the class-conditional statistics to obtain semantic directions, we find it ineffective to do this on minority classes due to the insufficient training data. To this end, we propose a novel approach to learn transformed semantic directions with meta-learning automatically. In specific, the augmentation strategy during training is dynamically optimized, aiming to minimize the loss on a small balanced validation set, which is approximated via a meta update step. Extensive empirical results on CIFAR-LT-10/100, ImageNet-LT, and iNaturalist 2017/2018 validate the effectiveness of our method.
1. Introduction
Long-tailed class imbalance makes standard supervised learning perform poorly, while conventional input augmentation cannot provide enough diversity for minority classes. MetaSAug learns class-wise semantic directions through meta-learning, optimizing augmentation using a balanced validation set.
- Long-tailed data contains many fewer minority-class samples, and standard cross-entropy training severely degrades network performance.
- Conventional augmentation operates on inputs, so minority classes with few examples receive inherently limited augmented-sample diversity.
- ISDA augments deep features by translating them along meaningful, class-preserving semantic directions, but minority data cannot reliably estimate its covariance matrices.
- MetaSAug learns more meaningful class-wise covariance by updating augmentation to minimize loss on a small balanced validation set.
- MetaSAug is a plug-in module that can be combined with prior methods, including focal loss and LDAM loss.
- Experiments on artificially and naturally long-tailed CIFAR, ImageNet, and iNaturalist datasets demonstrate the method’s effectiveness.
2. Related Work
Related work addresses long-tailed recognition through re-sampling, re-weighting, multi-stage learning, meta-learning, knowledge transfer, and data augmentation. MetaSAug differs from ISDA by using meta-learning to learn class-wise covariance for semantic augmentation.
- Re-sampling: Re-sampling balances data by over-sampling minority classes or under-sampling majority classes, but can respectively cause over-fitting or weaken majority feature learning.
- Re-weighting: Re-weighting assigns class- or instance-level weights, including inverse-frequency weighting, effective-number weighting, meta-class-weight, margins, focal loss, and related methods.
- Representation and classifier learning: Some methods separate representation learning from classifier re-balancing, while BBN unifies the stages through cumulative learning.
- Meta-learning and head-to-tail knowledge transfer: Meta-learning methods learn sample weights, and other approaches transfer knowledge from head classes to tail classes.
- Data augmentation: Traditional augmentation alters inputs, whereas semantic augmentation changes feature semantics; MetaSAug uses meta-learning to learn covariance that ISDA cannot estimate reliably for minority classes.
3. Method
MetaSAug learns class-wise covariance matrices for semantic augmentation through meta-learning, using balanced validation loss to improve minority-class augmentation in long-tailed training. It combines weighted training objectives with online updates of classifier parameters and covariance matrices.
- Implicit Semantic Data Augmentation: ISDA performs semantic augmentation by translating deep features along directions sampled from class-wise Gaussian covariance estimates, but scarce minority data yields unsatisfactory covariance matrices.The method targets class identity-preserving transformations in a lower-complexity deep feature space.
- Meta Semantic Augmentation: MetaSAug learns more meaningful class-wise covariance matrices by minimizing loss on a small balanced validation set.The covariance matrices are treated as training hyperparameters optimized through a meta-learning objective.
- Meta Semantic Augmentation: Class-conditional weights down-weight majority-class losses so the weighted objective does not mainly augment majority classes.The weights are defined as ϵc ≈ (1 − β)/(1 − β^nc), with β recommended as (N − 1)/N.
- Meta Semantic Augmentation: An online strategy updates classifier parameters and covariance matrices through one-step loops instead of two nested optimization loops.The classifier update uses step size α, while the covariance update uses step size γ.
- Discussion: MetaSAug produces a relatively balanced singular-value distribution for the rare-class covariance matrix, retaining more important principal components than ISDA.The authors associate this distribution with potentially more semantic directions and diverse transformation vectors.
4. Experiment
Experiments evaluate MetaSAug across artificially and naturally long-tailed datasets, showing consistent gains from meta-learned semantic augmentation across settings, losses, and backbone networks.
- CIFAR-LT results: MetaSAug consistently improves the performance of the three evaluated basic losses on long-tailed CIFAR-10 and surpasses mixup and re-weighting methods.The gains remain stable when the imbalance factor is 10, indicating no reported degradation under a relatively balanced setting.
- CIFAR-LT results: 3.56%: MetaSAug with LDAM loss exceeds Meta-class-weight with LDAM loss on long-tailed CIFAR-100.MetaSAug achieves the best results in each reported loss group.
- Analysis: Removing re-weighting or meta-learning causes performance drops, while fixed-ISDA meta-weighting methods do not reach MetaSAug’s notable accuracy gains.The ablation supports the usefulness of both components, particularly meta-learning for covariance estimation.
- Analysis: MetaSAug consistently outperforms competing methods across different backbone networks and can generate label-preserving semantic variations for rare classes, including truck.The visualized transformations alter colors, backgrounds, and object shapes while preserving class identity.
5. Conclusion
MetaSAug tackles long-tailed visual recognition through minority-class semantic augmentation, learning class-wise covariance matrices with meta-learning. Extensive benchmark experiments validate its effectiveness and versatility.
- MetaSAug learns appropriate class-wise covariance matrices to augment minority classes and improve classifier learning.
- The method addresses long-tailed recognition from a data augmentation perspective, an area the paper identifies as insufficiently explored.
- MetaSAug is orthogonal to existing long-tailed methods such as LDAM and focal loss.
- Extensive experiments across several benchmarks validate MetaSAug's effectiveness and versatility.