Source-linked AI summary
Low-shot Learning via Covariance-Preserving Adversarial Augmentation Networks
Hang Gao, Zheng Shou, Alireza Zareian, Hanwang Zhang, Shih-Fu Chang
TL;DR
Low-shot learning must avoid over-fitting and catastrophic forgetting while augmenting novel classes from very few examples. CP-AAN learns novel-class latent distributions by translating related base examples and preserving their covariance. It generates realistic, diverse features and substantially improves ImageNet performance over the state of the art.
Problem
Low-shot methods need to augment novel classes without relying on arbitrary base classes or naive finite-example generation that fails to model novel-class latent distributions.
Method
CP-AAN uses adversarial imbalanced set-to-set translation to learn each novel-class distribution from related base classes while preserving base covariance.
Results
The method significantly outperforms the state of the art on ImageNet and generates realistic, diverse data for low-shot learning.
Takeaways & Limitations
Covariance-preserving augmentation produces generated distributions that more closely resemble real-data diversity and improves low-shot recognition.
Takeaways & Limitations
Direct adversarial training cannot explicitly incorporate base intra-class variance and can produce collapsed mappings, while cycle consistency remains weak under limited observations.
Abstract
from arXiv · showhide
Deep neural networks suffer from over-fitting and catastrophic forgetting when trained with small data. One natural remedy for this problem is data augmentation, which has been recently shown to be effective. However, previous works either assume that intra-class variances can always be generalized to new classes, or employ naive generation methods to hallucinate finite examples without modeling their latent distributions. In this work, we propose Covariance-Preserving Adversarial Augmentation Networks to overcome existing limits of low-shot learning. Specifically, a novel Generative Adversarial Network is designed to model the latent distribution of each novel class given its related base counterparts. Since direct estimation of novel classes can be inductively biased, we explicitly preserve covariance information as the `variability' of base examples during the generation process. Empirical results show that our model can generate realistic yet diverse examples, leading to substantial improvements on the ImageNet benchmark over the state of the art.
1 Introduction
Low-shot fine-tuning can overfit novel classes and forget base classes, motivating feature augmentation. CP-AAN translates related base examples into novel classes while preserving covariance, and achieves state-of-the-art ImageNet performance.
- Fine-tuning with few novel examples can cause catastrophic forgetting and over-fitting, degrading performance on the broader base-class set.
- Feature Hallucination and Feature Imagination augment novel classes but respectively suffer from poor generalization or mode collapse.
- CP-AAN models each novel-class latent distribution from related base classes using imbalanced set-to-set translation and explicit covariance preservation.
- The method systematically evaluates objective functions and achieves state-of-the-art performance on the challenging ImageNet benchmark.
2 Related Works
Prior low-shot approaches use meta-learning or feature augmentation to adapt classifiers, while GAN-based translation offers a route to diverse feature generation. The central challenge is modeling novel-class distributions from extremely limited observations without unrelated transfers or collapsed modes.
- Meta-learning methods tune base classifiers for few-shot adaptation, while feature augmentation improves both conventional and generalized low-shot evaluation settings.
- Conditional GAN translation can generate diverse augmented features from related novel-base class pairs, but not all examples are semantically translatable.
- Estimating latent distributions from a handful of observations is biased and inaccurate, motivating hierarchical, sequential, or mixture-based generation strategies.
- Feature Hallucination randomly applies mappings between same-class base examples, whereas the proposed motivation favors semantically similar pairs and preserves base intra-class variance.
3 Imbalanced Set-to-set Translation
The paper formulates low-shot feature augmentation as imbalanced set-to-set translation and develops CP-AAN to generate novel-class features while preserving related base-class covariance.
- 3 Imbalanced Set-to-set Translation: CP-AAN models low-shot augmentation as translating abundant base features into novel categories using a learned mapping between disjoint label spaces.The formulation uses base and novel datasets in a shared D-dimensional semantic space, with only a handful of novel observations.
- 3 Imbalanced Set-to-set Translation: Base classes are paired with novel classes through semantic similarity, using either soft scores over all bases or a hard top-k neighborhood mapping.The soft mapping weights base classes by prototype distance, whereas the hard mapping treats the selected k nearest classes equally.
- 3 Imbalanced Set-to-set Translation: Naive c-GAN training cannot incorporate base-class intra-class variance and can collapse synthesized features onto existing observations.These limitations produce unfavorable generation results in the toy experiment.
- 3 Imbalanced Set-to-set Translation: Cycle-consistent variants improve diversity over c-GAN, but can still under- or over-estimate novel-class intra-class variance.The cCyc-GAN and cDeLi-GAN results are shown in Figure 3c and 3d; cycle consistency also stabilizes training, while injected noise supports variability.
- 3 Imbalanced Set-to-set Translation: cCov-GAN preserves covariance from relevant base classes through a covariance-distance objective and improves low-shot generation quality.The model combines adversarial learning for realistic generation, cycle consistency for semantic consistency, and covariance preservation for diversity.
4 Experiments
Experiments on ImageNet evaluate CP-AAN in low-shot and generalized low-shot settings, then analyze its sampling strategy and generation quality. The best model improves over augmentation baselines while producing more diverse, realistic embeddings.
- Experimental Setup: ImageNet evaluation reports mean top-5 accuracy over five trials for conventional LSL and generalized GLSL, with held-out test splits used for final results.LSL tests novel classes only, whereas GLSL tests all categories including base classes.
- Main Results: Almost 2% performance gain over baselines across different K values under both LSL and GLSL settings, with a largest low-shot boost of ~9% over the naive baseline and 2.6% over Feature Imagination.These results use the best CP-AAN model and are reported in Table 1 under the stated experimental settings.
- Ablation Analysis: Applying NBS improves low-shot recognition, while NBS-H is sensitive to k and soft assignment is preferable when computation permits.The ablation measures absolute performance gain over standard batch sampling.
- Generation Quality: The covariance-preserving objective achieves the best hallucination quality because its generated distribution more closely matches real-data diversity.Cycle-consistency and Gaussian-mixture noise improve accuracy and diversity but can under- or over-estimate diversity; underestimation is more damaging to accuracy than overestimation.
- Generation Quality: With K = 5, CP-AAN generates more diverse examples aligned with the latent distribution, whereas Feature Hallucination and Feature Imagination cluster synthesized points together.The qualitative comparison uses t-SNE visualizations with real examples as crosses and synthesized examples as stars.
5 Conclusion
The paper presents CP-AAN as a cyclic-GAN approach for augmenting novel-class data while shaping intra-class variability through related base classes. It reports state-of-the-art ImageNet performance and realistic, diverse generated data across settings.
- Conclusion: CP-AAN augments novel-class data with a cyclic GAN while shaping intra-class variability through similar base classes.The conclusion describes this as the paper’s central approach to low-shot learning.
- Conclusion: The proposed model significantly outperforms the state of the art on ImageNet across various settings.Quantitative and qualitative evaluations support its effectiveness for low-shot learning.
- Conclusion: The method generates realistic and diverse data given very few examples, supporting low-shot learning.The conclusion attributes this assessment to both quantitative and qualitative evaluations.
A Details about Neighborhood Batch Sampling
Neighborhood Batch Sampling selects semantically related base classes for each novel class, using either hard nearest-neighbor assignments or soft weighting over all base classes. The soft strategy is more effective when computational resources permit.
- A Details about Neighborhood Batch Sampling: NBS-H is sensitive to k and treats selected base classes as equally related, which slows convergence and hurts performance.Its lower computational cost comes with an additional hyper-parameter and less flexible similarity weighting.
- A Details about Neighborhood Batch Sampling: NBS-S considers all base classes and weights them by softmax scores over learned similarity metrics, whereas NBS-H selects k nearest classes with uniform weights.Both strategies aim to pair each novel class with semantically similar base classes for translation.
- A Details about Neighborhood Batch Sampling: NBS improves low-shot recognition, while NBS-S outperforms NBS-H when sufficient computational resources are available.The ablation supports soft assignment as the preferable choice when its computational cost is acceptable.
B Details about Intermediate GAN Objectives
The paper formulates intermediate variants for imbalanced set-to-set translation with different GAN objectives. These include a conditional GAN objective and a cycle-consistent GAN objective combining adversarial and cycle-consistency losses.
- B Details about Intermediate GAN Objectives: The intermediate variants use distinct GAN objectives for imbalanced set-to-set translation, including conditional and cycle-consistent formulations.The section presents these objectives as alternatives derived for the translation problem.
- B Details about Intermediate GAN Objectives: c-GAN uses a basic minimax adversarial objective for translating between the imbalanced sets.The supplied formulation identifies its objective as L_adv(G_n, D_n, B, N).
- B Details about Intermediate GAN Objectives: cCyc-GAN combines adversarial losses in both translation directions with a cycle-consistency loss weighted by λ_cyc.Its objective includes L_adv(G_n, D_n, B, N), L_adv(G_b, D_b, N, B), and λ_cyc L_cyc(G_n, G_b).
C Details about Computing Subgradient of Ky Fan m-norm
The paper defines the Ky Fan m-norm through the singular values of an m-truncated SVD and uses its sub-differential to derive the needed subgradient. The proof proceeds by rewriting the norm via its sub-differential and substituting that form into Equation 17.
- C Details about Computing Subgradient of Ky Fan m-norm: The Ky Fan m-norm is the sum of the m largest singular values obtained from an m-truncated SVD.For ˜X = UΣV^T, the norm is written as ∥[X]_m∥_* = Σ_i σ_i(˜X).
- C Details about Computing Subgradient of Ky Fan m-norm: The subgradient derivation rewrites the Ky Fan m-norm using its sub-differential set before substituting that representation into Equation 17.This establishes the proof route for computing the subgradient used by the method.