Source-linked AI summary
f-VAEGAN-D2: A Feature Generating Framework for Any-Shot Learning
Yongqin Xian, Saurabh Sharma, Bernt Schiele, Zeynep Akata
TL;DR
Scarce labeled data limits feature generation because existing methods rely on labeled image-feature and attribute pairs and cannot exploit abundant unlabeled samples. The paper proposes f-VAEGAN-D2, combining conditional VAE and GAN components with an unconditional discriminator for unlabeled data. It reports effective performance across any-shot settings and interpretable generated features, while noting detail and explanation-quality issues for high-variation classes.
Problem
Existing feature-generation approaches rely on labeled image-feature and class-attribute pairs, limiting their use of unlabeled data and their ability to model complex distributions.
Method
f-VAEGAN-D2 combines conditional VAE and GAN components with a non-conditional discriminator that learns from unlabeled novel-class features.
Results
The framework improves low-shot learning across inductive and transductive zero-shot and few-shot tasks on five datasets and produces visually and textually interpretable features.
Takeaways & Limitations
The learned features support classifier augmentation for any-shot learning and can be visualized through pixel inversion and explained with class-specific language.
Takeaways & Limitations
Generated images can lack detail or color fidelity for high-interclass-variation classes, while explanations may contain repetitions or unknown words.
Abstract
from arXiv · showhide
When labeled training data is scarce, a promising data augmentation approach is to generate visual features of unknown classes using their attributes. To learn the class conditional distribution of CNN features, these models rely on pairs of image features and class attributes. Hence, they can not make use of the abundance of unlabeled data samples. In this paper, we tackle any-shot learning problems i.e. zero-shot and few-shot, in a unified feature generating framework that operates in both inductive and transductive learning settings. We develop a conditional generative model that combines the strength of VAE and GANs and in addition, via an unconditional discriminator, learns the marginal feature distribution of unlabeled images. We empirically show that our model learns highly discriminative CNN features for five datasets, i.e. CUB, SUN, AWA and ImageNet, and establish a new state-of-the-art in any-shot learning, i.e. inductive and transductive (generalized) zero- and few-shot learning settings. We also demonstrate that our learned features are interpretable: we visualize them by inverting them back to the pixel space and we explain them by generating textual arguments of why they are associated with a certain label.
1. Introduction
The paper addresses shortcomings in feature generation for scarce-label learning by proposing f-VAEGAN-D2, which combines conditional VAE and GAN modeling with unlabeled data use and interpretable generated features.
- Existing feature-generation methods struggle to capture complex distributions, generalize to underrepresented classes, and produce interpretable features.
- f-VAEGAN-D2 generates CNN image features from class embeddings using labeled samples when available and generalizing to unknown concepts without labels.
- The model combines a conditional encoder, shared conditional decoder/generator, conditional discriminator, and non-conditional discriminator.
- The first three networks learn conditional CNN-feature distributions from labeled seen-class data, while the last learns the marginal feature distribution from unlabeled novel-class features.
- Generated features improve low-shot learning across inductive and transductive zero-shot and few-shot settings, while supporting pixel-space inversion and visual explanations.
2. Related Work
Related work covers zero-shot and few-shot learning, transductive use of unlabeled unseen-class data, and generative models for data augmentation. The paper positions its approach as a feature generator operating across inductive and transductive settings.
- Zero-shot Learning: Zero-shot learning predicts unseen classes, while generalized zero-shot learning predicts both seen and unseen classes using class-level semantic embeddings.
- Zero-shot Learning: Transductive zero-shot methods additionally use unlabeled unseen-class data, including approaches based on label propagation, semantic manifolds, or latent-variable estimation.
- Zero-shot Learning: Unlike most transductive approaches that rely on label propagation, this model learns a feature generator from labeled seen-class and unlabeled unseen-class data.
- Few-shot Learning: Few-shot learning trains models with only a few samples, creating overfitting risk and motivating generalization from data-rich classes without learning new parameters.
- Generative Models: Generative models learn data distributions for random sampling and augmentation; GANs use generators and discriminators, while this model combines VAE and GAN advantages with an additional discriminator.
3. f-VAEGAN-D2 Model
f-VAEGAN-D2 unifies conditional feature generation from labeled seen classes with marginal-distribution learning from unlabeled novel-class features. It combines VAE and GAN objectives through shared decoder/generator parameters and uses a second discriminator to support transductive learning.
- Framework scope: The framework addresses inductive and transductive zero-shot settings by generating features for sparsely labeled or unlabeled classes.Inductive training uses labeled seen-class samples, whereas transductive training additionally uses unlabeled samples; inference is unchanged.
- Conditional feature generation: The generator produces CNN features from random noise and class embeddings, while a conditional discriminator scores feature–class compatibility.The generated features can augment linear softmax classifier training.
- Conditional feature generation: The VAE encoder maps feature–condition pairs to latent representations, and the decoder reconstructs features from latent variables and conditions.The conditional prior is modeled as N(0, 1), with reconstruction specified through the decoder.
- VAE–GAN combination: VAE and GAN components share decoder/generator parameters, combining their losses with γ controlling the relative weighting.The model motivates this combination by treating VAE and GAN objectives as complementary for complicated multimodal distributions.
- Unlabeled-feature modeling: The non-conditional discriminator D2 distinguishes real and generated unlabeled novel-class features and learns their feature manifold.Its WGAN objective captures marginal CNN-feature structure, providing signals for generating transferable features across classes.
- Implementation: The implementation uses MLP generators and discriminators with two fully connected layers and 4096 hidden units.Gaussian noise and class embeddings are concatenated as generator input, while discriminators concatenate image features with class embeddings.
4. Experiments
The experiments evaluate f-VAEGAN-D2 across zero-shot, generalized zero-shot, few-shot, generalized few-shot, large-scale, and interpretability settings. Results show strong performance from combining VAE-GAN feature generation with unlabeled-data learning, while visual quality degrades for classes with high inter-class variation.
- Experimental settings: The model is evaluated on five zero-shot datasets, including medium-scale fine-grained, coarse-grained, and large-scale fine-grained settings.The datasets are CUB, FLO, SUN, AWA2, and ImageNet, which contains more than 14 million images and 21K classes.
- Zero-shot learning: 61.0% ZSL accuracy makes VAE-GAN stronger than VAE at 59.1% and GAN at 58.4% in the inductive ablation.With unlabeled samples, the VAE-GAN model reaches 71.1% ZSL and 63.2% GZSL accuracy.
- Zero-shot learning: 72.9% on CUB, 70.4% on FLO, 65.6% on SUN, and 70.3% on AWA are reported for fine-tuned inductive ZSL.For fine-tuned transductive ZSL, the reported results are 82.6% on CUB, 95.4% on FLO, 72.6% on SUN, and 89.3% on AWA.
- Large-scale experiments: The model significantly improves over CLSWGAN on ImageNet across ZSL and GZSL 2H, 3H, and All splits.The experiments use per-class word2vec representations because ImageNet classes lack human-annotated attributes.
- Few-shot learning: With one real sample per class, the model nearly reaches the full accuracy obtained with 20 samples per class in CUB and FLO.With 10 or 20 samples per class, methods perform similarly because a softmax classifier using real ResNet-101 features reaches state-of-the-art performance.
- Few-shot learning: 60.6% versus 54.7% at 1 shot and 70.3% versus 66.8% at 2 shots show transductive gains over PMN w/G* on ImageNet FSL.In GFSL, the model remains comparable with the state-of-the-art despite PMN w/G* using meta-learning.
- Interpretability: Synthetic features produce class-specific visual explanations and reconstructions with essential classification attributes, including color and distinctive flower structures.Generated images from synthetic and real features have similar quality, including for unseen classes whose real features were not observed.
- Interpretability: High inter-class variation is associated with poorer reconstructions, including missing detail, color mismatches, repeated phrases, and unknown words.Examples include Balloon Flower and Sweat Pea visualizations and explanations.
5. Conclusion
The framework synthesizes CNN image features from class embeddings while incorporating both labeled seen-class data and unlabeled unseen-class data. It supports broad any-shot evaluation and produces features that are visually and linguistically interpretable.
- The framework synthesizes CNN image features from class embeddings to address scarce labeled training data.
- A conditional VAE-GAN model is combined with a non-conditional discriminator that learns from unlabeled unseen-class data.The additional discriminator learns the unseen-class manifold and improves generalization of the feature generator.
- The generated features are effective across ZSL, GZSL, FSL, and GFSL tasks on CUB, FLO, SUN, AWA, and ImageNet.
- Inverting generated features produces detailed images, while generated language explanations are class-specific.