Source-linked AI summary
A Generative Adversarial Approach for Zero-Shot Learning from Noisy Texts
Yizhe Zhu, Mohamed Elhoseiny, Bingchen Liu, Xi Peng, Ahmed Elgammal
TL;DR
Zero-shot learning must recognize unseen classes despite sparse or absent visual data, while text descriptions are noisy and conventional methods often rely on one-to-one visual-semantic mappings. The paper uses a conditional GAN to generate pseudo visual features from Wikipedia text, with a fully connected noise-suppression layer and visual pivot regularization. It reports consistent state-of-the-art performance across multiple zero-shot tasks and benchmarks.
Problem
Sparse visual data for unseen concepts and noisy textual descriptions challenge conventional zero-shot learning approaches based largely on visual-semantic embedding.
Method
GAZSL generates pseudo visual features for unseen classes from Wikipedia-derived text using a conditional GAN, an additional fully connected layer, and visual pivot regularization.
Results
GAZSL consistently outperforms state-of-the-art methods across zero-shot recognition, generalized zero-shot learning, and zero-shot retrieval on several benchmarks.
Takeaways & Limitations
Generated pseudo data converts zero-shot learning into supervised classification while preserving intra-class diversity and inter-class discrimination.
Takeaways & Limitations
Sparse training data makes GANs alone struggle to model high-dimensional visual-feature distributions, motivating explicit visual-pivot supervision.
Abstract
from arXiv · showhide
Most existing zero-shot learning methods consider the problem as a visual semantic embedding one. Given the demonstrated capability of Generative Adversarial Networks(GANs) to generate images, we instead leverage GANs to imagine unseen categories from text descriptions and hence recognize novel classes with no examples being seen. Specifically, we propose a simple yet effective generative model that takes as input noisy text descriptions about an unseen class (e.g.Wikipedia articles) and generates synthesized visual features for this class. With added pseudo data, zero-shot learning is naturally converted to a traditional classification problem. Additionally, to preserve the inter-class discrimination of the generated features, a visual pivot regularization is proposed as an explicit supervision. Unlike previous methods using complex engineered regularizers, our approach can suppress the noise well without additional regularization. Empirically, we show that our method consistently outperforms the state of the art on the largest available benchmarks on Text-based Zero-shot Learning.
1. Introduction
The paper recasts zero-shot learning as imagining visual features for unseen classes from noisy text, then uses those features to train a supervised classifier. GAZSL combines noise suppression and visual-pivot guidance, achieving consistent gains across zero-shot tasks and benchmarks.
- Sparse visual data for many real-world concepts makes conventional object classification infeasible, motivating zero-shot learning.
- GAZSL learns a stochastic one-to-many mapping from textual descriptions to visual features instead of a one-to-one semantic-to-visual projection.Text descriptions can correspond to varied visual instances, motivating adversarial training with stochasticity.
- The model hallucinates pseudo visual features for unseen classes from Wikipedia-derived semantic representations, converting zero-shot learning into supervised classification.The generated features provide intra-class diversity while preserving inter-class discrimination.
- A fully connected layer before the generator suppresses noisy textual information and raises CUB ZSL performance from 40.85% to 43.74%.
- The visual pivot regularizer guides generated features toward discriminative class regions, addressing GAN difficulty with sparse samples and high-dimensional visual features.Without this guidance, generated features can disperse and destroy real-feature cluster structure.
- GAZSL consistently outperforms state-of-the-art methods across zero-shot recognition, generalized zero-shot learning, and zero-shot retrieval.The introduction reports gains of 6.5% on CUB and 5.3% on NAB.
2. Related Work
Prior zero-shot learning work is dominated by visual-semantic embedding, while another line generates pseudo samples to enable supervised classification. Text descriptions are expressive but noisy, creating a motivation for the paper’s approach.
- DAP and ALE established attribute-based zero-shot learning, with ALE framing the task as visual-semantic embedding without attribute-independence assumptions.
- Most subsequent methods project visual and semantic information into a shared embedding space for unseen-class prediction.
- Pseudo-sample methods convert zero-shot recognition into supervised classification, but prior approaches impose distribution assumptions, one-to-one mappings, or dataset-limited synthesis.
- Wikipedia articles are more expressive and distinguishable than class taxonomies but contain superfluous information irrelevant to visual recognition.TF-IDF features are commonly used for this noisy textual representation.
3. Background
GANs generate data through an adversarial generator-discriminator game, while conditional variants incorporate side information such as class labels, text, or images. The paper builds on these ideas and visual-part feature extraction.
- GANs use a generator and discriminator that respectively produce synthetic data and distinguish it from real data.
- WGAN uses Wasserstein distance and weight clipping, while later work replaces clipping with gradient penalty to address pathological behavior.
- Conditional GANs guide generation and discrimination with side information including class labels, texts, or images.
- The proposed model employs an ACGAN-based structure with Wasserstein distance and gradient penalty objectives.
- VPDE-net extracts visual features from detected semantic object parts using ROI pooling over a VGG-based backbone.
4. Methodology
GAZSL models zero-shot learning as generation: noisy class text conditions a GAN that hallucinates visual features, while visual pivots guide class-specific feature statistics. Generated and extracted features support adversarial training and subsequent classification.
- GAZSL defines zero-shot learning with disjoint seen and unseen classes, predicting unseen labels from semantic representations and visual features.
- The model combines a feature extractor, generator, and discriminator to provide real features, synthesize features, and distinguish realism while predicting classes.
- Generator: Noisy class text is compressed through a fully connected layer, concatenated with Gaussian noise, and fed to the generator to produce plausible visual features.
- Discriminator: The discriminator receives extracted or synthesized features and uses separate branches for real-versus-fake discrimination and n-way category classification.
- Adversarial training: The discriminator objective combines Wasserstein terms, a gradient penalty enforcing a Lipschitz constraint, and classification losses for real and synthesized features.
- Visual Pivot Regularization: Visual pivot regularization matches each class’s generated-feature mean to the corresponding real-feature mean, addressing sparse data and high-dimensional feature simulation.
5. Experiments
Experiments evaluate the method on CUB and NAB under two split settings across recognition, generalized recognition, and retrieval. The method consistently outperforms competing approaches, with visual pivot regularization and the GAN both contributing to performance.
- Experiment Setting: Experiments use CUB and NAB, with SCS and SCE splits and comparisons against seven state-of-the-art algorithms.CUB contains 200 bird categories, while NAB contains 1,011 classes before its reorganized 404-class setting.
- Zero-Shot Recognition: 6.5% and 5.3%: the method’s SCS-split advantage over ZSLPP on CUB and NAB, respectively.The comparison concerns conventional zero-shot recognition on the two benchmark datasets.
- Ablation Study: 20.91% and 11.36%: removing visual pivot regularization reduces accuracy on CUB and NAB, respectively.The reported drops are from 43.74% to 22.83% on CUB and from 35.58% to 24.22% on NAB.
- Ablation Study: 2% to 3%: adding the textual fully connected layer generally improves performance across most settings.The layer compacts high-dimensional TF-IDF features to 1000 dimensions and is optimized end-to-end.
- Generalized Zero-Shot Learning: 42.6% and 56.7%: the method’s AUSUC scores exceed the runner-up on CUB and NAB, respectively.The advantage is reported for generalized zero-shot learning and indicates more balanced seen-unseen performance; NAB SCS is an exception where WAClinear overfits.
6. Conclusion
The paper presents a GAN-based approach that imagines visual features from noisy Wikipedia descriptions for zero-shot learning and reports favorable performance across multiple zero-shot tasks.
- The approach uses GANs to imagine visual features from noisy textual descriptions for zero-shot learning.The descriptions come from Wikipedia, and the generated features support recognition of unseen classes.
- A visual pivot regularizer explicitly guides generated imagery samples toward the proper direction.
- Adding a fully connected layer to textual features provides comparable noise suppression.
- The method performs favorably against state-of-the-art methods on multiple zero-shot tasks.