Source-linked AI summary
Generative Dual Adversarial Network for Generalized Zero-shot Learning
He Huang, Changhu Wang, Philip S. Yu, Chang-Dong Wang
TL;DR
Generalized zero-shot learning must classify images from both seen and unseen classes, but prior methods often model only one direction between visual and semantic spaces. GDAN combines generation, regression, and similarity discrimination with cyclic consistency and dual adversarial learning, and reports strong performance across four benchmark datasets, including improved unseen-class results and balanced seen/unseen accuracy.
Problem
Generalized zero-shot learning seeks to classify images from both seen and unseen classes, where unseen classes have no training images.
Method
GDAN combines a generator, regressor, and discriminator to unify semantic-to-visual generation, visual-to-semantic mapping, and metric learning under cyclic consistency and dual adversarial losses.
Results
Across four benchmark datasets, GDAN effectively classifies unseen classes while preserving high seen-class accuracy and balancing performance between seen and unseen classes.
Takeaways & Limitations
Jointly training the three components helps GDAN balance seen- and unseen-class accuracy in generalized zero-shot learning.
Abstract
from arXiv · showhide
This paper studies the problem of generalized zero-shot learning which requires the model to train on image-label pairs from some seen classes and test on the task of classifying new images from both seen and unseen classes. Most previous models try to learn a fixed one-directional mapping between visual and semantic space, while some recently proposed generative methods try to generate image features for unseen classes so that the zero-shot learning problem becomes a traditional fully-supervised classification problem. In this paper, we propose a novel model that provides a unified framework for three different approaches: visual-> semantic mapping, semantic->visual mapping, and metric learning. Specifically, our proposed model consists of a feature generator that can generate various visual features given class embeddings as input, a regressor that maps each visual feature back to its corresponding class embedding, and a discriminator that learns to evaluate the closeness of an image feature and a class embedding. All three components are trained under the combination of cyclic consistency loss and dual adversarial loss. Experimental results show that our model not only preserves higher accuracy in classifying images from seen classes, but also performs better than existing state-of-the-art models in in classifying images from unseen classes.
1. Introduction
Generalized zero-shot learning requires classifying images from both seen and unseen classes, unlike conventional zero-shot learning, which tests only on unseen classes. GDAN unifies visual-semantic mapping, semantic-visual generation, and metric learning through cyclic consistency and dual adversarial learning.
- Problem: Generalized zero-shot learning tests images from the combined seen and unseen label space, removing the assumption that the model knows whether a class was seen during training.Conventional zero-shot learning tests only on unseen classes.
- Existing Approaches: Existing approaches include visual-to-semantic embedding, semantic-to-visual generation, and metric learning based on visual-semantic similarity.These categories are illustrated in Figure 1.
- Proposed Model: GDAN unifies the three approaches with a generator, regressor, and discriminator trained through cyclic consistency and dual adversarial losses.The generator produces image features from class embeddings, the regressor reconstructs class embeddings, and the discriminator scores feature compatibility.
- Results: Experiments on four benchmark datasets show effective classification of unseen classes while preserving high accuracy on seen classes.The authors also report component analysis and visualization of synthetic unseen-class features.
2. Related Work
Related work spans GAN-based synthesis and dual learning, while prior zero-shot methods use generative models with known limitations. GDAN adapts cyclic consistency to combine bidirectional mappings without requiring two generative networks.
- Generative Adversarial Networks: GAN-based methods were developed for image synthesis but can suffer from unstable training and mode collapse.WGAN, WGAN-GP, and LS-GAN were proposed to mitigate these issues.
- Dual Learning: Dual learning jointly trains a task with its inverse, and prior vision models such as Cycle-GAN and DualGAN use cyclic consistency.The paper borrows cyclic consistency while requiring a different architecture for zero-shot learning.
- Zero-shot Learning: Cyclic consistency cannot be transferred directly from two-generator models because semantic representations are fixed and semantic generation may produce diverse features.The paper therefore introduces a novel architecture for incorporating cyclic consistency into zero-shot learning.
- Zero-shot Learning: Prior generative zero-shot methods use variational autoencoders, but CVAE-ZSL is described as suffering from prior collapse.SE-GZSL is another variational-autoencoder approach for generalized zero-shot learning.
3. Proposed Model
GDAN unifies generator, regressor, and discriminator components for generalized zero-shot learning, linking semantic-to-visual generation, visual-to-semantic mapping, and metric learning through cyclic and adversarial training.
- Problem Definition and Notations: Generalized zero-shot learning trains on seen-class image features but must classify test images from both seen and unseen classes.The seen and unseen label sets are disjoint during training, while the test classifier predicts over their union.
- Model Overview: GDAN combines semantic-to-visual generation, visual-to-semantic regression, and metric learning in one framework.Its generator produces visual features from class embeddings, the regressor maps visual features back to semantic features, and the discriminator evaluates visual-semantic compatibility.
- Feature Generation Network: The CVAE generator can produce visual features conditioned on class embeddings, avoiding direct generation of blurry raw images.The model generates image features rather than raw images because sharp object attributes matter for zero-shot learning.
- Regressor Network: The generator and regressor form a dual learning framework connected by cyclic consistency, with the regressor trained on real and generated visual features.The regressor receives real training features and CVAE-generated features, and maps them to corresponding class embeddings.
- Discriminator Network: The discriminator scores visual-semantic compatibility and uses generated-feature, mismatched-class, and regressor-produced pairs in adversarial training.The discriminator replaces predefined similarity measures with a learned compatibility score, while the CVAE and regressor learn from it through dual adversarial loss.
- Full Objective and Training Procedure: GDAN pretrains the CVAE, then jointly trains the model adversarially; unseen-class prediction uses generated samples combined with training data and a 1-NN classifier.The full objective weights its component losses with hyperparameters, and synthetic unseen-class features support classification over seen and unseen classes.
4. Experiments
Experiments on four benchmark datasets evaluate GDAN under generalized zero-shot learning, where both seen and unseen classes appear at test time. Results indicate that the unified framework improves unseen-class recognition while maintaining a balance between seen- and unseen-class accuracy.
- Datasets and Settings: GDAN is evaluated on SUN, CUB, aPY, and AWA2 using publicly available image features and class embeddings.The evaluation uses average per-class top-1 accuracy and harmonic mean for seen- and unseen-class performance.
- Results: On SUN, GDAN achieves the highest accuracy for both seen and unseen classes among the compared baselines.The authors report a significant performance gain and a large margin over deep embedding, generative, and metric-learning models.
- Results: 49.5% harmonic mean on CUB is 2.5% higher than RelationNet, despite slightly lower seen-class accuracy than SP-AEN.GDAN also achieves the highest unseen-class accuracy on CUB.
- Results: On AWA2, GDAN outperforms SP-AEN and PSR-ZSL on unseen-class and harmonic-mean accuracy, while DEM and RelationNet have less than 1% higher harmonic mean.GDAN’s unseen-class accuracy exceeds those methods by 2.7%.
- Results: On aPY, GDAN achieves 16% higher unseen-class accuracy than previous models and the highest harmonic mean while retaining high seen-class accuracy.The authors characterize aPY as harder because its train-test attribute-variance cosine similarity is 0.58.
- Component Analysis: Jointly training the CVAE, Regressor, and Discriminator improves synthetic unseen-class samples and boosts the Regressor and Discriminator compared with independent training.The CVAE alone is comparable to several baselines, whereas the Regressor and Discriminator are weak alone; combining components produces major gains.
- Effect of the Number of Synthetic Samples: For CUB and SUN, unseen-class accuracy generally increases with more synthetic samples and saturates at about 400, whereas AwA2 benefits little.The improvement is larger for CUB and SUN, which have more unseen classes.
- Qualitative Evaluation: t-SNE visualizations show synthetic features close to real features for many AwA2 and aPY classes, including densely overlapping aPY regions.The rat class is identified as a failure case in AwA2.
5. Conclusion
GDAN unifies visual–semantic mapping, semantic–visual mapping, and metric learning in a generative dual adversarial framework for generalized zero-shot learning. Experiments on four benchmark datasets support its effectiveness in balancing seen- and unseen-class accuracy, while component analysis and visualization support joint training benefits and high-quality synthetic features.
- GDAN unifies visual→semantic mapping, semantic→visual mapping, and metric learning through a generator, regressor, and discriminator.The generator produces visual features from class embeddings, the regressor maps visual features back to embeddings, and the discriminator evaluates their closeness.
- The generator and regressor learn from each other through dual learning, while both learn from the discriminator through dual adversarial loss.
- Experiments on four benchmark datasets demonstrate GDAN’s effectiveness in balancing accuracy between seen and unseen classes.
- Component analysis shows that all three components benefit from joint training.
- Visualization indicates that GDAN generates high-quality visual features for unseen classes.