Source-linked AI summary
Multi-modal Cycle-consistent Generalized Zero-Shot Learning
Rafael Felix, B. G. Vijay Kumar, Ian Reid, Gustavo Carneiro
TL;DR
GZSL lacks visual training data for unseen classes, biasing visual-to-semantic mappings and making generated unseen-class representations difficult to constrain. The paper adds a multi-modal cycle consistency regularization to GAN-based feature generation, enforcing reconstruction of original semantic features. Across several public datasets, the approach reports significant ZSL and GZSL improvements and current best results, while GAN training remains challenging at large scale.
Problem
Missing unseen-class visual training data biases visual-to-semantic mappings toward seen classes, while unconstrained generation may produce unrepresentative unseen-class features.
Method
The method regularizes GAN-based visual feature generation with a multi-modal cycle consistency loss that reconstructs each generated feature's original semantic representation.
Results
The approach reports significant improvements in ZSL and GZSL classification across CUB, FLO, SUN, AWA, and ImageNet, with current best results for these datasets.
Takeaways & Limitations
Constraining unseen-class sample generation targets more representative visual features for training GZSL classifiers.
Takeaways & Limitations
GAN-based generation remains difficult to train in unconstrained, large-scale problems, especially as the number of classes grows.
Abstract
from arXiv · showhide
In generalized zero shot learning (GZSL), the set of classes are split into seen and unseen classes, where training relies on the semantic features of the seen and unseen classes and the visual representations of only the seen classes, while testing uses the visual representations of the seen and unseen classes. Current methods address GZSL by learning a transformation from the visual to the semantic space, exploring the assumption that the distribution of classes in the semantic and visual spaces is relatively similar. Such methods tend to transform unseen testing visual representations into one of the seen classes' semantic features instead of the semantic features of the correct unseen class, resulting in low accuracy GZSL classification. Recently, generative adversarial networks (GAN) have been explored to synthesize visual representations of the unseen classes from their semantic features - the synthesized representations of the seen and unseen classes are then used to train the GZSL classifier. This approach has been shown to boost GZSL classification accuracy, however, there is no guarantee that synthetic visual representations can generate back their semantic feature in a multi-modal cycle-consistent manner. This constraint can result in synthetic visual representations that do not represent well their semantic features. In this paper, we propose the use of such constraint based on a new regularization for the GAN training that forces the generated visual features to reconstruct their original semantic features. Once our model is trained with this multi-modal cycle-consistent semantic compatibility, we can then synthesize more representative visual representations for the seen and, more importantly, for the unseen classes. Our proposed approach shows the best GZSL classification results in the field in several publicly available datasets.
1 Introduction
GZSL must classify both seen and unseen classes despite having visual training representations only for seen classes. The paper addresses this by regularizing GAN-generated visual features to reconstruct their original semantic features, improving benchmark performance.
- GZSL trains with semantic features for seen and unseen classes but visual representations only for seen classes, then tests on both class subsets.
- GAN-based methods synthesize visual representations for seen and unseen classes to train a classifier, but unconstrained generation may produce unrepresentative unseen-class features.
- The proposed multi-modal cycle consistency loss minimizes reconstruction error between the semantic input used for synthesis and the semantic feature reconstructed from the generated visual feature.
- The regularization constrains GAN optimization so synthesized visual representations remain semantically compatible and useful for GZSL classifier training.
- The approach reports significant improvements in both GZSL and ZSL classification across CUB, FLO, SUN, AWA, and ImageNet.
2 Literature Review
Prior ZSL and GZSL methods often map visual inputs into semantic space, but missing unseen-class visual data biases this mapping toward seen classes. GAN-based feature synthesis addresses this gap while introducing an unconstrained-generation problem that cycle consistency helps constrain.
- Benchmark methods include compatibility models such as ALE, DEVISE, SJE, ESZSL, and SAE, alongside classifier-based and hybrid approaches.
- Because unseen classes lack visual training data, visual-to-semantic mappings can become biased toward seen-class semantic features, especially for unseen test images.
- GAN methods synthesize visual representations for seen and unseen classes, which can then train a classifier covering both subsets.
- Unconstrained unseen-class synthesis may generate samples far from the actual visual distribution, creating the unpaired-training problem addressed with cycle consistency.
3 Multi-modal Cycle-consistent Generalized Zero Shot Learning
This section defines GZSL and presents a GAN-based feature-generation framework regularized by multi-modal cycle consistency. The model reconstructs semantic features from generated visual representations and supports classifier training for ZSL and GZSL.
- Problem setup: GZSL partitions classes into seen and unseen subsets, using visual representations only from seen classes during training while testing on both subsets.Training includes semantic features for both subsets; unseen-class training samples provide semantic features and labels but not visual representations.
- Baseline model: The f-CLSWGAN baseline generates visual representations from semantic features and Gaussian noise, while a discriminator distinguishes real from synthesized representations.A classification regularizer also encourages generated representations to be discriminative for their class labels.
- Multi-modal cycle consistency: The proposed cycle-consistency loss constrains generated visual representations to reconstruct their original semantic features through a regressor.The loss includes terms for semantic-feature distributions from both seen and unseen classes, when both are used in training.
- Feature generation: The regressor is first pretrained on seen-class image-semantic pairs, after which generator and discriminator training incorporates WGAN and cycle-consistency objectives.Training may use both seen and unseen classes, producing cycle-(U)WGAN, or only seen classes, producing cycle-WGAN.
- Feature generation: A second strategy additionally pretrains a seen-class softmax classifier and combines classification with cycle-consistency training, yielding cycle-CLSWGAN.The final classifier is trained on generated representations from unseen classes for ZSL, or from seen and unseen classes for GZSL.
- Learning and testing: At test time, class prediction uses an input visual representation, while generated features train the final classifier for the selected ZSL or GZSL task.The classifier-training class set is task-dependent: unseen only for ZSL and seen plus unseen for GZSL.
4 Experiments
The experiments evaluate the proposed models on established ZSL/GZSL benchmarks using average per-class top-1 accuracy and cross-validated settings. They use standard splits for four datasets and a carefully selected ImageNet split without overlap between seen and unseen classes.
- Datasets: Experiments cover CUB, FLO, SUN, AWA, and ImageNet under the Xian et al. evaluation setup.CUB, FLO, and SUN are fine-grained datasets, while AWA is coarse; ImageNet uses a 100-class testing split.
- Datasets: ImageNet evaluation uses 100 carefully selected unseen classes with no overlap with the 1K training seen classes.The authors note that existing ImageNet GZSL splits may contain semantically overlapping seen and unseen classes.
- Evaluation Protocol: ZSL and GZSL results are based on average per-class top-1 accuracy.ZSL averages accuracy over unseen classes; GZSL separately evaluates seen and unseen classes, with the seen score denoted by s.
- Implementation Details: The generator is an MLP with one 4096-node LeakyReLU hidden layer and a 2048-node ReLU output layer.The implementation section describes the generator, discriminator, regressor, and hyper-parameter settings used in training.
- Implementation Details: All experiments generate 300 visual representations per class for classifier training.Fewer than 100 representations were not competitive, while more than 200 became competitive but unstable.
- Implementation Details: The study re-implements f-CLSWGAN as a softmax-classifier baseline and compares it with reported results.For ImageNet, direct comparison is limited because the authors use a different split, though similar splits produce similar results.
5 Results
The results section compares cycle-WGAN, cycle-(U)WGAN, and cycle-CLSWGAN with f-CLSWGAN and other baselines on ZSL and GZSL benchmarks. Results are reported for CUB, FLO, SUN, AWA, and ImageNet.
- Benchmark Results: Tables 4 and 5 report GZSL and ZSL results, respectively, for the proposed cycle models and several baseline methods on CUB, FLO, SUN, and AWA.The compared proposed models are cycle-WGAN, cycle-(U)WGAN, and cycle-CLSWGAN, with f-CLSWGAN used as the main baseline.
6 Discussion
The proposed cycle-consistent feature-generation methods improve GZSL and ZSL performance over the baseline, especially on unseen classes, while convergence and dataset scale affect outcomes. The multi-modal cycle consistency regularization reconstructs semantic features and supports state-of-the-art results across several datasets.
- All proposed feature-generation methods outperform the baseline on the unseen GZSL test set.
- 0.7% to more than 4% improvements are achieved in unseen-class top-1 accuracy, with cycle-(U)WGAN best on three of four datasets.
- 1% to 2.2% improvements in harmonic mean are reported over the baseline for GZSL.
- 0.9% to 3.5% improvements over the baseline are obtained for ZSL top-1 accuracy, yielding the best ZSL results on these datasets.
- On ImageNet, baseline and cycle-WGAN accuracy remains low, but the proposed approach produces more accurate ZSL and GZSL classification.
- The reconstruction loss decreases steadily, and cycle-WGAN converges faster than the baseline on three of four datasets.
7 Conclusions and Future Work
The paper regularizes GAN training in GZSL with multi-modal cycle consistency, requiring generated visual representations to map back to their original semantic features. Experiments report improvements over f-CLSWGAN in both GZSL and ZSL, while large-scale GAN training remains an open issue.
- Conclusions: The proposed multi-modal cycle consistency loss constrains generated visual representations by enforcing reconstruction of their original semantic features.The constraint is motivated by cycle consistency loss and targets unconstrained generation from unseen classes.
- Conclusions: Experiments show that the proposed loss improves classification over f-CLSWGAN in both GZSL and ZSL.
- Future Work: GAN-based GZSL remains difficult to train, particularly for unconstrained and large-scale problems.The authors identify these training challenges as priorities for future work.
- Future Work: Future research should study additional regularization approaches and GAN use in large-scale, real-life GZSL datasets and scenarios.