Source-linked AI summary
Generalized Zero-Shot Learning via Synthesized Examples
Vinay Kumar Verma, Gundeep Arora, Ashish Mishra, Piyush Rai
TL;DR
The paper addresses generalized zero-shot learning, where test examples can come from both seen and unseen classes and conventional models are biased toward seen-class predictions. It uses a feedback-enhanced conditional VAE to synthesize class-specific exemplars for training off-the-shelf classifiers, reporting improved performance across conventional and generalized zero-shot benchmarks. The framework assumes class-attribute vectors for seen and unseen classes and can incorporate unlabeled examples.
Problem
Generalized zero-shot learning is challenging because conventional zero-shot models trained only on seen-class labels are biased toward predicting seen classes when test classes overlap training classes.
Method
A conditional VAE with discriminator-driven attribute feedback generates exemplars for specified seen or unseen class attributes, which train an off-the-shelf classifier.
Results
The model outperforms previous approaches on unseen-class test accuracy and harmonic mean in GZSL, with consistent improvements across conventional ZSL datasets and settings.
Takeaways & Limitations
Synthesized unseen-class examples let the learned classifier use labeled examples from both seen and unseen classes, mitigating seen-class bias in GZSL.
Takeaways & Limitations
The framework assumes class-attribute vectors are available for every seen and unseen class, despite lacking labeled unseen-class examples.
Abstract
from arXiv · showhide
We present a generative framework for generalized zero-shot learning where the training and test classes are not necessarily disjoint. Built upon a variational autoencoder based architecture, consisting of a probabilistic encoder and a probabilistic conditional decoder, our model can generate novel exemplars from seen/unseen classes, given their respective class attributes. These exemplars can subsequently be used to train any off-the-shelf classification model. One of the key aspects of our encoder-decoder architecture is a feedback-driven mechanism in which a discriminator (a multivariate regressor) learns to map the generated exemplars to the corresponding class attribute vectors, leading to an improved generator. Our model's ability to generate and leverage examples from unseen classes to train the classification model naturally helps to mitigate the bias towards predicting seen classes in generalized zero-shot learning settings. Through a comprehensive set of experiments, we show that our model outperforms several state-of-the-art methods, on several benchmark datasets, for both standard as well as generalized zero-shot learning.
1. Introduction
Generalized zero-shot learning addresses the bias of conventional zero-shot models toward seen classes when test examples may come from both seen and unseen classes. The paper proposes synthesizing class-conditioned exemplars with a feedback-driven generative model and using them to train classifiers.
- Zero-shot learning leverages class attributes or textual descriptions to categorize objects from previously unseen classes.
- Most existing approaches are biased toward seen-class predictions because they learn from labeled data only from seen classes.
- Generalized zero-shot learning permits overlap between training and test classes, and prior methods’ accuracies drop significantly in this setting.
- The proposed generative approach synthesizes exemplars for unseen and optionally seen classes, then trains an off-the-shelf classifier on them.
- A conditional VAE coupled with a multivariate-regressor discriminator feeds attribute-prediction loss back to the generator to improve exemplar generation.
- Training classification models with labeled synthesized examples from unseen classes reduces their dependence on seen-class data and mitigates seen-class bias.
- The final classifier directly predicts class labels rather than class attributes, avoiding a separate nearest-neighbor search that can suffer from hubness.
2. Background and Notation
The framework distinguishes seen classes with labeled examples from unseen classes without labeled examples, while assuming class attributes are available for both. It addresses both conventional ZSL and GZSL, focusing experimentally on the latter.
- Seen classes have labeled training examples, whereas unseen classes have no labeled training examples.
- Test examples may come exclusively from unseen classes in conventional ZSL or from both seen and unseen classes in GZSL.
- The paper focuses on GZSL while applying its model to both conventional and generalized settings.
- For every seen or unseen class, the framework assumes that the corresponding class-attribute vector is available.
- ZSL transfers information from seen to unseen classes by leveraging class-attribute information.
- Seen-class training data are represented as feature-vector and class-attribute pairs used to learn a classifier for test examples.
3. The Basic Model
The basic model is a conditional VAE that combines an unstructured latent code with a class-attribute vector to generate class-specific exemplars. A regressor maps generated outputs back to attributes, providing feedback that encourages discriminative generation and supports unlabeled examples.
- The generator is conditioned on a class-attribute vector and an unstructured code, enabling exemplar synthesis for any specified class.
- The architecture assumes the latent code captures class-independent content while the class attribute captures class-specific discriminative information.
- A multivariate regressor maps decoder outputs to class-attribute vectors and is learned jointly with the rest of the model.
- Regressor feedback encourages the generator to produce exemplars that are easier to discriminate.
- The regressor can incorporate unlabeled examples by computing a probability distribution over their class-attribute vectors.
- After training, random latent codes and specified class attributes produce labeled exemplars that train classifiers such as SVMs.
4. The Complete Model Architecture
The model combines a conditional VAE with a regressor that feeds attribute-prediction feedback into exemplar generation. Alternating objectives train the regressor, generator, and encoder to produce class-specific exemplars for downstream classification.
- Architecture: The architecture uses a stochastic VAE encoder and a conditional generator pG(x|z, a), allowing exemplar synthesis by specifying a class-attribute vector.The latent code is paired with the class attributes so generated exemplars can differ across classes.
- Architecture: The regressor maps real or synthesized examples to their corresponding class-attribute vectors, and its backpropagated loss improves generator representativeness.This feedback mechanism is a central difference from a standard conditional VAE.
- Architecture: The regressor can use unlabeled examples by replacing unavailable class attributes with its output distribution p(a|x).This supports semi-supervised use of the model.
- Training procedure: Regressor training combines supervised losses on labeled seen-class examples with unsupervised losses on generated examples using sampled latent codes and attributes.The sampled attributes may come from both seen and unseen classes, and the generator distribution is fixed during this optimization step.
- Training procedure: Generator and encoder training combines VAE reconstruction, attribute-consistency, regularization, and encoder-consistency objectives in the second step of alternating optimization.The encoder also encourages generated exemplars to preserve the relevant latent distribution and disentangle z from a.
- Training procedure: Generated exemplars are intended to match true-data quality closely enough to train the final classification model.The complete generator objective is a weighted combination of the component losses.
5. Related Work
Related ZSL methods learn attribute mappings, shared embeddings, classifier combinations, or class-conditional distributions. Generalized ZSL is harder because seen-class bias and evaluation-protocol differences complicate fair comparisons, while synthesized examples offer a route to classification and domain adaptation.
- ZSL approaches: Existing ZSL methods include instance-to-attribute mappings, shared semantic embeddings, similarity-weighted seen-class classifier combinations, and extrapolated class-conditional distributions.These approaches transfer information from seen to unseen classes through attributes or semantic relationships.
- ZSL approaches: Embedding-based methods commonly use nearest-neighbor search in attribute space, but this approach is associated with the hubness problem.The nearest class-attribute vector determines the predicted class after projection.
- Generalized ZSL: Generalized ZSL is more challenging because training and test classes overlap, creating a bias toward predicting seen classes.Generative methods address this setting by synthesizing labeled unseen-class examples for classifier training.
- Generalized ZSL: Recent generative approaches synthesize examples for seen and unseen classes using class-conditional distribution approximation or adversarial training.These synthesized examples are combined with labeled seen-class examples for generalized zero-shot classification.
- Generalized ZSL: Synthesized unseen-class examples can also support supervised or semi-supervised domain adaptation when seen and unseen class distributions differ.This connects generative ZSL methods to the domain-shift problem.
- Evaluation: Differences in data splits and evaluation protocols can hinder fair comparison, so the experiments follow published guidelines as much as possible.The concern applies across traditional and generalized ZSL evaluations.
6. Experiments
Experiments evaluate SE-GZSL across multiple benchmark datasets and both conventional and generalized zero-shot settings. The results show strong classification performance, reduced seen-class bias, and generated samples that resemble real unseen-class data.
- Experimental setup: The evaluation covers SUN, CUB, AwA1, AwA2, and Imagenet using standard and proposed zero-shot splits.Experiments use ResNet features except for AwA1 and Imagenet, which use VGG19 and GoogLeNet features respectively.
- Evaluation criteria: Average per-class accuracy is used because it reduces bias from classes with more test examples.For GZSL, performance is also summarized by the harmonic mean of seen- and unseen-class accuracies.
- Generalized zero-shot learning: In GZSL, synthesized examples for seen and unseen classes train a multi-class linear SVM, and the model outperforms previous approaches on unseen accuracy and harmonic mean.The harmonic mean aggregates performance across seen and unseen test classes, while the number of synthesized samples can be tuned for accuracy and evaluation time.
- Conventional zero-shot learning: In conventional ZSL, generated unseen-class samples train a linear SVM, with improvements reported across datasets varying in scale and image complexity.The reported settings include large-scale Imagenet, fine-grained CUB, and the many-class SUN dataset.
- Ablation: Removing the feedback mechanism in an ablation study demonstrates benefits from the feedback mechanism and the designed loss function.The comparison is conducted in the conventional ZSL setting.
- Synthesized-sample quality: Generated samples are effective for classification, and t-SNE embeddings show significant overlap between generated and real samples for two unseen classes.The overlap is presented as evidence that generated samples resemble samples from the true distribution.
7. Discussion and Conclusion
The paper presents a conditional VAE framework with discriminator-driven feedback that generates class-specific exemplars for generalized zero-shot learning. These exemplars support classification across seen and unseen classes, semi-supervised operation, and future extensions to adversarial, transductive, and online settings.
- The framework combines a conditional VAE with discriminator-driven feedback to generate high-quality, class-specific exemplars.The discriminator feedback encourages exemplars to be discriminative.
- Generated exemplars from unseen and optionally seen classes can train any classification model, reducing reliance on labeled seen-class data in GZSL.The classifier leverages synthesized examples from unseen classes alongside labeled examples from seen classes.
- The framework can incorporate unlabeled examples and therefore operate in semi-supervised settings.
- The authors identify adversarial training, transductive ZSL, and online few-shot learning as possible extensions.These directions are presented as future work or extensions rather than evaluated results.