Source-linked AI summary
Zero-Shot Learning via Class-Conditioned Deep Generative Models
Wenlin Wang, Yunchen Pu, Vinay Kumar Verma, Kai Fan, Yizhe Zhang, Changyou Chen, Piyush Rai, Lawrence Carin
TL;DR
The paper addresses zero-shot recognition when classes lack labeled training images, where existing point-based class representations inadequately capture variability. It proposes a class-conditioned latent-distribution prior for a supervised VAE, trained end-to-end on seen classes and extendable with unlabeled data. The framework outperforms state-of-the-art baselines in inductive and transductive ZSL, including a reported 37.4% margin on large-scale inductive datasets and about 8% higher average accuracy in the transductive setting than inductive learning.
Problem
Zero-shot learning must recognize unseen classes from side information despite having no corresponding labeled images, while common point-based representations inadequately account for intra-class variability.
Method
The model assigns each class an attribute-conditioned latent Gaussian prior for a supervised VAE, learns all parameters end-to-end, and can incorporate unlabeled data in transductive learning.
Results
The model outperforms state-of-the-art baselines in inductive and transductive ZSL, with a 37.4% margin over the best baseline on large-scale inductive datasets.
Takeaways & Limitations
Matching latent distributions rather than point embeddings provides a distribution-based prediction rule that is reported as more robust to intra-class variability.
Takeaways & Limitations
The standard training setting provides images only for seen classes, while unseen classes contribute attribute vectors without corresponding training images.
Abstract
from arXiv · showhide
We present a deep generative model for learning to predict classes not seen at training time. Unlike most existing methods for this problem, that represent each class as a point (via a semantic embedding), we represent each seen/unseen class using a class-specific latent-space distribution, conditioned on class attributes. We use these latent-space distributions as a prior for a supervised variational autoencoder (VAE), which also facilitates learning highly discriminative feature representations for the inputs. The entire framework is learned end-to-end using only the seen-class training data. The model infers corresponding attributes of a test image by maximizing the VAE lower bound; the inferred attributes may be linked to labels not seen when training. We further extend our model to a (1) semi-supervised/transductive setting by leveraging unlabeled unseen-class data via an unsupervised learning module, and (2) few-shot learning where we also have a small number of labeled inputs from the unseen classes. We compare our model with several state-of-the-art methods through a comprehensive set of experiments on a variety of benchmark data sets.
Introduction
Zero-shot learning seeks to recognize classes without labeled training examples, using class-level side information to transfer knowledge from seen to unseen classes. This paper addresses limitations of point-based representations with a class-conditioned generative model built around a supervised VAE.
- Introduction: Zero-shot learning uses class attributes or related side information to transfer knowledge from labeled seen classes to unseen classes.The motivation is learning new concepts when supervision is scarce or absent.
- Introduction: Existing methods often represent each class as a fixed semantic point, which does not adequately model intra-class variability.They also commonly lack a generative model for learning expressive representations and incorporating unlabeled data.
- Introduction: The proposed model learns attribute-specific latent Gaussian distributions and uses them as priors in a supervised VAE.The attribute vector is given for training images and inferred for test images, linking predictions to unseen labels.
- Introduction: Figure 1 depicts only the training stage, with the CNN serving as a replaceable initial feature extractor rather than part of the proposed framework.The class attribute vector is supplied for training data and inferred for test data.
- Introduction: At test time, inferred attributes can identify classes whose images were absent from training.The framework also supports semi-supervised or transductive learning by incorporating unlabeled data through an unsupervised module.
Variational Autoencoder
A VAE learns a latent-variable density model by approximating the posterior over latent representations. This paper uses a supervised VAE to incorporate class attributes and support prediction of unseen classes.
- Variational Autoencoder: A VAE learns complex data density models through latent variables and approximates their posterior with a recognition model.Its objective combines expected reconstruction likelihood with a KL divergence regularizer.
- Variational Autoencoder: The recognition model qφ(z|x) maps an input to a probabilistic latent encoding whose parameters are produced by a neural network.After training, latent samples can be generated efficiently from qφ(z|x).
- Variational Autoencoder: The supervised VAE incorporates class attribute vectors to learn a generative model that can predict labels for unseen classes.Unseen-label prediction links inferred attributes to their associated class labels.
Deep Generative Model for ZSL
The paper develops inductive and transductive zero-shot learning models that represent classes with attribute-conditioned latent distributions and use them within a VAE framework. The transductive extension exploits unlabeled unseen-class inputs through probability sharpening and KL-based regularization.
- Inductive ZSL: The model combines a VAE decoder and encoder with class-specific priors, learning parameters end-to-end from labeled seen-class data.The objective includes a variational lower bound and a margin regularizer that separates the true class distribution from competing class distributions.
- Inductive ZSL: Attribute-conditioned Gaussian priors replace the standard shared VAE prior, allowing each seen or unseen class to have its own latent-space distribution.The prior parameters are generated from class attributes, with unseen-class distributions obtained from their associated attribute vectors.
- Inductive ZSL: At test time, the model assigns an unseen image to the class whose latent distribution best matches its VAE posterior, rather than comparing point prototypes with Euclidean distance.This distributional comparison can account for multimodal class structure.
- Transductive ZSL: The transductive setting incorporates unlabeled unseen-class test inputs through an additional unsupervised regularizer added to the inductive objective.The method uses predicted class distributions because the unseen inputs lack labels.
- Transductive ZSL: A naïve reconstruction-only use of unlabeled inputs produces only marginal improvements because it ignores their label information.This motivates the label-aware probability regularizer used by the proposed transductive model.
- Transductive ZSL: The transductive regularizer sharpens predicted unseen-class probabilities and uses KL-based matching so confident assignments can guide ambiguous inputs.Class-marginal normalization prevents large classes from distorting the latent space.
Related Work
Prior zero-shot methods commonly map inputs and classes into semantic spaces or represent classes through combinations of seen or basis classes. The paper positions its model as a nonlinear generative alternative that addresses fixed-point representations and can use unlabeled unseen-domain data.
- Semantic Embedding Methods: Many zero-shot methods represent classes as points in semantic space and classify with nearest-neighbor-style comparisons.These methods learn projections from inputs to semantic representations using regression or ranking models.
- Class-Combination Models: Other approaches model unseen classes as linear or convex combinations of seen or shared basis classes.The proposed framework is described as a flexible generalization because its latent-distribution parameters are shared across seen and unseen classes.
- Transductive ZSL: Domain shift occurs when seen and unseen classes come from different domains, and transductive methods can use unlabeled unseen-domain data to address it.The cited prior transductive approach adapts classifiers using unlabeled test inputs from unseen classes.
- Generative ZSL Models: Compared with semantic autoencoders and related encoder-decoder methods, the proposed approach provides a generative model, represents classes by distributions, and extends to the transductive setting.The paper also contrasts its nonlinear generative model with prior linear generative approaches.
- Related Deep Generative Models: The paper distinguishes its recognition goal from deep generative methods focused on image generation or learning from very small numbers of labeled examples.Those methods are presented as related but aimed at different tasks.
Experiments
The experiments evaluate VZSL across inductive and transductive zero-shot, few-shot, and transductive few-shot settings on benchmark datasets. VZSL outperforms state-of-the-art baselines, while visualizations show separated latent representations.
- Experiments: Experiments use AwA, CUB-200, SUN, and ImageNet with VGG-19 fc7 inputs and dataset-provided semantic attributes, except word2vec features for ImageNet.The model uses MLP encoder and decoder networks and repeats reported experiments over 10 runs.
- Inductive ZSL: VZSL outperforms state-of-the-art baselines in inductive ZSL, with a 37.4% margin over the best baseline on large-scale datasets.The model also performs better on small-scale datasets and remains comparable to most baselines without the reconstruction term.
- Transductive ZSL: VZSL again outperforms state-of-the-art methods in transductive ZSL, improving accuracy by about 8% on average over inductive ZSL.Removing the reconstruction term can substantially reduce performance, particularly on CUB-200.
- Few-Shot Learning (FSL): With only 2 or 5 additional labeled examples per unseen class, few-shot learning significantly improves over ZSL and outperforms a multiclass SVM.The comparison varies the number of labeled examples per unseen class, including 2, 5, 10, 15, and 20.
- Few-Shot Learning (FSL): VZSL outperforms baselines in transductive few-shot recognition, using labeled and unlabeled examples from each unseen class.Table 4 evaluates the setting with three randomly labeled images per test class and the remaining images unlabeled.
- t-SNE Visualization: t-SNE visualizations show reasonably separated classes in VZSL’s latent embeddings and reconstructions under both inductive and transductive settings.The experiments use AwA data and compare raw CNN features, latent codes, and reconstructed features.
Conclusion
The framework models unseen classes and inputs with latent-space distributions and a VAE, achieving superior classification accuracy in both inductive and transductive ZSL.
- The generative framework yields superior classification accuracies compared with existing ZSL methods in both inductive and transductive settings.It matches each test input's VAE posterior with unseen-class latent-space distributions and adds a regularizer for unlabeled unseen-class inputs.
- Unlike point-based ZSL methods, the framework represents each class with a class-specific latent-space distribution and each input with a VAE decoder model.This distribution matching provides more robustness than point-based Euclidean distance, while the generative model leverages intrinsic input-space structure.