A Generative Model For Zero Shot Learning Using Conditional Variational Autoencoders

Ashish Mishra; M Shiva Krishna Reddy; Anurag Mittal; Hema A Murthy

Source-linked AI summary

A Generative Model For Zero Shot Learning Using Conditional Variational Autoencoders

Ashish Mishra, M Shiva Krishna Reddy, Anurag Mittal, Hema A Murthy

arXiv:1709.00663v2Sep 3, 2017cs.CV

TL;DR

Zero-shot learning must classify classes absent from image training data using semantic descriptions or attributes, a challenge because collecting images for every possible class is difficult. The paper instead trains a conditional variational autoencoder to generate class-conditioned image features for classification, and reports state-of-the-art improvements across four benchmarks, particularly in generalized zero-shot learning. The authors conclude that the results provide evidence that conditional variational autoencoders can model image generation for unseen classes.

Problem
Zero-shot learning seeks to classify novel classes without training images when semantic class embeddings are available, because obtaining images for all possible classes is difficult.
Method
The paper treats unseen-class examples as missing data and uses a conditional variational autoencoder to generate image features from class embeddings for classifier training.
Results
The model compares favorably with the state of the art on four benchmark datasets and outperforms existing methods in the generalized zero-shot setting.
Takeaways & Limitations
The results provide evidence that conditional variational autoencoders capture the underlying image-generation process and can generate features for previously unseen classes.
Takeaways & Limitations
The generated feature visualization shows mode dropping for some classes, indicating that the model does not represent every mode of the data reliably.

Abstract

from arXiv · show

Zero shot learning in Image Classification refers to the setting where images from some novel classes are absent in the training data but other information such as natural language descriptions or attribute vectors of the classes are available. This setting is important in the real world since one may not be able to obtain images of all the possible classes at training. While previous approaches have tried to model the relationship between the class attribute space and the image space via some kind of a transfer function in order to model the image space correspondingly to an unseen class, we take a different approach and try to generate the samples from the given attributes, using a conditional variational autoencoder, and use the generated samples for classification of the unseen classes. By extensive testing on four benchmark datasets, we show that our model outperforms the state of the art, particularly in the more realistic generalized setting, where the training classes can also appear at the test time along with the novel classes.

1. Introduction

Zero-shot learning addresses classification when novel classes lack training images but have semantic information such as attributes or textual descriptions. This paper models unseen-class image features with a conditional variational autoencoder rather than a direct transfer function, and reports improvements across four benchmarks, especially in generalized ZSL.

Motivation: Zero-shot learning classifies novel-class images using semantic information even when those classes have no training images.Available information may include attributes or textual descriptions.
Limitations of prior work: Prior methods commonly learn mappings between image features and class embeddings, but domain shift can make mappings learned on seen classes unreliable for unseen classes.The image space may be more complex than the semantic space because of the image-generation process.
Proposed approach: The paper treats unseen-class examples as missing data and trains a conditional variational autoencoder to model image generation conditioned on class embeddings.Generated data fills the missing unseen-class observations.
Proposed approach: The model learns the conditional distribution of image features given class embeddings, aiming to reduce domain shift through a more sophisticated image-generation model.This approach models a probability distribution rather than only a simple mapping.
Results: Extensive testing on four benchmark datasets shows significant improvement over the state of the art, particularly in the harder generalized zero-shot setting.Generalized ZSL allows both training and novel classes to appear during testing.
Results: The authors report evidence that conditional variational autoencoders capture the underlying image-generation process and generalize to new tasks.This conclusion is tied to the model's ability to generate image features for previously unseen classes.

2. Related Work

Related work primarily learns visual-semantic mappings or nonlinear compatibility functions for zero-shot recognition. This paper builds on semantic class embeddings and prior conditional image-generation research while generating data in feature space.

Embedding and compatibility methods: Traditional zero-shot methods map visual features into semantic space and assign unseen examples to the nearest class embedding.The mapping is typically parameterized and may use attribute-based classification.
Embedding and compatibility methods: Nonlinear compatibility methods were proposed because image classification generally involves nonlinear decision boundaries that linear approaches may not model sufficiently.Examples include piecewise-linear multimodal learning and neural networks with hidden layers.
Embedding and compatibility methods: Other approaches align semantic and visual spaces using basis-vector combinations, graph manifolds, or sparse-coding frameworks.These methods continue to model relationships between class semantics and visual features rather than directly generating features.
Semantic embeddings: Semantic class embeddings can be human-labeled visual attributes or distributed representations such as word2vec when attribute vectors are unavailable.The paper uses attribute vectors where available and word2vec otherwise.
Generative approaches: Prior conditional image-generation work motivates this paper, but the proposed model operates in feature space instead of generating full images.Earlier work generated synthetic images from textual descriptions that appeared natural.

3. Method Description

The method treats unseen-class images as missing data and generates class-conditional features with a CVAE, then trains an SVM on generated pseudo-data. In generalized ZSL, it also generates pseudo-data for seen classes to reduce bias toward them.

Conditional Variational Autoencoder: The model conditions a variational autoencoder on each class’s semantic embedding to model the distribution of image features.The CVAE learns p(x|A_y), where A_y is the semantic embedding for class y.
Generating pseudo-data: For an unseen class, the decoder generates features by concatenating its embedding with latent vectors sampled from a standard normal distribution.These generated features form the pseudo train data for the class.
Training objective: Training combines reconstruction loss with a KL-divergence term while using neural-network encoder and decoder components.The encoder estimates a Gaussian latent distribution, samples a latent vector, and the decoder reconstructs the input feature.
Classification pipeline: The classifier is trained with generated unseen-class pseudo-data and uses an SVM for final classification.The pipeline trains the generative model, constructs pseudo-data, and fits an SVM classifier.
Generalized zero-shot learning: In generalized zero-shot learning, pseudo-data are generated for both seen and unseen classes because using original seen data with generated unseen data biases classification toward seen classes.The generalized procedure replaces the unseen-class set with the union of seen and unseen classes when generating pseudo-data.

4. Evaluation Protocol

The evaluation distinguishes standard zero-shot learning, where seen and test classes are disjoint, from generalized zero-shot learning, where both may appear at test time. The generalized setting is more realistic and more difficult.

Standard setting: Standard ZSL assumes that seen classes do not occur among the test classes.Formally, the class sets satisfy Y_tr ∩ Y_te = Φ.
Generalized setting: Generalized ZSL allows images from both seen and unseen classes during testing.This setting removes the disjoint-class assumption and is described as more difficult than standard ZSL.

5. Experiments

Experiments use four benchmark datasets spanning coarse- and fine-grained recognition, with additional generalized-ZSL results on three datasets. Deep CNN features and class embeddings provide the experimental inputs.

Datasets: The experiments cover AwA, CUB, SUN, and ImageNet, with generalized-ZSL evaluations reported for AwA, CUB, and SUN.AwA is coarse-grained, whereas CUB and SUN are fine-grained; ImageNet supplies the large-scale evaluation.
Implementation: The implementation uses Keras with a TensorFlow backend, and the authors state that code and data will be released for reproducibility.The passage describes the implementation framework and planned public release.
Visual features: The study uses CNN-extracted deep features, including VGG features for AwA and ResNet101 features for AwA-2, CUB, and SUN.The AwA images are unavailable publicly, so the authors use features supplied by the dataset authors.
Class embeddings: Class embeddings come from author-provided attribute annotations for AwA, SUN, and CUB, and from 1000-dimensional word2vec features for ImageNet.The ImageNet word2vec features were trained with a skip-gram model on Wikipedia.

5.2. Train-Test splits: An Important Note

The experiments use test-train splits designed to prevent pretrained feature extractors from having seen test classes, addressing an important source of inflated zero-shot results.

Avoiding pretraining overlap: Test classes overlapping ImageNet pretraining classes can inflate reported accuracy because the feature extractor has already encountered them.The authors therefore use splits ensuring that no test class occurs among ImageNet’s 1000 pretraining classes.
Evaluation consequences: On the proposed non-overlapping splits, performance decreases significantly across methods, indicating that older splits were not representative of true model performance.The paper reports 85.81% on the widely used standard AwA split but presents results on the newer splits for evaluation validity.

5.3. Parameters

The model uses fixed optimization and architecture settings, with key hyperparameters selected through cross-validation on training classes.

Training uses Adam with learning rate 10^-3, β1 = 0.9, and β2 = 0.999.
The neural network is initialized with Glorot initialization for both encoder and decoder.
The latent variable size is 100 and the SVM cost parameter is 100, selected by cross-validation on training classes.

5.4. Evaluation Metric

The evaluation uses per-class accuracy for AwA, CUB, and SUN, top-5 accuracy for Imagenet, and reports generalized results with harmonic mean accuracy.

AwA, CUB, and SUN are evaluated using average per-class accuracy, which is suited to imbalanced test sets.
Table 2 reports disjoint-setting results with per-class accuracy, while Table 3 reports generalized-setting results using harmonic mean over seen and unseen accuracy.
Imagenet performance is measured with top-5 accuracy, counting a prediction as correct when the true label appears among the five highest-ranked predictions.

5.5. The Generalized Zero Shot Setting

The generalized zero-shot evaluation includes both seen and unseen classes and uses held-out seen data together with generated samples to reduce seen-class bias.

The generalized protocol holds out 20% of seen-class data and trains on the remaining 80%.
The SVM is trained with generated data from both seen and unseen classes rather than actual seen-class data to reduce bias toward seen data.
Performance is evaluated separately on held-out seen data and unseen test data using their harmonic mean.

5.6. Results

The proposed model performs strongly across zero-shot benchmarks, with especially large gains in generalized zero-shot learning, while visualizations show accurate but unimodal generated features.

The proposed model performs significantly better than the state of the art on AwA-1 and AwA-2 in disjoint zero-shot learning.It improves AwA-1 performance by 5.8%; gains are also significant on SUN, while CUB performance is slightly lower than the state of the art.
7.9% improvement is achieved on Imagenet despite its greater difficulty and lack of explicit attribute vectors.
20% absolute improvement is achieved on coarse-grained datasets in generalized zero-shot learning, with state-of-the-art performance on fine-grained datasets.The authors attribute this to better distribution modeling with a variational autoencoder and reduced bias toward seen classes from generating seen-class features.
The performance gains are higher in generalized zero-shot learning because both seen and unseen classes occur at test time, making the setting harder.
Generated features are close to original unseen-class features, but the learned distribution is unimodal and omits several underlying modes.The visualization compares true and generated AwA-1 feature vectors using t-SNE; mode dropping suggests multimodality as a future improvement area.

6. Conclusion

The paper presents zero-shot learning as a missing-data problem and reports favorable results across four benchmarks, outperforming prior methods in generalized zero-shot learning.

The paper models zero-shot learning as a missing-data problem and compares favorably with the state of the art on four benchmark datasets.It reports outperforming competing methods on all four datasets in the harder generalized zero-shot setting.