Source-linked AI summary
Zero-Shot Learning Through Cross-Modal Transfer
Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D. Manning, Andrew Y. Ng
TL;DR
The paper addresses visual classification when unseen categories have no labeled images. It maps images into an unsupervised semantic word space and uses outlier detection to route images between seen- and unseen-class recognition. The joint model reports strong performance on known classes and reasonable performance on unseen classes, without manually defined semantic or visual attributes.
Problem
Zero-shot learning must classify visual instances from categories with no training images, using only indirect knowledge about those categories.
Method
The model maps images into unsupervised semantic word vectors and combines outlier detection with separate recognition of seen and unseen classes.
Results
The joint model achieves state-of-the-art accuracy on known classes and reasonable performance on unseen classes; zero-shot-only classification reaches above 80% accuracy and up to 90% in the conclusion.
Takeaways & Limitations
Unsupervised language representations can transfer knowledge across visual categories while supporting classification of both categories with many examples and categories with none.
Abstract
from arXiv · showhide
This work introduces a model that can recognize objects in images even if no training data is available for the objects. The only necessary knowledge about the unseen categories comes from unsupervised large text corpora. In our zero-shot framework distributional information in language can be seen as spanning a semantic basis for understanding what objects look like. Most previous zero-shot learning models can only differentiate between unseen classes. In contrast, our model can both obtain state of the art performance on classes that have thousands of training images and obtain reasonable performance on unseen classes. This is achieved by first using outlier detection in the semantic space and then two separate recognition models. Furthermore, our model does not require any manually defined semantic features for either words or images.
1 Introduction
The paper introduces zero-shot visual classification using knowledge from unsupervised natural-language representations, including a model that predicts both seen and unseen classes. It combines semantic image mapping with outlier detection and separate recognition behavior, while avoiding manually defined attributes.
- Motivation: Zero-shot learning classifies instances from visual classes without labeled examples, addressing categories such as new species, products, activities, gadgets, and car models.The motivation is to use knowledge available in natural language to identify unseen objects.
- Contribution: The model can determine whether an image belongs to an unseen class such as cat or to a known class such as dog or horse without seeing cat images.
- Method: Images are mapped into a semantic word space learned from a large unsupervised text corpus, grounding distributional word vectors in visual information.
- Method: Outlier detection estimates whether an image lies on the manifold of known categories, after which a standard classifier handles known images and semantic likelihoods handle unseen images.
- Contribution: The joint model reports state-of-the-art accuracy on known classes and reasonable performance on unseen classes, without manually defined semantic or visual attributes.
2 Related Work
Related work spans zero-shot, one-shot, attribute-transfer, domain-adaptation, and multimodal-embedding approaches. The paper distinguishes its setting by using unsupervised language representations to classify categories with either zero or many training images.
- Zero-Shot Learning: Prior zero-shot work mapped fMRI scans into manually designed feature spaces and differentiated zero-shot classes, but did not classify new instances across both seen and unseen classes.
- Zero-Shot Learning: Some zero-shot methods describe unseen classes with canonical examples or ground-truth human attribute labels.
- One-Shot Learning: Unlike one-shot learning, this model classifies object categories without any training data through cross-modal transfer from natural language while retaining high performance on classes with many examples.
- Knowledge and Visual Attribute Transfer: Attribute-transfer methods use manually designed visual attributes, whereas this work uses distributional word features from unsupervised, nonparallel corpora for categories with thousands or zero training images.
- Domain Adaptation: Domain adaptation assumes data for each class but addresses feature differences between domains, unlike this paper’s zero-training-data setting for unseen categories.
- Multimodal Embeddings: Related multimodal embedding work also uses large unsupervised text corpora, but requires a small amount of training data for each category.
3 Word and Image Representations
The framework represents words with distributional vectors learned from context and images with unsupervised features extracted from raw pixels. These representations provide the inputs for cross-modal semantic transfer.
- Word Representations: Distributional word representations encode semantic similarity through vectors of contextual co-occurrence characteristics.
- Word Representations: The model initializes word vectors with pretrained 50-dimensional representations learned from free Wikipedia text using local and global document context.
- Image Representations: Image features are extracted from raw pixels with an unsupervised method, representing each image as a vector x ∈ R^F.
4 Projecting Images into Semantic Word Spaces
The model projects image features into a 50-dimensional semantic word space shared by seen and unseen class vectors. This projection visually clusters seen images near their class words and places unseen images near semantically related seen classes.
- Class and Word Spaces: Seen classes have training data, unseen classes have none, and their word vectors together form W = W_s ∪ W_u for semantic transfer.
- Image Projection: Training images from each seen class are mapped to the word vector corresponding to that class name by minimizing a distance-based objective.
- Visualization: The t-SNE visualization displays word vectors and mapped images for trained and unseen classes, including cat and truck.
- Semantic Grounding: Projecting images into word space implicitly gives word semantics a visual grounding that supports queries such as prototypical visual instances or average color.
- Visualization: Seen images cluster near their corresponding word vectors, while unseen cat images map near dog and horse and unseen truck images map near semantically related classes.
- Model Design: The observed clustering motivates first detecting outliers and then classifying them using unseen word vectors.
5 Zero-Shot Learning Model
The model predicts both seen and unseen classes by combining an outlier-based visibility decision with separate seen-class and zero-shot recognition models. It maps images into a semantic word space and uses Gaussian likelihoods around semantic vectors for unseen-class recognition.
- The model predicts p(y|x) for both seen and unseen classes by introducing a binary visibility variable V.V indicates whether an image belongs to a seen or unseen class.
- An outlier probability identifies whether a test image lies on the manifold of known categories by thresholding its marginal likelihood.The marginal is estimated with a mixture of Gaussians fitted to mapped seen-class training points.
- The Gaussian model uses each class's semantic word vector as its mean, estimates covariance from mapped training points, and restricts covariances to be isometric to prevent overfitting.The class likelihood is modeled as a weighted sum over seen classes.
- For seen images, the model can use a standard classifier, while unseen images are classified using an isometric Gaussian distribution around each zero-shot semantic word vector.The implementation uses a softmax classifier on original image features for seen classes.
6 Experiments
Experiments on CIFAR10 evaluate zero-shot transfer between held-out classes and joint recognition of seen and unseen classes. Performance depends on semantic similarity to seen categories and on the outlier threshold used at test time.
- Experimental setup: CIFAR10 contains 10 classes with 5000 32 × 32 × 3 RGB images per class, and two classes are omitted from training for zero-shot analysis.Images are represented using 12,800-dimensional unsupervised features.
- Zero-Shot Classes Only: Zero-shot accuracy is close to random when no seen class is sufficiently similar to the held-out classes.Holding out cat and dog performs poorly because the remaining categories do not provide a suitable mapping basis.
- Zero-Shot Classes Only: When cat and truck are held out, transfer from dogs and cars supports very high zero-shot classification performance.The example links cat to dogs and truck to cars in the semantic transfer process.
- Zero-Shot Classes Only: Above 80% accuracy is obtained when distinguishing images from only two zero-shot classes across outlier-detection cutoffs.The cutoff is defined on the negative log-likelihood of each mapped image under the marginal model.
- Zero-Shot and Seen Classes: At approximately 80% seen-class accuracy, unseen-class accuracy ranges from 30% to 15%, compared with 10% random chance.The seen/unseen balance changes with the threshold that separates the two groups at test time.
7 Conclusion
The paper concludes that learned semantic word vectors can transfer knowledge between categories, while Bayesian outlier detection combines zero-shot and standard classification. A fully unsupervised model reaches up to 90% when distinguishing among zero-shot classes only.
- The proposed framework jointly performs standard and zero-shot classification using deep learned word and image representations.
- Unsupervised semantic word vectors support knowledge transfer between categories, and Bayesian outlier detection combines seen- and unseen-class recognition.
- Up to 90% accuracy is obtained for differentiating among zero-shot classes with a fully unsupervised model.