Source-linked AI summary
Zero-Shot Learning by Convex Combination of Semantic Embeddings
Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg S. Corrado, Jeffrey Dean
TL;DR
Zero-shot recognition must generalize from labeled training categories to unseen categories despite the cost and maintenance burden of expanding annotations. ConSE converts a pretrained classifier’s probabilistic outputs into a convex combination of semantic label embeddings without additional training, and outperforms more elaborate approaches on ImageNet zero-shot learning.
Problem
Expanding labeled image categories is costly and zero-shot learning must recognize categories with no training images, requiring semantic information to relate seen and unseen labels.
Method
ConSE uses a pretrained classifier’s class probabilities to form a convex combination of semantic embeddings, then ranks unseen labels by cosine similarity.
Results
ConSE outperforms more elaborate joint-training approaches on ImageNet zero-shot learning and achieves 9.4% hit@1 and 24.7% hit@5 on 1600 unseen categories.
Takeaways & Limitations
The framework can leverage existing visual classifiers and text embeddings to extend predictions beyond the training labels.
Takeaways & Limitations
Results depend on the choice of training corpus used for the word embeddings.
Abstract
from arXiv · showhide
Several recent publications have proposed methods for mapping images into continuous semantic embedding spaces. In some cases the embedding space is trained jointly with the image transformation. In other cases the semantic embedding space is established by an independent natural language processing task, and then the image transformation into that space is learned in a second stage. Proponents of these image embedding systems have stressed their advantages over the traditional \nway{} classification framing of image understanding, particularly in terms of the promise for zero-shot learning -- the ability to correctly annotate images of previously unseen object categories. In this paper, we propose a simple method for constructing an image embedding system from any existing \nway{} image classifier and a semantic word embedding model, which contains the $\n$ class labels in its vocabulary. Our method maps images into the semantic embedding space via convex combination of the class label embedding vectors, and requires no additional training. We show that this simple and direct method confers many of the advantages associated with more complex image embedding schemes, and indeed outperforms state of the art methods on the ImageNet zero-shot learning task.
1 Introduction
Standard image recognition requires costly, revisable annotations for expanding label sets, motivating zero-shot methods that map images into semantic spaces. ConSE instead combines an existing probabilistic classifier with word embeddings and achieves strong ImageNet zero-shot results without additional training.
- Fine-grained image annotation is costly, time-consuming, difficult to maintain, and requires classifier retraining as new visual entities emerge.
- Zero-shot learning seeks to annotate images with labels for previously unseen object categories using semantic label embeddings.
- ConSE combines any probabilistic n-way image classifier with a word embedding model containing the classifier’s labels, avoiding additional image-embedding training.
- ConSE maps an image to a continuous semantic vector by weighting class-label embeddings with the classifier’s predictive probabilities, then extrapolates predictions to test labels.
- 9.4% hit@1 and 24.7% hit@5 were achieved on 1600 unseen ImageNet categories using a CNN trained on 1000 categories.
2 Previous work
Prior zero-shot approaches define semantic label spaces using attributes or unsupervised language embeddings, then learn mappings from images into those spaces. Attribute-based representations face scalability challenges, while language embeddings offer a more scalable alternative.
- Zero-shot learning differs from one-shot learning because held-out test categories have no training images, making label side information more essential.
- Attribute-based methods represent labels with visual characteristics such as materials, colors, textures, and object parts, predicting attributes with multiple classifiers.
- Annotating thousands of attributes for thousands of classes is ambiguous and challenging, limiting supervised attributes’ scalability to large-scale zero-shot learning.
- Unsupervised neural language modeling learns multi-dimensional word vectors in which words sharing contexts receive similar embeddings.
3 Problem Statement
Zero-shot learning trains on labeled classes and evaluates on disjoint unseen classes, so semantic embeddings provide the side information needed to relate them. Earlier methods learn an image-to-semantic-space regression function before ranking unseen labels.
- The training and test label sets are disjoint: test labels Y1 are omitted from the training set Y0.
- Zero-shot learning aims to train on D0 and perform reasonably on the unseen test set D1.
- Without side information relating Y0 and Y1, zero-shot learning is infeasible because the two label sets do not overlap.
- Semantic embeddings s(y) place training and test labels in a joint space where similar labels have nearby vectors.
- Previous methods learn a regression function f: X → S from image features to semantic embeddings, then use nearest-neighbor search to rank unseen labels.
4 ConSE: Convex combination of semantic embeddings
ConSE retains a standard classifier and deterministically converts its top predictions into a semantic embedding by probability-weighted convex combination. Nearest-neighbor comparison in semantic space then transfers predictions from training labels to unseen labels.
- Model Description: ConSE trains a classifier p0 on D0 to estimate p0(y | x) for training labels instead of explicitly learning an image-to-semantic-space regression.
- Model Description: The model selects the top T classifier predictions and combines their corresponding semantic label vectors using their probabilities as weights.
- Model Description: T controls the maximum number of semantic embedding vectors included in the convex combination.
- Model Description: A confident classifier prediction makes f(x) close to one label vector, while uncertainty between lion and tiger places f(x) between their embeddings.
- Model Description: ConSE ranks unseen labels by cosine similarity to f(x), using a standard classifier rather than a learned regression model to define the embedding.
- Difference with DeViSE: Unlike DeViSE, ConSE keeps the CNN Softmax layer intact and performs no further neural-network training after using its top T predictions.
5 Experiments
On ImageNet, ConSE combines a trained 1000-class CNN with semantic label embeddings and generalizes to unseen categories, outperforming DeViSE across zero-shot settings while retaining semantically coherent predictions.
- Experimental setup: ConSE combines the same skipgram semantic space and ImageNet CNN used by DeViSE, evaluating zero-shot transfer on increasingly difficult 2-hops and 3-hops datasets.The skipgram model uses 5.4 billion Wikipedia words to construct normalized 500-dimensional embeddings.
- Qualitative results: Qualitative examples show generally coherent ConSE(10) predictions with few outliers, whereas DeViSE produces more semantically distant labels.The figure examples are hand-picked to include both successful ConSE cases and failures.
- Zero-shot results: ConSE consistently outperforms DeViSE across all tested datasets and all values of T, with ConSE(10) performing best among the variants.The comparison uses flat hit@k results, while hierarchical precision results also confirm ConSE(10)'s zero-shot advantage.
- Evaluation caveats: Including training labels reduces ConSE(10)'s 2-hops hit@1 from 9.4% to 0.3%, revealing strong bias toward training labels in candidate selection.The authors report that excluding training labels gives the best performance and that vanilla nearest-neighbor search may need improvement when training labels are candidates.
- Generalization: On the original 1,000-class task, DeViSE performs better than ConSE, but its stronger training-label performance does not transfer as well to unseen categories.The authors attribute this pattern to DeViSE learning a more complex embedding function that fits training labels but generalizes less reliably to novel label embeddings.
6 Conclusion
ConSE recasts classifier outputs as a convex combination of semantic word vectors, while remaining compatible with broader visual classification systems. Its vector norm also provides an implicit confidence signal that the experiments did not exploit.
- 6 Conclusion: ConSE treats classifier scores as weights in a convex combination of word vectors to map images into semantic embedding space.The method is described as a direct recasting of an n-way image classifier as an image embedding system.
- 6 Conclusion: Any visual classifier producing relative scores over classes can be used within the ConSE framework.The authors used a deep convolutional network with Softmax scores, but do not restrict the framework to those model details.
- 6 Conclusion: The assigned embedding norm implicitly represents confidence, becoming lower for uncertain predictions or combinations of semantically disparate labels.Norms near 1.0 occur for nearly unambiguous images or closely related labels, whereas uncertainty and semantic disparity reduce magnitude.