Source-linked AI summary
Learning a Deep Embedding Model for Zero-Shot Learning
Li Zhang, Tao Xiang, Shaogang Gong
TL;DR
ZSL must recognize unseen classes from seen-class training data and semantic relationships, yet existing deep end-to-end models have shown little advantage over non-end-to-end alternatives. The paper proposes DEM, which embeds semantic representations into CNN visual feature space and fuses modalities end to end. Across four benchmarks, DEM achieves state-of-the-art performance, while the model assumes disjoint seen and unseen classes.
Problem
Existing deep end-to-end ZSL models have shown little advantage over alternatives, motivating a better choice of embedding space for nearest-neighbour recognition.
Method
DEM uses a CNN output visual feature space as the embedding space, projects semantic representations into it, and supports multimodal end-to-end learning.
Results
DEM achieves state-of-the-art performance across four benchmark datasets, and experiments validate the importance of selecting the correct embedding space.
Takeaways & Limitations
Using visual feature space can reduce hubness in nearest-neighbour search while allowing multiple semantic spaces to be fused and jointly optimised.
Takeaways & Limitations
The ZSL setup requires training and test classes to be disjoint.
Abstract
from arXiv · showhide
Zero-shot learning (ZSL) models rely on learning a joint embedding space where both textual/semantic description of object classes and visual representation of object images can be projected to for nearest neighbour search. Despite the success of deep neural networks that learn an end-to-end model between text and images in other vision problems such as image captioning, very few deep ZSL model exists and they show little advantage over ZSL models that utilise deep feature representations but do not learn an end-to-end embedding. In this paper we argue that the key to make deep ZSL models succeed is to choose the right embedding space. Instead of embedding into a semantic space or an intermediate space, we propose to use the visual space as the embedding space. This is because that in this space, the subsequent nearest neighbour search would suffer much less from the hubness problem and thus become more effective. This model design also provides a natural mechanism for multiple semantic modalities (e.g., attributes and sentence descriptions) to be fused and optimised jointly in an end-to-end manner. Extensive experiments on four benchmarks show that our model significantly outperforms the existing models. Code is available at https://github.com/lzrobots/DeepEmbeddingModel_ZSL
1. Introduction
The paper frames ZSL as a way to recognize unseen classes from seen-class supervision and semantic relationships, then argues that deep ZSL success depends on choosing the embedding space. DEM embeds semantic representations into visual feature space, supports multimodal end-to-end learning, and outperforms prior models across four benchmarks.
- Motivation: Supervised recognition models typically require at least 100s of annotated samples per class, limiting recognition of rare categories.ZSL instead transfers knowledge from seen classes to unseen classes without requiring visual samples for each target class.
- ZSL framework: ZSL learns a joint space for semantic prototypes and visual features, then matches unseen class prototypes to projected image features by nearest-neighbour search.Semantic representations may use attributes, word vectors, or sentence descriptions.
- Motivation: Existing end-to-end deep ZSL models offer joint optimisation, transfer-learning flexibility, and multimodal fusion, but often perform only comparably to or worse than non-deep alternatives.The paper identifies this gap despite the intrinsic advantages of end-to-end neural models.
- Design rationale: High-dimensional semantic or intermediate embedding spaces make nearest-neighbour search vulnerable to hubness, while projecting visual features into semantic space can shrink variance and aggravate it.Hubness occurs when a few unseen prototypes become nearest neighbours for many data points.
- Proposed model: DEM uses a CNN’s output visual feature space as the embedding space and projects semantic vectors into it rather than projecting visual features into semantic space.The model also fuses multiple semantic modalities and enables end-to-end learning of semantic representations.
- Evaluation: Experiments on four benchmarks, including AwA, CUB, and ILSVRC 2010 and 2012, show that DEM often beats state-of-the-art models by a large margin.The contribution claim reports broad benchmark superiority without restricting it to a single dataset or metric.
2. Related Work
Prior ZSL work uses attribute, word-vector, or text-description semantic spaces, which differ in annotation cost and suitability for scale. Existing methods also vary in embedding-space choice and loss, while this paper combines semantic modalities and uses least-square loss to address hubness.
- Semantic space: ZSL commonly uses attribute, word-vector, or text-description semantic spaces, each trading annotation requirements against scalability.Attributes and descriptions can perform well but require manual annotation, whereas word vectors are attractive for large-scale recognition because they are free.
- Multimodal fusion: Multiple semantic spaces can be complementary, motivating fusion strategies that combine their information for recognition.The related work contrasts simple score-level fusion with more sophisticated approaches.
- Embedding and losses: Existing ZSL models differ in whether they embed into semantic or intermediate spaces and in their use of margin, Euclidean, dot-product, or other losses.The paper positions its least-square loss as both effective and theoretically motivated for hubness.
- End-to-end learning: The proposed model jointly learns a neural language encoder with the embedding model, fuses text descriptions with other semantic spaces, and reports better performance than a prior end-to-end model.This comparison specifically concerns the model described in [37].
3. Methodology
The model embeds both visual inputs and semantic class representations into the visual feature space, where classification uses nearest-prototype search. It supports single or multiple semantic modalities, including text encoded by a bidirectional LSTM, and motivates this direction through hubness analysis.
- Model architecture: The CNN visual branch produces D-dimensional features, which serve as the embedding space for both images and semantic class representations.A semantic encoding subnet maps L-dimensional class representations into the same D-dimensional visual space.
- Model architecture: Training minimizes a least-squares discrepancy between visual features and embedded class representations, together with parameter regularization.The two fully connected layers use ReLU nonlinearities and l2 regularization, while λ weights the regularization terms.
- Classification: At test time, the model predicts the unseen class whose embedded prototype has the smallest distance to the test image feature.The prototype is obtained by applying the semantic embedding network to the unseen class’s semantic vector.
- Multiple semantic space fusion: Multiple semantic representations are mapped into a shared fusion space and added element-wise before producing the semantic embedding.This design supports combinations such as attribute vectors and word vectors.
- Bidirectional LSTM encoder for description: Text descriptions are encoded into fixed-length semantic vectors with a bidirectional LSTM, whose encoding subnet can be jointly learned with the rest of the ZSL model.At test time, text encodings are averaged per class to form class prototypes.
- The hubness problem: The ridge-regression interpretation suggests that mapping semantic representations into visual space shrinks their variance, whereas reversing the direction can make near-origin semantic vectors hubs.Experiments reportedly confirm that embedding direction substantially affects hubness severity and ZSL performance; hubness is measured using Nk-distribution skewness.
4. Experiments
Experiments across small-scale and ImageNet benchmarks evaluate DEM under conventional, generalized, and revised zero-shot settings. Results support visual-space embedding as a central design choice, with reduced hubness and stronger classification performance than semantic-space alternatives.
- Datasets and settings: The evaluation covers AwA, CUB, and ImageNet benchmarks under conventional, generalized, old, and GBU zero-shot settings.The GBU setting prevents test classes from overlapping with ImageNet 1K pretraining classes, while generalized zero-shot learning mixes seen and unseen samples at test time.
- Small-scale datasets: 67.1% and 61.9%, DEM outperforms alternatives by large margins on the reported AwA comparison.
- ImageNet: 60.7% hit@5 accuracy on ILSVRC 2010 beats the nearest rival PDDM by over 12 percentage points under the shared pretrained-CNN setting.For other methods, DEM uses an AlexNet subnet pretrained from scratch on the 800 training classes for a fair comparison.
- Embedding-space analysis: Mapping visual features into the semantic embedding space reduces AwA performance by 26.1% compared with using the CNN output visual feature space.This ablation directly tests the paper’s embedding-space selection hypothesis.
- Embedding-space analysis: Visual-space embeddings form compact, prototype-aligned class clusters, whereas semantic-space embeddings show weaker separation and hub prototypes associated with misclassification.The corresponding hubness analysis reports more severe hubness when the semantic space is selected.
- Training analysis: Least-square loss is used in the proposed model, and it performs better than hinge ranking loss when the visual feature space is the embedding space.With semantic-space embedding, hinge ranking loss is slightly inferior; the stronger loss advantage appears for visual-space embedding.
5. Conclusion
The paper proposes a deep embedding model for zero-shot learning that uses the CNN output feature space as its embedding space. It supports multiple semantic spaces, end-to-end learning, and achieves state-of-the-art performance on benchmark datasets.
- The model uses the CNN output feature space as the embedding space, unlike existing models using semantic or intermediate spaces.The authors hypothesize that this choice reduces hubness in nearest-neighbour search.
- The model flexibly utilises multiple semantic spaces and supports end-to-end learning when semantic spaces are computed by neural networks.
- The model achieves state-of-the-art performance on a number of benchmark datasets.The experiments also validate the hypothesis that selecting the embedding space is key to performance.