PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-Shot Recognition using Dual Visual-Semantic Mapping Paths

Yanan Li, Donghui Wang, Huanhang Hu, Yuetan Lin, Yueting Zhuang

arXiv:1703.05002v2cs.CV

TL;DR

Zero-shot recognition must transfer a visual-semantic mapping learned from seen classes to unseen classes despite differences between image and semantic manifolds. The paper formalizes their relationship and proposes DMaP, which jointly optimizes the mapping and semantic space through dual paths; it reports state-of-the-art results on four datasets and both conventional and generalized settings.

  • Problem

    Zero-shot recognition seeks to recognize unseen classes, but recognition performance varies with the semantic embedding space and its relationship to image features.

  • Method

    DMaP jointly optimizes the visual-semantic mapping fs and semantic space K using dual mapping paths and iterative manifold alignment.

  • Results

    The method outperforms state-of-the-art approaches on four challenging datasets and achieves state-of-the-art performance in conventional and generalized ZSR settings.

  • Takeaways & Limitations

    More consistent semantic and image-feature manifolds can improve the transfer ability of visual-semantic mapping to unseen classes.

  • Takeaways & Limitations

    The method can fail on visually close classes, such as chimpanzees classified as gorillas because the classes are close on the image-space manifold.

Abstract

from arXiv · show

Zero-shot recognition aims to accurately recognize objects of unseen classes by using a shared visual-semantic mapping between the image feature space and the semantic embedding space. This mapping is learned on training data of seen classes and is expected to have transfer ability to unseen classes. In this paper, we tackle this problem by exploiting the intrinsic relationship between the semantic space manifold and the transfer ability of visual-semantic mapping. We formalize their connection and cast zero-shot recognition as a joint optimization problem. Motivated by this, we propose a novel framework for zero-shot recognition, which contains dual visual-semantic mapping paths. Our analysis shows this framework can not only apply prior semantic knowledge to infer underlying semantic manifold in the image feature space, but also generate optimized semantic embedding space, which can enhance the transfer ability of the visual-semantic mapping to unseen classes. The proposed method is evaluated for zero-shot recognition on four benchmark datasets, achieving outstanding results.

1. Introduction

The paper reframes zero-shot recognition as joint optimization of the visual-semantic mapping and semantic manifold, then introduces DMaP with dual mapping paths to improve transfer to unseen classes. Evaluations on four datasets and two ZSR settings report state-of-the-art performance.

  • Motivation: Zero-shot recognition addresses recognition of classes without labeled images, motivated by long-tailed categories and the cost of fine-grained annotation.The paper highlights endangered birds, rare plants, and specialized annotation as examples of these challenges.
  • Motivation: Semantic embeddings encode relationships between seen and unseen classes through a geometric semantic manifold that differs across embedding types.The paper discusses attribute vectors and word vectors as alternative semantic embeddings whose manifold structures can affect recognition performance.
  • Core idea: The paper formalizes a connection between semantic-manifold consistency and the transfer ability of the visual-semantic mapping to unseen classes.It proposes inter-class semantic consistency as a measure of how well two semantic manifolds match.
  • Core idea: ZSR is cast as joint optimization of the semantic manifold in K and the visual-semantic mapping fs, especially when image and semantic manifolds are inconsistent.This formulation seeks to refine K so it becomes more consistent with the image-feature manifold.
  • Proposed method: DMaP uses dual visual-semantic mapping paths to learn an optimized mapping and a semantic space correlated with the image-feature space.The framework infers an underlying class-level manifold in Xs, generates a homologous semantic space, and iteratively aligns the two manifolds.
  • Evaluation: The method achieves state-of-the-art performance on four datasets under both conventional and generalized ZSR settings.The evaluated datasets are Animals with Attributes, Caltech-UCSD Birds, Stanford Dogs, and ImageNet.

2. Related work

Related work is organized around visual-semantic mapping paths, ZSR evaluation settings, and the distinction between inductive and transductive recognition. Existing methods commonly align heterogeneous image and semantic spaces, while generalized ZSR evaluates seen and unseen classes together.

  • Visual-semantic mapping path: ZSR methods construct a path from image feature space X to semantic space K, either directly or through intermediary spaces.Examples include direct projection, shared embedding spaces, and dictionary-learning coefficient spaces.
  • Visual-semantic mapping path: Because image and textual semantic spaces are heterogeneous, learning a projection between them can increase complexity and risk over-fitting on seen classes.The paper positions its approach as using a different strategy from mandatory direct alignment.
  • ZSR taxonomy: Inductive ZSR receives unseen samples serially during testing, whereas transductive ZSR uses manifold information from unseen samples.The related-work taxonomy separates methods according to whether unseen-class image data are used during testing.
  • Generalized ZSR: Generalized ZSR tests data from both seen and unseen classes and requires classification across the whole label space.This differs from the default setting, which assumes seen classes are absent during testing and discriminates only among unseen categories.

3. Methodology

The method treats zero-shot recognition as aligning class-level manifolds in image-feature space X and semantic space K, while jointly optimizing the visual-semantic mapping and a refined semantic space. Its dual-path training detects projection defects, measures inter-class consistency, constructs class prototypes from mapped instances, and iteratively aligns the manifolds.

  • Problem setting: Zero-shot recognition predicts a test image’s semantic embedding with a mapping from X_s to K, then assigns the nearest seen or unseen class embedding.The standard pipeline first estimates the semantic embedding and then infers the label by comparison with class embeddings.
  • Pre-inspection of semantic space K: If two unseen embeddings have identical orthogonal projections onto span(K_s), the semantic space cannot transfer their classes distinctly.This defect is especially relevant when seen classes are far fewer than unseen classes and motivates pre-inspection.
  • Inter-class Relationship Consistency: Inter-class relationship consistency holds when the image-space and semantic-space relationship matrices satisfy X̃_sR_x = X̃_sR_k.The method represents each unseen prototype through associations with seen prototypes and compares these relationships across X and K.
  • Inter-class Relationship Consistency: Consistent manifolds imply that the learned mapping can preserve unseen-class relationships, motivating optimization of K toward the intrinsic class-level manifold in X.The paper states that greater consistency between the two manifolds improves the mapping’s transfer ability to unseen classes.
  • Training process: DMaP learns an initial mapping, constructs a new semantic space from local neighborhoods of mapped training instances, and alternately refines the mapping and semantic space.The three-step procedure learns f_s, generates K̃_s from the inferred class-level manifold, then iteratively aligns the two manifolds.
  • Training process: The initial mapping may use a linear model with a regularized loss, and the experiments apply squared loss with a closed-form least-squares solution.The training process can converge within the first few refinement iterations.

4. Experiments

Experiments evaluate DMaP across four datasets, multiple visual features and semantic embeddings, and both conventional and generalized ZSR settings. Results show that semantic-manifold consistency correlates with performance, while iterative alignment substantially improves recognition and achieves strong benchmark results.

  • Experimental setup: Experiments cover AwA, CUB, Dogs, and ImageNet using VGG, GoogLeNet, or ResNet features with attribute or Word2Vec embeddings.The evaluation includes conventional ZSR, generalized ZSR, and top-1 accuracy reporting on ImageNet.
  • Inter-class relationship consistency: ZSR performance is positively correlated with CM, and the CM(X|att+skipgram) > CM(X|att) > CM(X|skipgram) trend also holds for performance.The results support the role of semantic-manifold structure and indicate complementary information in attributes and word vectors.
  • Evaluation of DMaP: Using two iterations, manifold alignment improves DMaP-I by an average accuracy of 10.71%, with a 22.3% improvement on AwA.On Dogs, one iteration increases accuracy from 30.90% to 40.97%.
  • Evaluation of DMaP: Figure 4 shows fast convergence on CUB, with remarkable improvement generally achieved after one or two iterations.The plotted quantities are ZSR average accuracy and corresponding CM values across iteration counts.
  • Generalized ZSR: In generalized ZSR, DMaP-I performs considerably worse than in conventional ZSR, while DMaP-T is evaluated on predictions across all target classes.On Dogs and ImageNet, nearly all unseen test data are misclassified into seen classes before the proposed improvements.
  • Comparison with state-of-the-art: Compared with state-of-the-art methods, DMaP significantly outperforms others on all three reported datasets and reaches 44.59% on Dogs and 38.94% hit@1 on ImageNet 2012 1K.The comparison uses two alignment iterations and includes different semantic embeddings and inductive or transductive baselines.

5. Discussion and Conclusion

The paper analyzes semantic embeddings for zero-shot recognition and connects manifold structure with visual-semantic transfer ability. It introduces DMaP to align semantic and image-feature spaces, reporting state-of-the-art performance across four challenging datasets.

  • 5. Discussion and Conclusion: The analysis links inter-class semantic consistency between spaces to the transfer ability of visual-semantic mappings.The paper identifies this consistency as central to effective zero-shot recognition.
  • 5. Discussion and Conclusion: DMaP generates a semantic space more consistent with the image feature space while learning a more effective visual-semantic mapping.
  • 5. Discussion and Conclusion: The method outperforms state-of-the-art approaches on four challenging datasets.

Supplementary Material: Zero-Shot Recognition using Dual Visual-Semantic Mapping Paths

The supplementary material provides practical implementation details that were omitted from the main text.

  • Supplementary Material: Zero-Shot Recognition using Dual Visual-Semantic Mapping Paths: The supplementary material contains practical implementation details omitted from the main text.
  • Supplementary Material: Zero-Shot Recognition using Dual Visual-Semantic Mapping Paths: The material supplements the main paper with details of how the proposed approach was implemented.
  • Supplementary Material: Zero-Shot Recognition using Dual Visual-Semantic Mapping Paths: Its stated purpose is to document implementation information below the main text.

6. Implementation Details

The implementation uses regularized reconstruction and a linear visual-semantic mapping, with specified hyper-parameter settings and a closed-form solution.

  • 6. Implementation Details: The experiments use ℓ2 regularization for inter-class relationship extraction, exploiting the global structure of Xs.
  • 6. Implementation Details: The visual-semantic mapping is linear, with parameter V learned from training features Xs, labels Ys, and seen-class embeddings Ks.
  • 6. Implementation Details: The linear mapping objective includes g(V) = γ||VKs||2_F, and its solution is expressed in closed form.
  • 6. Implementation Details: The identity matrix I appears in the closed-form solution for the mapping.
  • 6. Implementation Details: λ is set to 10^-4, while γ and η are selected from 10^[1.2,1.5] and 10^[4.2,5.4], respectively.

4. Dimensions of the image features and the semantic

The experiments use deep image features from several convolutional networks and semantic embeddings from skipgram and GloVe word vectors with specified dimensions.

  • 4. Dimensions of the image features and the semantic: Deep image features are extracted using VGG, GoogLeNet, and ResNet across all datasets.
  • 4. Dimensions of the image features and the semantic: VGG and ResNet use 1000-dimensional last-fully-connected-layer activations as image features.
  • 4. Dimensions of the image features and the semantic: GoogLeNet features come from 1024-dimensional activations of the top-layer pooling units.
  • 4. Dimensions of the image features and the semantic: The semantic experiments use skipgram and GloVe word vectors trained on Wikipedia.
  • 4. Dimensions of the image features and the semantic: Skipgram and GloVe embeddings have dimensions 500 and 300, respectively.

7. Additional experimental results

Additional experiments visualize zero-shot recognition on CUB and Dogs and examine why the proposed pre-inspection step is necessary. The analysis shows that too few seen classes make unseen classes difficult to discriminate.

  • Additional recognition results: Additional experiments visualize U →T results on CUB and U →U results on Dogs.Figures 7 and 8 compare DMaP-I, DMaP-T with one iteration, and ground-truth unseen labels.
  • Pre-inspection analysis: The pre-inspection analysis splits classes at different seen/unseen ratios and measures pairwise Euclidean distances between unseen-class projections onto the seen-embedding subspace.These distances are visualized for CUB and ImageNet in Figures 9 and 10.
  • Pre-inspection analysis: When seen classes are much fewer than unseen classes, many pairwise distances approach 0, making unseen classes difficult for f_s learned from seen classes to discriminate.This observation motivates the necessity of the proposed pre-inspection step.
  • Experimental setup: Table 4 compares cZSR performance on AwA, CUB, and Dogs across methods using different semantic representations and feature settings.The table includes comparisons involving word vectors, attributes, low-level features, and transductive or inductive methods.
  • Experimental setup: The supplementary setup specifies feature and semantic-embedding conventions, including OverFeat and DeCaF features, word2vec and GloVe vectors, and WordNet-derived hierarchical embeddings.In the DMaP comparison, skipgram is used for word vectors.
Loading 1703.05002v2…