PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-Shot Visual Recognition via Bidirectional Latent Embedding

Qian Wang, Ke Chen

arXiv:1607.02104v4cs.CV

TL;DR

Zero-shot visual recognition must bridge a semantic gap between visual features and unseen-class semantics, while hubness can impair nearest-neighbor prediction. The paper proposes a two-stage bidirectional latent embedding framework and reports effective comparative performance across benchmark settings.

  • Problem

    Zero-shot visual recognition struggles because visual and semantic spaces differ, making knowledge transfer to unseen classes difficult; hubness further harms existing models.

  • Method

    The framework learns a discriminative latent space from labeled visual data, then embeds unseen-class semantics into it using landmark-guided semi-supervised mapping.

  • Results

    Comparative experiments demonstrate effectiveness, including 79.1% per-class accuracy with Vgg19 features on AwA in the attribute-based inductive setting.

  • Takeaways & Limitations

    The learned latent space supports nearest-neighbor prediction while preserving semantic relatedness across known and unseen classes.

  • Takeaways & Limitations

    The authors identify extending the framework to other zero-shot problems, such as multi-label visual recognition, as future work.

Abstract

from arXiv · show

Zero-shot learning for visual recognition, e.g., object and action recognition, has recently attracted a lot of attention. However, it still remains challenging in bridging the semantic gap between visual features and their underlying semantics and transferring knowledge to semantic categories unseen during learning. Unlike most of the existing zero-shot visual recognition methods, we propose a stagewise bidirectional latent embedding framework to two subsequent learning stages for zero-shot visual recognition. In the bottom-up stage, a latent embedding space is first created by exploring the topological and labeling information underlying training data of known classes via a proper supervised subspace learning algorithm and the latent embedding of training data are used to form landmarks that guide embedding semantics underlying unseen classes into this learned latent space. In the top-down stage, semantic representations of unseen-class labels in a given label vocabulary are then embedded to the same latent space to preserve the semantic relatedness between all different classes via our proposed semi-supervised Sammon mapping with the guidance of landmarks. Thus, the resultant latent embedding space allows for predicting the label of a test instance with a simple nearest-neighbor rule. To evaluate the effectiveness of the proposed framework, we have conducted extensive experiments on four benchmark datasets in object and action recognition, i.e., AwA, CUB-200-2011, UCF101 and HMDB51. The experimental results under comparative studies demonstrate that our proposed approach yields the state-of-the-art performance under inductive and transductive settings.

1 Introduction

Zero-shot visual recognition transfers knowledge from known visual classes to unseen classes, but semantic gaps and hubness make this difficult. The paper proposes a stagewise bidirectional latent embedding framework and evaluates it across object and action recognition benchmarks.

  • Zero-shot learning aims to recognize classes unseen during learning by exploiting semantic relatedness between known and unseen classes.
  • The central challenge is transferring knowledge from low-level visual representations of known classes to unseen classes through their underlying semantics.
  • Semantic gaps arise because visual-feature distributions can differ from semantic distributions, while hubness causes nearest-neighbor methods to favor a few high-dimensional-space instances.
  • The proposed framework learns a visual latent space bottom-up, forms class landmarks from projected training means, and embeds unseen-class semantics top-down using those landmarks.
  • The framework extends to multiple visual or semantic representations and transductive settings, with experiments on several object and action recognition benchmarks.

2 Related Work

Related zero-shot methods include direct mapping, model parameter transfer, and common-space learning. The paper distinguishes its two-stage common-space strategy from joint embedding approaches and uses subspace-learning techniques to preserve useful structure.

  • Zero-Shot Visual Recognition: Direct mapping learns a function from visual features to semantic representations and assigns unseen labels using semantic relatedness.
  • Zero-Shot Visual Recognition: Model parameter transfer estimates unseen-class models from known-class parameters, but may lack sufficient information because relationships among unseen classes are not considered.
  • Zero-Shot Visual Recognition: Common-space learning projects visual and semantic representations into a shared space to bridge the semantic gap in zero-shot recognition.
  • Zero-Shot Visual Recognition: Unlike methods that jointly embed visual and semantic spaces, the proposed framework uses two subsequent stages and explicitly addresses intra-class and inter-class variability.
  • Subspace Learning: Supervised locality preserving projection learns a low-dimensional latent space from visual data while using label information and preserving locality.
  • Subspace Learning: Semi-supervised Sammon mapping extends non-parametric multidimensional scaling by fixing some embeddings and learning the remaining ones.

3 Bidirectional Latent Embedding

BiDiLEL learns a latent space from labeled visual data, then embeds unseen-class semantics into that space using training-class landmarks. Recognition assigns each test instance to the nearest unseen-class embedding.

  • Overview: BiDiLEL uses sequential bottom-up and top-down stages to create a shared latent space for zero-shot visual recognition.The bottom-up stage learns the space from visual data; the top-down stage embeds unseen-class labels into it.
  • Bottom-up Latent Space Learning: The bottom-up stage learns a projection with supervised locality preserving projection to preserve same-class locality and increase class separability.Training-class embeddings are estimated as normalized means of projected training instances and used as landmarks.
  • Bottom-up Latent Space Learning: The latent space represents training-class embeddings as landmarks that guide the placement of unseen-class labels.These landmarks are the normalized mean points of the training classes in the learned latent space.
  • Top-down Latent Embedding Learning: Landmark-based Sammon mapping embeds unseen-class labels while preserving semantic relatedness among known and unseen classes.The method treats the fixed training-class embeddings and semantic distances as a semi-supervised multidimensional scaling problem.
  • Zero-Shot Recognition: Zero-shot recognition projects a test instance into the latent space and assigns the nearest unseen-class embedding.The framework uses a nearest-neighbor rule after the two embedding stages are completed.
  • Extensions: The framework is extended to multiple complementary visual representations and transductive structured prediction.Structured prediction establishes a one-to-one correspondence between clusters and unseen classes.

4 Experimental Settings

The experiments evaluate the framework across four object and action-recognition datasets using multiple visual and semantic representations. They also investigate hyper-parameter effects and tune them through classwise cross-validation.

  • Datasets: Experiments use four benchmark datasets: AwA and CUB-200-2011 for object recognition, plus UCF101 and HMDB51 for human action recognition.The datasets provide established settings for comparing zero-shot visual recognition methods.
  • Datasets: AwA contains 30,475 images from 50 classes with a 40/10 known/unseen class split.
  • Datasets: CUB-200-2011 contains 11,788 images from 200 bird species and uses a 100/50/50 training/validation/test class split.
  • Datasets: UCF101 uses 51/50 and 81/20 known/unseen class splits, while HMDB51 uses 26/25 splits with 30 independent splits for the evaluated settings.
  • Representations: The study combines deep CNN image features, IDT and C3D video representations, and attribute or word-vector semantics, including fused semantic distances.The fused distance is Δ = γΔWV + (1 − γ)ΔAtt, with γ tuned by grid search.
  • Hyper-parameter selection: Classwise cross-validation tunes α, dy, kG, and optionally kST using coarse grid search followed by sequential fine-tuning.The fusion weight γ is separately selected from values 0.1 through 0.9.

5 Experimental Results

Experiments across object and human-action recognition evaluate hyperparameters, enabling techniques, and comparative performance under inductive and transductive settings. The framework generally performs favorably, with results showing dataset- and representation-dependent benefits from bottom-up learning, top-down embedding, and transductive post-processing.

  • Enabling techniques: SLPP generally performs best for bottom-up learning, while LSM generally outperforms SVR for top-down unseen-class embedding.SLPP preserves visual structure and discrimination, whereas LSM and SVR differ in their knowledge-transfer behavior.
  • Enabling techniques: Bottom-up learning improves performance on three datasets but may be redundant on AwA when deep features already capture intrinsic visual clusters.The authors attribute poor LDA generalization to overfitting and conclude that bottom-up learning should preserve structure while promoting discrimination.
  • Zero-shot object recognition: On AwA, Vgg19 reaches 79.1% per-class accuracy inductively, while structured prediction raises transductive accuracy to 95.0%.Self-training reaches 86.2% with GoogLeNet and 88.5% with Vgg19 before structured prediction further improves both results.
  • Zero-shot object recognition: On CUB-200-2011, the approach reaches 49.7% inductively with GoogleNet and 62.8% under attribute-based transductive evaluation using structured prediction.For word-vector settings, Vgg19 reaches 37% inductively, while self-training and structured prediction reach 40.9% and 40.6%, respectively.
  • Zero-shot human action recognition: On UCF101 and HMDB51, the approach achieves the highest reported averages across both inductive and transductive classwise splits.For example, transductive averages reach 29.8% and 23.0% on the 51/50 split, and 57.1% and 49.3% on the 81/20 split, using attributes and word vectors respectively.
  • Comparative study: Overall, comparative experiments find the framework favorable and generally comparable to existing state-of-the-art zero-shot visual recognition methods.The study evaluates object and action recognition datasets under both inductive and transductive settings.

6 Concluding Remarks

The paper presents a bidirectional latent embedding framework for zero-shot visual recognition and reports competitive performance across four benchmark datasets. It also identifies extensions to other zero-shot tasks and domains as future work.

  • 6 Concluding Remarks: The framework learns a latent space bottom-up from visual structure and labels, then embeds unseen-class semantics top-down using landmark guidance.Test labels are predicted by nearest-neighbor matching in the resulting latent space.
  • 6 Concluding Remarks: Comparative evaluation across four benchmark datasets finds performance competitive with most state-of-the-art zero-shot visual recognition approaches.
  • 6 Concluding Remarks: Future work includes improving enabling techniques and extending the framework to multi-label zero-shot visual recognition.
  • 6 Concluding Remarks: The authors also anticipate applications to zero-shot audio classification, music genre recognition, and multimedia information retrieval.

Appendix A Derivation of Gradient on the LSM Cost Function

The appendix derives the gradient of the LSM cost function with respect to unseen-class latent embeddings. It rewrites the cost using simplified distance notation and applies the chain rule to obtain the gradient.

  • Appendix A Derivation of Gradient on the LSM Cost Function: The derivation simplifies notation by defining distance metrics in the latent and semantic spaces.
  • Appendix A Derivation of Gradient on the LSM Cost Function: The LSM cost function is rewritten using the simplified notation before differentiation.
  • Appendix A Derivation of Gradient on the LSM Cost Function: The gradient with respect to unseen-class embeddings is derived by applying the chain rule.
  • Appendix A Derivation of Gradient on the LSM Cost Function: The derivation separately evaluates the first and second terms of the rewritten cost function.
  • Appendix A Derivation of Gradient on the LSM Cost Function: Substituting the component derivatives into the chain-rule expression yields the gradient of the LSM cost with respect to the unseen-class embeddings.

Appendix B Extension to the Joint Use of Multiple Visual Representations

The appendix extends the framework to multiple visual representations by combining their similarity and kernel representations before learning one projection. Landmark generation and top-down learning remain unchanged, while test instances are fused and classified in the latent space.

  • Appendix B Extension to the Joint Use of Multiple Visual Representations: Multiple visual representations are mapped into a common kernel space so their latent embedding can be learned jointly despite differing dimensionalities.
  • Appendix B Extension to the Joint Use of Multiple Visual Representations: Similarity and kernel matrices from the representations are combined using arithmetic averages.
  • Appendix B Extension to the Joint Use of Multiple Visual Representations: The method assumes equal contribution from different visual representations and does not address selecting a proper weighted fusion algorithm.
  • Appendix B Extension to the Joint Use of Multiple Visual Representations: The combined representation is used to learn the projection, while landmark generation and top-down learning remain unchanged.
  • Appendix B Extension to the Joint Use of Multiple Visual Representations: For a test instance, representations are averaged through a combined kernel, projected into the latent space, and assigned the nearest class label.

Appendix C Visual Representation Complementarity Measurement and Selection

The appendix defines and uses a complementarity measure to select diverse visual representations for joint zero-shot recognition. On UCF101 and HMDB51, incrementally adding the selected representations consistently improves performance, with the combined set outperforming any single representation.

  • C.1 The Complementarity Measurement: The proposed complementarity measure is motivated by local-distribution diversity because prior work lacked quantitative measurement of representation complementarity.
  • C.1 The Complementarity Measurement: For two representations, complementarity compares their nearest-neighbor label subsets and increases as the representations become more complementary.
  • C.1 The Complementarity Measurement: The measure extends from one pair of representations to one representation and a set of remaining representations using set unions.
  • C.2 Finding Complementary Visual Representations: The selection algorithm initializes with the best-performing candidate, then repeatedly adds the candidate with highest complementarity until a predefined stopping condition.
  • C.3 Application in Zero-shot Human Action Recognition: On UCF101 and HMDB51, the selected set contains C3D and four FV-based IDT representations, and performance consistently improves as representations are added.
  • C.3 Application in Zero-shot Human Action Recognition: The joint-use extension combines multiple representations through an averaged kernel, while the appendix anticipates broader pattern-recognition applications.
Loading 1607.02104v4…