PaperlayerView exact revision on arXiv

Source-linked AI summary

Transductive Multi-view Zero-Shot Learning

Yanwei Fu, Timothy M. Hospedales, Tao Xiang, Shaogang Gong

arXiv:1501.04560v2cs.CVcs.DScs.MM

TL;DR

Conventional zero-shot learning applies auxiliary-domain projections to disjoint target classes and represents each target class with only one prototype. The paper aligns low-level and semantic views through transductive multi-view embedding and uses TMV-HLP for label propagation. It reports state-of-the-art recognition across AwA, CUB, and USAA while enabling cross-view annotation tasks.

  • Problem

    Conventional zero-shot learning is limited by projection domain shift between disjoint auxiliary and target classes and by prototype sparsity.

  • Method

    The paper combines transductive multi-view embedding with heterogeneous multi-view hypergraph label propagation to align views and propagate labels from prototypes through target-data manifolds.

  • Results

    The approach significantly outperforms existing methods for zero-shot and N-shot recognition on three image and video benchmark datasets.

  • Takeaways & Limitations

    The learned embedding supports zero-shot, N-shot, and zero+N-shot recognition and enables novel cross-view annotation tasks.

  • Takeaways & Limitations

    With very few training labels, N-shot TMV-HLP+ accuracy can initially decrease before improving as more labels are added.

Abstract

from arXiv · show

Most existing zero-shot learning approaches exploit transfer learning via an intermediate-level semantic representation shared between an annotated auxiliary dataset and a target dataset with different classes and no annotation. A projection from a low-level feature space to the semantic representation space is learned from the auxiliary dataset and is applied without adaptation to the target dataset. In this paper we identify two inherent limitations with these approaches. First, due to having disjoint and potentially unrelated classes, the projection functions learned from the auxiliary dataset/domain are biased when applied directly to the target dataset/domain. We call this problem the projection domain shift problem and propose a novel framework, transductive multi-view embedding, to solve it. The second limitation is the prototype sparsity problem which refers to the fact that for each target class, only a single prototype is available for zero-shot learning given a semantic representation. To overcome this problem, a novel heterogeneous multi-view hypergraph label propagation method is formulated for zero-shot learning in the transductive embedding space. It effectively exploits the complementary information offered by different semantic representations and takes advantage of the manifold structures of multiple representation spaces in a coherent manner. We demonstrate through extensive experiments that the proposed approach (1) rectifies the projection shift between the auxiliary and target domains, (2) exploits the complementarity of multiple semantic representations, (3) significantly outperforms existing methods for both zero-shot and N-shot recognition on three image and video benchmark datasets, and (4) enables novel cross-view annotation tasks.

1 INTRODUCTION

The paper identifies projection domain shift, prototype sparsity, and limited multi-view exploitation as central weaknesses of conventional zero-shot learning, then proposes a transductive multi-view framework to address them.

  • Motivation: Conventional zero-shot learning transfers a projection learned on labelled auxiliary classes to unlabelled target classes through a shared semantic representation.Typical representations include visual attributes and continuous word vectors.
  • Projection domain shift: Disjoint auxiliary and target classes make the transferred projection biased because their data distributions and ideal projection functions differ.The paper names this mismatch the projection domain shift problem.
  • Prototype sparsity: A single semantic prototype per target class is insufficient to represent intra-class variation or resolve inter-class similarities.This is the prototype sparsity problem.
  • Multi-view information: Existing approaches also underuse complementary information from multiple semantic views such as visual attributes and word vectors.The paper treats this as a further limitation of conventional zero-shot learning.
  • Proposed framework: Transductive multi-view embedding aligns biased semantic projections with target low-level features while exploiting multiple semantic spaces.The transductive setting uses unlabelled target data to improve generalisation.
  • Proposed framework: TMV-HLP propagates labels from sparse prototypes through heterogeneous hypergraphs, exploiting complementary views and target-data manifold structure.The framework supports zero-shot, N-shot, and combined zero+N-shot recognition.

2 RELATED WORK

Related work covers semantic representations and graph-based semi-supervised learning, while positioning heterogeneous hypergraphs as a way to combine complementary views and higher-order relations.

  • Semantic spaces for zero-shot learning: Prior zero-shot methods use attribute-based representations for images and videos, often with manually specified attribute ontologies.These methods treat prototypes as the few labelled instances in a transductive semi-supervised formulation.
  • Semantic spaces for zero-shot learning: Existing work generally recognizes only novel classes, while unifying zero-shot and supervised learning remains open.The paper’s method extends beyond zero-shot when labelled target instances are available.
  • Graph-based label propagation: Multi-view graph classification commonly mixes separate graphs, whereas this paper represents pairwise similarity through distributed heterogeneous hypergraphs.Hypergraphs connect many nodes and can encode higher-order relations across complementary views.

3 LEARNING A TRANSDUCTIVE MULTI-VIEW EMBEDDING SPACE

The framework projects target instances from low-level and semantic views into a shared transductive embedding space, using multi-view CCA and eigenvalue-based weighting to align and compare them.

  • 3.1 Problem setup: Target instances contain a low-level feature view and predicted projections into multiple semantic spaces, while target semantic labels are unavailable.The task is to estimate target class labels using projected semantics and class prototypes.
  • 3.1 Problem setup: The paper uses visual attributes and word vectors as complementary semantic representations alongside the low-level feature space.Word vectors are obtained from class names using a skip-gram model trained on English Wikipedia.
  • 3.2 Semantic projections: Projection functions map low-level features into each semantic space, but their target predictions exhibit projection domain shift.Support vector classifiers produce attribute projections and support vector regressors produce word-vector projections.
  • 3.3 Transductive multi-view embedding: Multi-view CCA learns projections for three target views into a shared embedding space, enabling alignment and direct comparison across views.The views are low-level features, word-vector projections, and attribute projections.
  • 3.3 Transductive multi-view embedding: The embedding retains all input dimensions and uses eigenvalues for soft weighting rather than selecting a lower-dimensional CCA subspace.The embedding dimensionality is the sum of the input view dimensions.
  • 3.3 Transductive multi-view embedding: Final view representations are weighted using a diagonal eigenvalue matrix, with power weight λ empirically set to 4, and compared using cosine similarity.The resulting representations are denoted Ψi in the common space Γ.

4 RECOGNITION BY MULTI-VIEW HYPERGRAPH LABEL PROPAGATION

The method represents target instances and class prototypes across multiple views, then uses heterogeneous hypergraphs and transductive label propagation to support zero-shot and N-shot recognition.

  • Multi-view representation: Target classes provide semantic prototypes in each view, while instances use low-level features and projected attribute and word representations.The prototypes and unlabelled instances are projected into a shared embedding space; the low-level-view prototype is synthesized when absent.
  • Label propagation: Label propagation exploits the manifold structure of unlabelled target data, using prototypes as labelled nodes for zero-shot learning and instances as labelled nodes for N-shot learning.The closed-form solution is Ẑ = η(ηΠ + L)^−1ΠZ.
  • Heterogeneous hypergraphs: Nearest-neighbour hyperedges group nodes across heterogeneous views, allowing similarity to be computed between hyperedges rather than individual nodes.For three views, the construction produces six across-view heterogeneous hypergraphs.
  • Heterogeneous hypergraphs: Hyperedge similarity uses inner-product-based similarities between projections and normalization to make heterogeneous similarity sets more comparable.The method selects K highest normalized similarities and uses soft incidence matrices to incorporate hyperedge strength.
  • Heterogeneous hypergraphs: The hypergraph representation distributes similarity across multiple connections, improving robustness to noisy nodes while potentially blurring class boundaries.The framework fuses hypergraphs with 2-graphs to recover discriminative power.

5 ANNOTATION AND BEYOND

The shared embedding connects low-level features with semantic views for target-instance annotation and cross-view class description or naming tasks.

  • Cross-view embedding: The embedding space bridges low-level features with attribute and word representations through cross-view mapping.This mapping supports transfer between visual attributes, linguistic naming, and low-level feature views.
  • Instance-level annotation: Target instances can receive attribute annotations by aligning their semantic projections with low-level features in the unlabelled target domain.This addresses the projection domain shift affecting direct attribute prediction.
  • Zero-shot class description: A novel class name can be mapped from word space to an inferred semantic attribute description.The task can provide a zero-shot attribute prototype without defining it through experts or an ontology.
  • Zero prototype learning: An attribute prototype can be mapped to a class name, enabling validation of proposed prototypes and semantic-property-based dictionary indexing.The paper presents this as the first attempt to evaluate class attribute prototypes by linking linguistic and visual attribute spaces.

6 EXPERIMENTS

Experiments on AwA, USAA, and CUB compare TMV-HLP with state-of-the-art methods and analyze its embedding, weighting, propagation, and annotation behavior. The method achieves strong zero-shot results, benefits from selected multi-view embeddings, and supports cross-view annotation tasks.

  • Benchmark evaluation: TMV-HLP is evaluated against state-of-the-art zero-shot methods on AwA, USAA, and CUB using multiple feature and semantic-representation settings.The benchmarks include image and video datasets, with comparisons spanning attribute-only, linguistic, combined, and other semantic approaches.
  • AwA results: 49.0% accuracy on AwA with hand-crafted features surpasses the previous 48.3% best result, despite that baseline requiring additional human annotation.Using OverFeat and DeCAF raises TMV-HLP accuracy to 73.5% and 80.5%, respectively, under the reported settings.
  • Embedding and propagation: Soft CCA weighting outperforms hard dimension selection by weighting all dimensions according to their CCA eigenvalues.Label propagation helps before and after CCA embedding, while CCA improves nearest-neighbor classification; combining both works best.
  • View combination: Embedding multiple views generally improves separability, but selecting six deep-feature views outperforms embedding all nine available views.The t-SNE visualization shows target classes becoming clearly separable after embedding all six OverFeat-based views.
  • Cross-view annotation: The embedding space supports attribute annotation and zero-shot description, including an F-measure of 0.236 versus 0.063 for direct word-space selection.For instance-level USAA annotation, TMV-HLP reports FM 0.341 and mAP 0.355, compared with baseline FM 0.299 and mAP 0.267.
  • Cross-view annotation: Embedding-space queries recover USAA class names from attributes at average rank 2.13, versus 110.24 for direct word-space queries.The task also evaluates incrementally adding attributes to a birthday-party query.

7 CONCLUSIONS

The paper addresses projection domain shift with transductive multi-view embedding and uses TMV-HLP to exploit semantic views and unlabelled-data structure across recognition and annotation tasks.

  • Transductive multi-view embedding rectifies biased projections between auxiliary and target datasets.
  • TMV-HLP uses heterogeneous across-view hypergraphs to exploit multiple semantic representations and unlabelled target-data manifolds.
  • The unified approach supports zero-shot, N-shot, and zero+N-shot learning tasks.
  • Experiments report state-of-the-art performance on AwA, CUB, and USAA.
  • The framework enables tasks relating textual class names to semantic attributes.

8 FURTHER EVALUATIONS ON ZERO-SHOT LEARNING

Further evaluations test graph choices, alternative transductive methods, embedding-data requirements, qualitative predictions, and auxiliary-versus-target CCA learning for zero-shot recognition.

  • 8.1 Heterogeneous hypergraph vs. other graphs: The Hete-hyper+2-gr graph model yields the best performance on both AwA and USAA.All graph models benefit from transductive embedding, while heterogeneous hypergraphs degrade drastically without view alignment.
  • 8.2 Comparing with other transductive methods: TMV-HLP in the embedding space outperforms C-MG and PST, while embedding also improves both alternatives.The improvement is attributed to alleviated projection domain shift through alignment of semantic projections and low-level features.
  • 8.2 Comparing with other transductive methods: TMV-HLP outperforms PST through multiple semantic views, hypergraph propagation, and less dependence on initial labelling heuristics.
  • 8.3 Transductive embedding: The CCA embedding learned from auxiliary data gives reasonable performance but performs worse than embedding learned transductively from unlabelled target samples.
  • 8.4 Qualitative results: TMV-HLP produces more reasonable top-five ranked class lists on AwA than DAP and PST.

9 N-SHOT LEARNING

N-shot experiments compare settings with and without prototypes across three datasets, showing that prototype-assisted TMV-HLP performs strongly with few labels but can behave non-monotonically.

  • 9 N-shot learning: TMV-HLP+ always achieves the best performance, particularly when few training examples are available.
  • 9 N-shot learning: Transductive label-propagation methods outperform methods that do not use transductive learning.
  • 9 N-shot learning: With more labelled instances, TMV-HLP- begins to outperform PST+ because it combines different views of training instances.The stronger effect of prototypes is eventually outweighed.
  • 9 N-shot learning: On AwA, TMV-HLP+ accuracy can decrease before increasing as more labelled instances are added.This occurs when only a few training labels are observed.
Loading 1501.04560v2…