PaperlayerView exact revision on arXiv

Source-linked AI summary

Synthesized Classifiers for Zero-Shot Learning

Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, Fei Sha

arXiv:1603.00550v3cs.CV

TL;DR

Zero-shot learning seeks to recognize unseen object classes without labeled examples by relating them to seen classes through semantic descriptions. The paper aligns semantic and visual-model spaces using adaptable phantom classes whose classifiers serve as bases for synthesizing real-class classifiers. On four benchmark datasets, including ImageNet with 20,345 unseen classes, the synthesized classifiers outperform state-of-the-art methods in most scenarios.

  • Problem

    Zero-shot learning must recognize unseen classes without labeled images while relating them to seen classes and retaining discriminative performance.

  • Method

    The method aligns semantic and model spaces with manifold learning and uses optimized phantom-class coordinates as bases for convexly synthesizing real-class classifiers.

  • Results

    Across four benchmark datasets, the approach outperforms state-of-the-art methods in most scenarios, including the large-scale ImageNet setting with 20,345 unseen classes.

  • Takeaways & Limitations

    Classifier synthesis learned from labeled seen classes can be readily applied to unseen classes, supporting recognition beyond the classes represented in training data.

  • Takeaways & Limitations

    The evaluated unseen-class counts can differ because semantic embeddings, specifically word vectors, are missing for some class names.

Abstract

from arXiv · show

Given semantic descriptions of object classes, zero-shot learning aims to accurately recognize objects of the unseen classes, from which no examples are available at the training stage, by associating them to the seen classes, from which labeled examples are provided. We propose to tackle this problem from the perspective of manifold learning. Our main idea is to align the semantic space that is derived from external information to the model space that concerns itself with recognizing visual features. To this end, we introduce a set of "phantom" object classes whose coordinates live in both the semantic space and the model space. Serving as bases in a dictionary, they can be optimized from labeled data such that the synthesized real object classifiers achieve optimal discriminative performance. We demonstrate superior accuracy of our approach over the state of the art on four benchmark datasets for zero-shot learning, including the full ImageNet Fall 2011 dataset with more than 20,000 unseen classes.

1. Introduction

Zero-shot learning must connect semantic descriptions of unseen classes to visual recognition models despite lacking labeled examples. The paper aligns semantic and model spaces through manifold embeddings and phantom-class bases that synthesize classifiers, outperforming state-of-the-art methods across four benchmarks.

  • Motivation: Zero-shot learning expands recognition from labeled seen classes to unseen classes without labeled images, requiring relationships between classes and discriminative classifiers.The challenge is especially important for rare and fine-grained objects whose combinations lack enough representative training images.
  • Motivation: Existing approaches use attributes and word vectors to associate seen and unseen classes, then map images or construct classifiers through semantic or input-feature representations.These methods include embedding-based, nearest-neighbor, probabilistic, and direct classifier-construction approaches.
  • Method: The paper models semantic class descriptions and visual recognition models as related weighted graphs, aligning the semantic space with the model space through manifold learning.Model-space coordinates are treated as projections of semantic graph vertices while preserving class relatedness.
  • Method: Adaptable phantom classes have adjustable coordinates in both spaces and are optimized using labeled data so synthesized real-class models perform well on discriminative tasks.Phantom classes do not directly represent real classes; instead, they parameterize the semantic graph and connect seen and unseen classes.
  • Method: Real-class classifiers are synthesized as convex combinations of phantom-class base classifiers, with unseen-class coefficients computed from semantic coordinates.The same synthesis mechanism can be learned from seen-class labeled data and applied to unseen classes.

2. Related Work

Prior zero-shot learning methods use semantic embeddings to transfer information between seen and unseen classes, with approaches differing in where image representations and class decisions are made. The paper’s convex synthesis also relates to shared-representation multi-task learning, but unlike that setting it requires no unseen-class data.

  • Semantic embeddings: Zero-shot learning transfers knowledge through semantic embeddings such as attributes, word vectors, Web-mined knowledge, or combinations of embeddings.
  • Embedding-based methods: Embedding-based methods map image representations into semantic space before selecting labels, while related approaches combine mapping and classification stages.
  • Connections: Convex combinations of base classifiers are related to multi-task learning with shared representations, but this method does not access labeled data from unseen classes.

3. Approach

The approach aligns semantic and visual model spaces through phantom classes whose learned coordinates support synthesizing classifiers for real and unseen classes.

  • The method classifies images from unseen classes into the unseen-label space using semantic embeddings and visual-space classifiers.
  • Phantom classes are virtual objects introduced to increase modeling flexibility rather than represent real objects directly.
  • The semantic and model spaces are aligned by treating real and phantom classifiers as embeddings that preserve the weighted graph structure.
  • Each real classifier is synthesized as a convex combination of virtual phantom classifiers, which serve as dictionary bases.
  • Training uses one-versus-other or Crammer-Singer losses, with an optional distance-based structured loss that incorporates semantic class relatedness.
  • Phantom semantic embeddings are parameterized as sparse combinations of seen-class attribute vectors and optimized alongside the base classifiers.

4. Experiments

The experiments evaluate the proposed classifiers on four zero-shot-learning benchmarks using varied semantic embeddings, visual features, and protocols. Results generally favor the proposed methods, with continuous attributes, deep features, complementary semantic spaces, and fewer base classifiers showing advantages in specific analyses.

  • Setup: The study evaluates four benchmark datasets—AwA, CUB, SUN, and ImageNet—using multi-way accuracy for the first three and Flat hit@K and Hierarchical precision@K for ImageNet.ImageNet uses a distinct large-scale evaluation setup with two metrics.
  • Main results: The proposed methods achieve the best performance in most scenarios, although comparisons vary in implementation details and some published methods use different settings.The authors exclude substantially different settings from the main text for fair comparison and report them in the supplement.
  • Large-scale zero-shot learning: The proposed methods consistently outperform both ConSE baselines on ImageNet, especially in the All setting with 20,345 unseen classes.Performance for both methods and ConSE decreases as the number of unseen classes increases from 2-hop to All.
  • Detailed analysis: Continuous attributes generally outperform binary attributes, while deep features significantly outperform shallow features across methods.GoogLeNet generally outperforms AlexNet, and deep-feature results use linear classifiers.
  • Detailed analysis: On AwA, attributes outperform word-vector embeddings individually, but combining both semantic spaces yields the best result.The combined result suggests that the two semantic spaces provide complementary information.
  • Detailed analysis: Approximately 60%–70% of the seen-class count is sufficient for the base classifiers to reach the performance plateau.Increasing the number of base classifiers beyond that range has no overwhelming effect.

5. Conclusion

The paper introduces phantom classes as a classifier-synthesis mechanism for zero-shot learning and applies it to unseen classes using seen-class data. The approach outperforms state-of-the-art methods on four benchmark datasets, while accuracy becomes stable with fewer phantom classes than seen classes.

  • Conclusion: Phantom classes provide shared base classifiers and coefficient functions for constructing both seen- and unseen-class classifiers.This supports learning the synthesis mechanism from labeled seen-class data before applying it to unseen classes.
  • Conclusion: Accuracy saturates when the number of phantom classes is about 60% of the number of seen classes.Figure 2 varies R relative to S and reports accuracy relative to the setting R = S.
  • Conclusion: Our approach outperforms state-of-the-art methods on four benchmark datasets in most scenarios.

Supplementary Material: Synthesized Classifiers for

The supplementary material provides omitted details on validation, metrics, experimental setup, implementation, and additional analyses. It introduces a cross-validation strategy intended to better imitate zero-shot testing.

  • Supplementary Material: The supplementary material covers cross-validation, semantic-similarity metrics, experimental setup, implementation details, and additional results.
  • Cross-validation strategies: Conventional cross-validation is unsuitable for zero-shot learning because its folds share the same class labels, unlike zero-shot testing.
  • Cross-validation strategies: The proposed cross-validation strategy is inspired by the zero-shot setting to imitate the test-stage class split.
  • Cross-validation strategies: Hyperparameter tuning first fixes b_r = a_r and tunes λ, σ, then fixes them while tuning η, γ.

B. Learning metrics for computing similarities between semantic embeddings

This section extends semantic-similarity learning beyond an isotropic metric by learning a diagonal Mahalanobis metric, optimized together with phantom-class locations.

  • Learning metrics: The baseline graph weights use the distance d(a_c, b_r) = (a_c − b_r)^T Σ^-1(a_c − b_r).
  • Learning metrics: The method focuses on R = S and learns a diagonal metric Σ^-1 = M^T M with diagonal M.
  • Learning metrics: The optimization uses squared hinge loss and an indicator identifying whether each label matches class c.
  • Learning metrics: Alternating optimization updates the metric and phantom-class locations on different but overlapping training-data subsets to reduce over-fitting.The phantom locations use the first four of five folds, while M uses the last four folds.

C.1. Datasets

The experiments evaluate zero-shot learning across animal, bird, scene, and large-scale ImageNet datasets using class splits, semantic attributes or word vectors, and shallow or deep visual features.

  • Datasets: AwA contains 30,475 images from 50 animal classes, with 40 seen classes and 10 unseen classes in its standard split.
  • Datasets: CUB contains 200 bird classes and 11,788 images, with four random disjoint class splits each containing 50 unseen classes.
  • Datasets: SUN contains 717 scene categories and 14,340 images, with ten disjoint class splits of 71 or 72 classes.
  • Datasets: ImageNet uses ILSVRC 2012 1K as seen-class data and the non-overlapping remainder of ImageNet Fall 2011 as unseen classes.The evaluation totals 20,842 unseen classes because one ILSVRC class is absent from the ImageNet 2011 21K release.
  • Semantic representations: Class-level SUN semantic descriptions are formed by averaging the 102 continuous-valued attributes over images in each class.
  • Semantic representations: ImageNet class semantics use 500-dimensional word vectors learned with a skip-gram language model from a Wikipedia corpus exceeding 3 billion words.
  • Visual features: Experiments use shallow features where provided and deep AlexNet or GoogLeNet features, focusing on GoogLeNet for SUN and ImageNet.
  • Evaluation: Hierarchical precision@K marks at least K classes as correct by expanding a radius around the true class in the relevant label space.

D. Implementation details

The experiments detail evaluation choices for semantic embeddings, classifiers, cross-validation, and dataset splits, then report broad zero-shot results across benchmarks. Results favor class-wise cross-validation, combined semantic sources, and the proposed method over compared approaches.

  • Evaluation and tuning: Class-wise cross-validation improves CUB results by about 2% over sample-wise cross-validation.This supports simulating the zero-shot setting during hyperparameter tuning.
  • Semantic embeddings: The method accepts different semantic embedding spaces, with higher-dimensional word vectors and combined attributes plus word vectors generally performing better.Human-annotated attributes outperform automatically learned word vectors, and combining sources improves over using either separately.
  • Benchmark comparisons: On CUB, the method outperforms all compared methods despite their use of extra images or semantic embedding information.The comparison includes methods operating in transductive or semi-supervised settings and methods using additional semantic or visual resources.
  • Benchmark comparisons: On SUN, the method again shows superior performance under the alternative split with 707 seen classes and 10 unseen classes.Some compared methods use VGGverydeep-19 features under this setting.
  • Class-split analysis: Increasing the number of seen CUB classes improves zero-shot performance, whereas increasing unseen classes with 50 seen classes causes accuracy to drop.The experiments use 20 random class selections; the unseen-class analysis covers up to 150 unseen classes.
  • Extensions: Learning metrics for semantic similarity further improves the current one-versus-other formulation in preliminary AwA results.The method can also combine different semantic sources using cross-validated weights.

E.6. Detailed results and analysis of experiments on ImageNet

ImageNet evaluation compares the proposed classifiers with ConSE across hierarchical unseen-class subsets and reports performance patterns tied to semantic proximity. Accuracy declines as unseen classes become more semantically dissimilar from the seen classes.

  • Results: Ourso-vs-o and Oursstruct outperform ConSE in almost all ImageNet category-set cases.The comparison includes ConSE with T = 1, 10, and 1,000, cross-validated T, and the published best result.
  • Evaluation setup: The ImageNet All split contains 20,345 unseen classes divided into 1,509 2-hop, 6,169 pure 3-hop, and 12,667 other classes.The subsets are disjoint for reporting, although the cumulative 2-hop, 3-hop, and All sets are not mutually exclusive.
  • Analysis: Accuracies decrease from 2-hop to pure 3-hop to others for both methods, correlating semantic distance from the 1K seen classes with classification accuracy.The paper identifies poor performance on semantically too-dissimilar unseen classes as a potential limitation.

E.7. Qualitative results

Qualitative analyses examine predictions for unseen classes on AwA, CUB, and SUN, including representative images and success or failure cases. The examples show that synthesized classifiers capture useful visual properties but fail when test appearances diverge from semantically related seen classes.

  • Qualitative results: The qualitative displays rank the top-5 images assigned to each unseen class and mark misclassified images with red boundaries.AwA, CUB, and SUN use the same format, with class labels shown above and the highest-scoring misclassified example shown below.
  • Qualitative results: The method reasonably captures discriminative visual properties of unseen classes using only their semantic embeddings.Some errors involve visually similar classes that humans may also find difficult to distinguish.
  • Success and failure analysis: Success and failure analyses compare each unseen class with its top-3 semantically similar seen classes and top-3 predicted unseen classes.The figures also show convex-combination weights and representative images for the related seen and predicted classes.
  • Success and failure analysis: Misclassified test images are visually dissimilar to images from semantically similar seen classes, causing the synthesized unseen classifiers to predict incorrectly.This pattern is illustrated across AwA, CUB, and SUN.
  • Feature analysis: Deep features outperform shallow features relative to other methods, plausibly because their classifier similarities are more congruent with semantic-space similarities.The explanation is presented as a proposed interpretation rather than a demonstrated causal result.

E.9. Analysis on the number of base classifiers

The analysis studies how many phantom or base classifiers are needed to synthesize effective real-class classifiers. CUB can use substantially fewer bases without losing performance, whereas AwA requires more components to capture classifier variance.

  • Performance versus basis size: Around 60% of the seen-class count in base classifiers yields comparable or better results, especially on CUB.This indicates that fewer phantom classifiers can suffice for the fine-grained CUB benchmark.
  • Classifier variance: AwA requires more PCA components than CUB to capture 95% of variance in classifier matrices.The comparison is summarized in Figure 12.
  • Classifier variance: The greater variance requirement on AwA explains the accuracy drop observed there when using fewer base classifiers, unlike on CUB.The analysis links the basis-size sensitivity to the number of components needed to represent the learned classifiers.
Loading 1603.00550v3…