PaperlayerView exact revision on arXiv

Source-linked AI summary

Predicting Visual Exemplars of Unseen Classes for Zero-Shot Learning

Soravit Changpinyo, Wei-Lun Chao, Fei Sha

arXiv:1605.08151v2cs.CV

TL;DR

Zero-shot learning must recognize unseen classes despite limited semantic representations and no labeled examples for those classes. This paper learns kernel-based mappings from seen-class semantics to visual exemplars, then uses predicted exemplars for classification or semantic refinement. Across standard benchmarks, including large-scale ImageNet, the method reports state-of-the-art or broadly superior performance, while results vary with evaluation conditions and semantic representations.

  • Problem

    Zero-shot learning is difficult because semantic representations may be inadequate and unseen-class data makes model selection challenging.

  • Method

    The method imposes a structural constraint requiring semantic representations to predict visual exemplars and learns this mapping from seen classes.

  • Results

    The approach achieves state-of-the-art performance on multiple standard benchmark datasets and outperforms baselines on ImageNet, including a 2% F@1 improvement on 2-hop with word-vector semantics.

  • Takeaways & Limitations

    Predicting visual exemplars provides a way to exploit structural relatedness between visual features and semantic information for zero-shot classification.

  • Takeaways & Limitations

    On ImageNet, the improvement is less significant when using MDS-WordNet semantic features, and SUN standard deviations may be unreliable because each class has only 20 images.

Abstract

from arXiv · show

Leveraging class semantic descriptions and examples of known objects, zero-shot learning makes it possible to train a recognition model for an object class whose examples are not available. In this paper, we propose a novel zero-shot learning model that takes advantage of clustering structures in the semantic embedding space. The key idea is to impose the structural constraint that semantic representations must be predictive of the locations of their corresponding visual exemplars. To this end, this reduces to training multiple kernel-based regressors from semantic representation-exemplar pairs from labeled data of the seen object categories. Despite its simplicity, our approach significantly outperforms existing zero-shot learning methods on standard benchmark datasets, including the ImageNet dataset with more than 20,000 unseen categories.

1. Introduction

Zero-shot learning addresses recognition when labeled examples for unseen classes are unavailable, but semantic transfer and model selection remain difficult. The paper exploits clustering structure by requiring semantic representations to predict visual class exemplars, yielding strong benchmark performance.

  • Motivation: Zero-shot learning transfers knowledge from labeled seen classes to unseen classes through a shared semantic space.The space may use attributes, word vectors, textual descriptions, ontologies, or human gazes.
  • Challenges: 1.5% accuracy on full ImageNet with 21K categories illustrates the difficulty of zero-shot recognition.The cited supervised-learning figure is 29.8%, though the comparison uses different training/test splits.
  • Challenges: Semantic representations can be nondiscriminative, difficult for machines to detect, correlated, or not category-independent.These properties make semantic representations difficult bases for knowledge transfer.
  • Challenges: ZSL model selection must balance flexible functions that overfit seen classes against simple functions that perform poorly on both seen and unseen classes.The paper emphasizes that transfer depends on the assumed mechanism matching the data.
  • Contribution: The proposed method imposes a predictive constraint: semantic representations should predict visual cluster centers, while cluster structure further regularizes the model.A kernel-based regressor learns this mapping from seen-class semantic representations to visual exemplars and applies it to unseen classes.
  • Results: Across four benchmark datasets, including ImageNet with more than 20,000 unseen classes, the approach outperforms existing ZSL methods in most cases.The paper also studies extensions from zero-shot to few-shot learning.

2. Approach

The approach learns a transformation from semantic representations to visual exemplars and uses predicted unseen-class exemplars for nearest-neighbor classification or improved semantic representations. Its class-level design supports scalability while retaining a nonlinear mapping and structural regularization.

  • Core approach: The method constrains semantic representations to predict visual exemplars, or cluster centers, and applies the learned function to novel semantic representations.The approach first learns the function and then uses it for zero-shot learning.
  • Exemplar construction: Each visual exemplar is formed by averaging PCA-projected data for a class, with one fixed PCA projection computed over seen-class training data.The passages define exemplars through class-level averages in the projected visual feature space.
  • Regression: The model learns d RBF-kernel support vector regressors, each predicting one visual-exemplar dimension from its semantic representation.PCA also decorrelates feature dimensions so they can be predicted independently.
  • Classification: For unseen classes, predicted exemplars can serve directly as nearest-neighbor data points for classification.The classifier assigns each novel input the label of the closest predicted exemplar using standardized Euclidean distance.
  • Integration: Predicted exemplars can alternatively replace semantic embeddings in existing ZSL frameworks such as ConSE and SynC.In SynC, they define similarities to base classes used to construct classifiers.
  • Scalability: Runtime and memory depend on the number of seen classes rather than training instances because learning and prediction occur at the exemplar level.The method uses a highly nonlinear transformation and contrasts with instance-level approaches.

3. Experiments

Across four benchmark datasets, the method predicts visual exemplars that capture unseen-class structure and improves zero-shot recognition, including large-scale ImageNet evaluation. Experiments also examine exemplar representations, distance metrics, few-shot protocols, dimensionality reduction, and regressor choice.

  • Experimental setup: The evaluation covers AwA, CUB, SUN, and full ImageNet, using diverse datasets and established protocols for comparison.ImageNet contains more than 20,000 unseen classes in the evaluation.
  • Predicted visual exemplars: Predicted exemplar distances correlate more strongly with real-exemplar distances than semantic-representation distances, despite no unseen-class data.The comparison uses pairwise Euclidean distance matrices over unseen classes.
  • Predicted visual exemplars: Predicted exemplars align with real images for many unseen classes, while clustering is more mixed on fine-grained CUB.The visualizations compare real images and predicted exemplars across AwA, CUB, SUN, and ImageNet.
  • Main results: The approach significantly outperforms recent state-of-the-art baselines in multi-way classification when using GoogLeNet features.The results are summarized across all four benchmark datasets.
  • Main results: EXEM improves over corresponding base methods by 5.9–6.8% for SYNC, 11.4–27.6% for CONSE, and 1.1–17.1% for LATEM.These relative improvements result from treating predicted visual exemplars as ideal semantic representations.
  • Large-scale zero-shot classification results: On ImageNet 2-hop evaluation, the method improves F@1 by 2% over the state of the art using word-vector semantics, but gains are smaller with MDS-WordNet features.The method also finds that 1-nearest-neighbor classifiers can outperform more powerful semantic-representation classifiers at very high class counts.
  • From zero-shot to few-shot learning: In zero-to-few-shot learning, EXEM leads with few peeked unseen classes, whereas SYNC eventually surpasses it as more labeled unseen data becomes available.The protocol reserves 20% of each unseen class for optional training and uses the remaining 80% for testing.
  • Analysis: SVR is more robust than MLP, while reducing PCA dimension from 1024 to 500 does not hurt EXEM (1NN) performance.The paper attributes MLP’s weaker robustness partly to overfitting with few seen-class training pairs.

4. Related Work

Related ZSL approaches connect visual features and unseen classes through semantic representations, while exemplar prediction reverses this direction by mapping class semantics to visual feature locations. This work differs from image-generation methods by deterministically predicting feature exemplars rather than synthesizing images.

  • ZSL methods commonly use attributes or word vectors to relate visual features of seen and unseen classes.
  • The proposed exemplar prediction performs inverse prediction: class semantic representations are mapped to visual exemplar locations in the semantic embedding space.
  • Unlike probabilistic image-generation methods, the approach deterministically predicts visual features directly, avoiding an intermediate step of generating realistic images.

5. Discussion

The paper proposes a simple ZSL model that predicts visual exemplars for unseen classes and uses them through semantic and prototypical-data views. It reports state-of-the-art performance across multiple benchmark datasets and extends the analysis to few-shot learning.

  • The model predicts cluster-center visual exemplars that characterize unseen-class visual features.
  • Predicted exemplars support zero-shot classification through ideal semantic representations and prototypical data points.
  • The approach achieves state-of-the-art performance on multiple standard benchmark datasets and includes an extension from zero-shot to few-shot learning.

Supplementary Material: Predicting Visual Exemplars of

The supplementary material adds methodological, experimental, analytical, and generalized-learning details beyond the main text. It covers the proposed method, evaluation setup, expanded results, few-shot learning, and PCA analysis.

  • Supplementary Section A provides details of the proposed zero-shot learning method.
  • Supplementary Section B documents experimental setup details, including datasets, baselines, and hyper-parameter tuning.
  • Supplementary Sections C and D expand predicted-exemplar and zero-shot-learning results with additional metrics, visualizations, baselines, and qualitative results.
  • Supplementary Sections E and F report generalized ZSL and zero-shot-to-few-shot learning results, including selection of peeked unseen classes.
  • Supplementary Sections G and H analyze PCA dimensionality and provide multi-layer perceptron details.

A. Details on our proposed zero-shot learning method

The exemplar predictor is implemented as independent RBF-kernel support vector regressors, one for each visual-exemplar dimension. Their outputs are assembled into the predicted exemplar representation, with shared tuned hyperparameters across dimensions.

  • The method learns d independent support vector regressors with an RBF kernel from semantic representation–visual exemplar pairs.
  • Each regressor predicts one dimension of the visual exemplar using the ν-SVR formulation.
  • The resulting predictor ψ(·) concatenates the outputs of the d regressors, each with its own target values and parameters.

B.1. Additional information on datasets

The experiments use four benchmark datasets and compare the proposed method primarily with SYNC under established ZSL protocols. Hyperparameters are selected through class-wise validation that simulates zero-shot learning, with exemplar quality assessed by distance and classification accuracy.

  • Datasets: Experiments cover AwA, CUB, SUN, and ImageNet, spanning animal, bird, scene, and large-scale object categories.ImageNet includes more than 20,000 unseen classes.
  • Baselines: SYNC is the primary baseline because prior evaluations identify it as a strong performer across the same datasets and settings.The comparison also includes BIDILEL, LATEM, and CCA with matched semantic representations, visual features, and metrics.
  • Validation: Class-wise cross-validation holds out seen classes as pseudo-unseen classes to simulate the target ZSL setting during hyperparameter tuning.The method fixes d = 500 and finds λ = 1 robust across datasets, while tuning remaining parameters by class-wise validation.
  • Evaluation: CV-distance measures Euclidean distance between predicted and ground-truth class exemplars, whereas CV-accuracy maximizes zero-shot classification accuracy.CV-accuracy generally performs slightly better for EXEM variants without additional hyperparameters.
  • Evaluation: Variants that combine EXEM with SYNC, CONSE, or LATEM require separate hyperparameter tuning after regressor tuning.This staged procedure uses CV-distance for regressors and then tunes the downstream classifier.

C.1. Another metric for evaluating the quality of predicted visual exemplars

The supplementary analysis evaluates whether predicted exemplars preserve visual neighborhood structure. It reports nearest-neighbor overlap and visualizations showing that relative distances can matter even when predictions are not exactly aligned with real images.

  • Neighborhood preservation: Nearest-neighbor overlap compares semantic-representation or predicted-exemplar neighborhoods against neighborhoods defined by real exemplar distances.The analysis reports overlap percentages for AwA, CUB, and SUN.
  • Neighborhood preservation: Pearson correlations are computed between rows of distance matrices to compare semantic and visual relationships.This supplements the nearest-neighbor overlap analysis.
  • Visual alignment: Predicted exemplars are often well aligned with real images across AwA, CUB, SUN, and ImageNet visualizations.Each class is shown with real images and predicted exemplars using distinct colors and marker shapes.
  • Visual alignment: Relative distance is sufficient for classification even when a predicted exemplar is not exactly aligned with its corresponding images.On AwA, an orange test image can still be correctly classified when the orange exemplar is the closest among class exemplars.

D.1. Expanded main results on small datasets

Expanded small-dataset results examine metric choices, feature differences, additional baselines, and qualitative predictions. The method remains competitive across these variations and outperforms baselines on CUB despite differing experimental settings.

  • Evaluation settings: Per-sample accuracy is more optimistic than per-class accuracy, so results using the two metrics should not be treated as directly equivalent.Prior work reported difficulty reproducing a per-sample-accuracy method using per-class accuracy.
  • Evaluation settings: Visual features substantially affect performance: VGG features outperform GoogLeNet by 5.8% for BIDILEL on AwA.This may explain strong AwA results reported with VGG-based features.
  • Results: The proposed method outperforms all baselines on CUB despite variations in experimental settings.The claim is reported across the expanded comparisons.
  • Results: On ImageNet, the expanded AlexNet comparison reports that the proposed method outperforms baseline approaches in all cases.The table evaluates all unseen classes using word-vector semantic representations.
  • Results: Additional AwA experiments show that the method can improve the quality of 1,000-dimensional word vectors.These experiments use GoogLeNet visual features and the same evaluation protocol.
  • Qualitative results: Qualitative AwA and SUN results generally align class names, attributes, and nearest images, although nonvisual attributes can produce semantically reasonable wrong-class images.The example predicts a hippopotamus exemplar from pig attributes, with an appearance still close to pigs.

E. Generalized zero-shot learning results

Generalized zero-shot learning evaluates test instances from both seen and unseen classes, while additional analyses study exemplar visualizations and the effects of labeled data from peeked unseen classes. The proposed approach outperforms baselines on AUSUC across the evaluated datasets, but the benefits of peeked data involve competing effects.

  • Generalized zero-shot learning: Generalized zero-shot learning tests instances from both seen and unseen classes using their union as the label space.The paper evaluates this setting with the Area Under Seen-Unseen accuracy Curve (AUSUC).
  • Generalized zero-shot learning: The proposed approach outperforms baselines on AUSUC for all evaluated datasets.The reported datasets are AwA, CUB, and SUN.
  • Visualizations: The t-SNE visualizations encode real images as crosses and predicted exemplars as circles, with colors identifying unseen classes.Predictions are ideal when same-color crosses and circles are well aligned.
  • Peeked unseen classes: The peeked-unseen analysis selects classes by a greedy procedure because finding the most diverse subset is NP-hard.The setup denotes the number of peeked unseen classes by B.
  • Peeked unseen classes: Increasing peeked unseen classes improves overall accuracy until the gain becomes flat, while real exemplars define an upper bound for EXEM (1NN).The remaining gap could potentially be reduced through more accurate visual-exemplar prediction than SVR provides.
  • Peeked unseen classes: As peeked classes increase, their test-instance weight rises roughly linearly while the remaining-unseen weight declines roughly linearly.This weighting contributes to upward and downward trends across the corresponding curves.
  • Peeked unseen classes: Peeked unseen data improves accuracy on peeked classes and reduces confusion among remaining classes, but also biases classifiers toward peeked classes.That bias can increase confusion between peeked and remaining classes; overall improvement occurs when benefits outweigh costs.

F.2.4 Results on additional metric, additional method, and additional rounds

Across per-image and per-class accuracy, heavy-toward-seen selection is preferred for strict metrics, while clustering is preferred for flexible metrics. EXEM variants outperform SYNCO-VS-O, and random-selection trends are stable across rounds.

  • Heavy-toward-seen selection is preferable for strict metrics with small K, whereas clustering is preferable for flexible metrics with large K across ZSL algorithms.
  • EXEM (1NNS) slightly outperforms EXEM (1NN) under matched evaluation settings, while both outperform SYNCO-VS-O.
  • Per-class accuracy vs. Per-image accuracy: Per-class accuracy is generally lower than per-image accuracy because ImageNet class frequencies differ and semantically similar unseen classes perform better.
  • Stability of peeked unseen class random selection: For random subset selection, the standard deviation across 10 rounds is small relative to gaps between methods, suggesting distribution matters more than the specific selected classes.
  • Real or predicted exemplars for peeked classes: Using real or predicted exemplars for peeked classes produces an unnoticeable difference in zero-shot performance because the nonlinear regressor has low training error.
  • At Flat Hit@1 with 1,000 heavy-toward-seen peeked classes, EXEM (1NN) per-class accuracy is 12.1%, 3.0%, and 0.8% for 2-hop, Pure 3-hop, and Rest.

G. Additional analysis on dimension for PCA

The analysis examines PCA projection dimension as a trade-off between running time and zero-shot performance. The approach remains robust across a wide range of sufficiently large dimensions.

  • The approach still outperforms all baselines on AwA and SUN with d=50, and on all three datasets with d=100.
  • The multi-layer perceptron maps semantic representations to visual exemplars using optimized weights W1 and W2 with ℓ2 regularization.
  • The implementation uses Adam with learning rate 0.0001, minibatch size S, and dataset-specific hidden-unit counts of 300 for AwA and 700 for CUB.
Loading 1605.08151v2…