PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wild

Wei-Lun Chao, Soravit Changpinyo, Boqing Gong, Fei Sha

arXiv:1605.04253v2cs.CV

TL;DR

The paper addresses the unrealistic assumption that test data contain only unseen classes by studying GZSL, where seen and unseen classes share the test label space. It evaluates existing ZSL classifiers, introduces calibration and AUSUC, and finds a large gap between current methods and an idealized semantic-embedding limit.

  • Problem

    Conventional ZSL assumes test data come only from unseen classes, whereas real recognition must handle both seen and unseen classes.

  • Method

    The paper empirically evaluates ZSL approaches in GZSL, proposes calibrated stacking, and introduces AUSUC to measure the seen–unseen trade-off.

  • Results

    Existing ZSL approaches perform poorly in GZSL, and their performance remains far below an upper bound based on idealized semantic embeddings.

  • Takeaways & Limitations

    Improving class semantic embeddings is vital for achieving GZSL performance closer to the idealized limit.

  • Takeaways & Limitations

    The paper estimates the best possible ZSL performance using a multiclass task with annotated unseen-class data.

Abstract

from arXiv · show

Zero-shot learning (ZSL) methods have been studied in the unrealistic setting where test data are assumed to come from unseen classes only. In this paper, we advocate studying the problem of generalized zero-shot learning (GZSL) where the test data's class memberships are unconstrained. We show empirically that naively using the classifiers constructed by ZSL approaches does not perform well in the generalized setting. Motivated by this, we propose a simple but effective calibration method that can be used to balance two conflicting forces: recognizing data from seen classes versus those from unseen ones. We develop a performance metric to characterize such a trade-off and examine the utility of this metric in evaluating various ZSL approaches. Our analysis further shows that there is a large gap between the performance of existing approaches and an upper bound established via idealized semantic embeddings, suggesting that improving class semantic embeddings is vital to GZSL.

1 Introduction

ZSL targets unseen-class recognition by transferring information through semantic embeddings, but conventional evaluation excludes seen classes. The paper advocates GZSL, introduces calibration and AUSUC, and finds a substantial gap from ideal semantic embeddings.

  • 1 Introduction: Long-tailed and newly emerging categories make recognition with few labeled examples a practical need.Rare natural classes and newly defined concepts may have zero or little labeled data.
  • 1 Introduction: ZSL learns unseen-class models by relating them to seen classes through shared semantic spaces such as attributes or word vectors.These embeddings transfer models from seen classes to unseen ones without labeled unseen-class training data.
  • 1 Introduction: Conventional ZSL evaluates only discrimination among unseen classes, assuming seen objects are absent at test time.This setup has been widely adopted, but it omits the ability to recognize previously learned classes.
  • 1 Introduction: GZSL evaluates test data from both seen and unseen classes in their joint labeling space, reflecting real-world class frequencies.The paper argues that useful systems must discriminate within each group and between the groups.
  • 1 Introduction: Existing ZSL classifiers perform poorly under GZSL, motivating calibrated stacking and the AUSUC metric for balancing seen- and unseen-class recognition.The study evaluates representative approaches on three benchmarks, including ImageNet Fall 2011 with approximately 21,000 unseen categories.
  • 1 Introduction: A large gap between existing methods and an idealized semantic-embedding upper bound indicates that semantic embeddings remain a central improvement target.The upper bound uses class-representative visual features as idealized embeddings.

2 Related Work

Prior generalized-ZSL work was sparse, and existing approaches either evaluated only unseen data or used limited unseen-class experiments.

  • 2 Related Work: Earlier methods allowed seen classes in classifier label spaces but still tested only on unseen-class data.This retained the conventional evaluation restriction despite broader classifier outputs.
  • 2 Related Work: A two-stage prior approach first separates seen from unseen data, but its experiments cover only 2 or 6 unseen classes.The paper compares with these methods in later experiments and supplementary material.

3 Generalized Zero-Shot Learning

GZSL classifies test examples into the union of seen and unseen classes rather than unseen classes alone. Directly stacking existing classifiers reveals severe unseen-class degradation, motivating balanced evaluation and calibration.

  • 3.1 Conventional and Generalized Zero-shot Learning: GZSL uses the joint label space T = S ∪ U, allowing each test example to belong to either seen or unseen classes.Training data contain labeled examples only from seen classes S.
  • 3.1 Conventional and Generalized Zero-shot Learning: Conventional ZSL restricts test examples to unseen classes, an assumption that is unrealistic because seen classes are common in real applications.GZSL removes this restriction and evaluates recognition in the broader setting.
  • 3.2 Classifiers: ZSL classifiers score classes using semantic embeddings, including synthesized linear models, embedding similarity, or probabilistic attribute models.Seen-class scoring functions are constructed analogously from their semantic embeddings.
  • 3.2 Classifiers: GZSL evaluation reports AU→U, AS→S, AS→T, and AU→T to separate within-group recognition from classification into the joint label space.The paper avoids reporting AT→T because averaging unequal seen- and unseen-class metrics can mislead.
  • 3.3 Generalized ZSL is hard: Direct stacking selects the highest-scoring class across T and applies this rule to DAP, IAP, ConSE, and SynC classifiers.Hyperparameters are tuned with class-wise cross-validation on AwA and CUB.
  • 3.3 Generalized ZSL is hard: Unseen-class accuracy AU→T drops significantly relative to AU→U, while seen-class accuracy AS→T remains roughly comparable to AS→S.Nearly all unseen test data are misclassified as seen classes.
  • 3.3 Generalized ZSL is hard: Seen-class scoring functions dominate because they are trained without negative examples from unseen classes, producing biased GZSL predictions.This explains the systematic tendency to assign new data to the seen label space.

4 Approach for GZSL

The paper replaces naive direct stacking with calibrated stacking, which adjusts seen-class scores to balance recognition of seen and unseen classes. Varying the calibration factor traces this trade-off in the SUC, summarized by AUSUC for evaluation and model selection.

  • Calibrated stacking: Calibrated stacking modifies direct stacking to address the failure of naively combining seen- and unseen-class classifiers.The paper motivates the adjustment because unseen test data are often classified as seen classes.
  • Calibrated stacking: The rule subtracts γ from every seen-class score while leaving unseen-class scores unchanged.The indicator identifies seen classes, and γ is the calibration factor.
  • Calibrated stacking: When γ increases toward positive infinity, predictions use only unseen classes; when γ decreases toward negative infinity, they use only seen classes.Thus, calibrated stacking spans conventional ZSL and standard seen-class classification as extreme cases.
  • Seen-Unseen accuracy Curve: Varying γ produces pairs of seen-to-joint and unseen-to-joint accuracies that form the Seen-Unseen accuracy Curve.γ = 0 corresponds to direct stacking, while the two extremes represent the endpoints of the curve.
  • AUSUC: AUSUC measures the area under the SUC, with higher values indicating better balance between seen-to-joint and unseen-to-joint accuracy.The paper advocates AUSUC for evaluating methods and tuning hyperparameters because conventional unseen-only accuracy may not optimize the generalized task.
  • Alternative approaches: Novelty detection instead estimates whether a sample is unseen using Gaussian or LoOP models before selecting the corresponding classifier.Gaussian uses a mixture-model likelihood, whereas LoOP uses distances to nearest seen classes.

5 Experimental Results

The experiments evaluate GZSL on AwA, CUB, and ImageNet using mixed seen/unseen test data and AUSUC. Calibrated stacking consistently outperforms alternative approaches, while SynC is generally strongest among the compared ZSL methods, especially beyond ImageNet hit@1.

  • Setup: The study uses AwA, CUB, and full ImageNet, with attributes for AwA/CUB and word2vec embeddings for ImageNet.Visual inputs are 1,024-dimensional GoogLeNet pooling activations across datasets.
  • Setup: GZSL test sets combine seen and unseen images, holding out 20% of seen-class images for AwA and CUB.ImageNet evaluations use Flat hit@K and scenarios defined by the distance of unseen classes in the label hierarchy.
  • Results: Calibrated stacking outperforms other methods across ZSL approaches, whereas Gaussian and LoOP novelty detection perform poorly.The paper attributes this contrast partly to novelty detection modeling seen classes collectively rather than preserving inter-class differences.
  • Results: SynC dominates the compared approaches across the plotted GZSL ranges for AwA and CUB when calibrated stacking is used.The crosses identify direct-stacking results rather than calibrated-stacking curves.
  • Results: On ImageNet, SynC generally outperforms ConSE, while their Flat hit@1 performances are nearly indistinguishable.SynC dominates ConSE for top-K hit rates when K > 1, but neither method dominates at Flat hit@1.

6 Analysis on (Generalized) Zero-shot Learning

The analysis evaluates GZSL against multi-class classification as an ideal reference and tests whether visual semantic embeddings reduce the performance gap. It also examines few-shot supervision and finds that modest labeled data can substantially improve GZSL.

  • The analysis estimates an ideal performance level using multi-class classifiers trained with labeled data from both seen and unseen classes.On AwA and CUB, 80% of labeled data train classifiers and the remaining 20% evaluate them; ImageNet uses the ImageNet-2K subsample for computational efficiency.
  • GZSL with default word2vec embeddings has a large gap from multi-class performance, which dominates GZSL across the Seen-Unseen accuracy curves.The multi-class classifiers also support balanced direct stacking, whereas direct stacking results are marked separately for GZSL.
  • Visual class features used as G-attr significantly reduce the gap to multi-class classification and can nearly match it without unseen-class classifier labels.G-attr averages visual features for each class; unseen-class examples are reserved for deriving embeddings rather than training classifiers.
  • Using about 100 labeled images per unseen class, GZSL quickly approaches multi-class performance, which uses about 1,000 labeled images per class.G-attr improves over word2vec more under Flat hit@K = 1 than when K > 1.
  • On full ImageNet, G-attr from 1 image improves Flat hit@1 threefold over word2vec, while G-attr from 100 images achieves over tenfold improvement.

7 Discussion

The paper frames GZSL as the appropriate setting for unconstrained test classes and introduces calibration and AUSUC to evaluate and balance seen- and unseen-class recognition. Its empirical analysis also identifies a substantial gap from idealized semantic embeddings.

  • GZSL removes conventional ZSL’s assumption that test data belong only to unseen classes, using the union of seen and unseen labels.
  • Straightforwardly applying existing ZSL classifiers performs poorly in GZSL, with unseen test data almost always assigned to seen classes.
  • Calibrated stacking introduces a calibration factor to balance recognition of seen and unseen data, while AUSUC measures this trade-off.
  • The study finds a large gap between existing approaches and the ideal performance limit, indicating that class semantic embeddings remain a central issue.

Supplementary Material: An Empirical Study and Analysis of Generalized Zero-Shot Learning

The supplementary material documents additional analyses, including AUSUC-based hyperparameter tuning through simulated GZSL cross-validation and supporting methodological details.

  • The supplement provides additional details, results, and analyses corresponding to the main text’s methods and experiments.
  • It covers hyperparameter tuning, novelty detection, comparative ZSL experiments, and further generalized ZSL analysis.
  • AUSUC-based tuning simulates generalized ZSL during cross-validation rather than optimizing only a single seen- or unseen-class accuracy.
  • The simulated validation procedure uses five class-disjoint folds, with 80% pseudo-training and 20% pseudo-testing data in each fold.Four folds train and validate while the remaining fold supplies validation examples from its pseudo-training set; the process repeats for five rounds.
  • AUSUC is computed by varying the calibration factor, while F-score or assumed seen/unseen priors can select a single factor.

A.2 Comparison to an alternative strategy

The supplementary comparison evaluates separate seen/unseen tuning against joint AUSUC tuning and describes novelty-detection alternatives based on Gaussian and LoOP models.

  • A.2 Comparison to an alternative strategy: Separate hyperparameter sets can optimize seen-class and unseen-class performance using standard and class-wise cross-validation, respectively.
  • A.2 Comparison to an alternative strategy: AUSUC-based joint cross-validation generally produces better GZSL models than separately optimizing seen and unseen classifiers on AwA and CUB.ConSE on AwA and DAP on CUB are reported exceptions.
  • A.2 Comparison to an alternative strategy: The alternative novelty-detection approaches use Gaussian and LoOP models to distinguish seen from unseen inputs.
  • A.2 Comparison to an alternative strategy: The Gaussian approach models mapped seen examples with class-specific means and isotropic covariance matrices, assigning novelty by negative log probability.
  • A.2 Comparison to an alternative strategy: LoOP constructs a context of k nearest mapped seen examples, computes Euclidean probabilistic distances, and uses them to define a local outlier factor and novelty score.
  • A.2 Comparison to an alternative strategy: The implementation learns a two-layer visual-to-semantic mapping and tunes λ jointly, although λ does not significantly affect novelty rankings empirically.

B.3 Additional results

Additional AwA results consistently favor calibrated stacking over Gaussian and LoOP, regardless of hyperparameter cross-validation strategy.

  • Calibrated stacking outperforms Gaussian and LoOP across all zero-shot learning approaches on AwA.This holds when hyperparameters are cross-validated using either accuracies or AUSUC.
  • Gaussian tends to classify more data into unseen categories than LoOP.The Gaussian and LoOP SUC curves cross in a pattern consistent with this tendency.
  • The SUC plots compare seen-unseen accuracy trade-offs for Gaussian, LoOP, and calibrated stacking.Hyperparameters are selected using either accuracies or AUSUC.

C.1 Additional results for Section 5.3 of the main text

Additional evaluations extend GZSL comparisons across CUB, ImageNet, ESZSL, and SUN, with SynC usually strongest among compared methods.

  • Additional SUC comparisons: SynC outperforms other approaches in most additional CUB and ImageNet SUC comparisons.The comparisons cover all CUB splits and ImageNet-3hop and ImageNet-All.
  • ESZSL: ESZSL outperforms DAP, IAP, and ConSE but remains below both versions of SynC on AwA and CUB.Calibrated stacking also outperforms both novelty detection methods for ESZSL.
  • SUN evaluation: SUN experiments evaluate GZSL over 717 scene categories using 10 disjoint-class folds and report averages across 10 rounds.The setup uses 1,024-dimensional GoogLeNet features and 102 continuous- and binary-valued attributes.
  • ImageNet-2K construction: ImageNet-2K contains 1,000 sampled unseen classes preserving the original proportions of 2-hop, pure 3-hop, and remaining classes.The sample includes 74 2-hop, 303 pure 3-hop, and 623 remaining classes.

D.2 Additional results

Additional analyses show that idealized semantic embeddings substantially narrow the gap between GZSL and fully supervised multi-class classification, especially on CUB.

  • Idealized semantic embeddings: GZSL with G-attr substantially reduces the gap to multi-class classification compared with visual attributes or word2vec.The comparison covers AwA, CUB, and ImageNet-2K.
  • Idealized semantic embeddings: On CUB, GZSL with G-attr almost matches multi-class classification performance.The effect of G-attr is described as particularly large on CUB.
  • Few-shot analysis: Using additional labeled data, GZSL with G-attr quickly approaches multi-class classifier performance on AwA and CUB.Large improvements are also observed for GZSL with visual attributes, despite those attributes being human-defined and annotated.
  • ImageNet evaluation: The full ImageNet comparison reports AUSUC for word2vec and G-attr with 21,000 unseen classes, averaging few-shot results over 20 rounds.
Loading 1605.04253v2…