Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly

Yongqin Xian; Christoph H. Lampert; Bernt Schiele; Zeynep Akata

Source-linked AI summary

Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly

Yongqin Xian, Christoph H. Lampert, Bernt Schiele, Zeynep Akata

arXiv:1707.00600v4Jul 3, 2017cs.CV

TL;DR

Zero-shot learning lacks a reliable, shared benchmark, and published evaluations can violate its disjoint-class assumption. The paper unifies protocols and splits, introduces AWA2, and evaluates many methods in classic and generalized settings, finding distinct method patterns and substantially lower generalized performance.

Problem
Zero-shot learning lacks an agreed benchmark, making published progress difficult to compare and allowing evaluations that violate the zero-shot assumption.
Method
The paper unifies evaluation protocols and dataset splits, evaluates many methods across datasets and settings, and introduces the publicly available AWA2 dataset.
Results
Generative and compatibility-learning methods have an edge in classic zero-shot learning, while generalized zero-shot accuracy is significantly lower than zero-shot accuracy.
Takeaways & Limitations
Generalized zero-shot learning and harmonic mean evaluation provide a setting and measure that consider performance on both seen and unseen classes.
Takeaways & Limitations
Some standard dataset splits include test classes in ImageNet features used for pre-training, violating disjoint training and test classes.

Abstract

from arXiv · show

Due to the importance of zero-shot learning, i.e. classifying images where there is a lack of labeled training data, the number of proposed approaches has recently increased steadily. We argue that it is time to take a step back and to analyze the status quo of the area. The purpose of this paper is three-fold. First, given the fact that there is no agreed upon zero-shot learning benchmark, we first define a new benchmark by unifying both the evaluation protocols and data splits of publicly available datasets used for this task. This is an important contribution as published results are often not comparable and sometimes even flawed due to, e.g. pre-training on zero-shot test classes. Moreover, we propose a new zero-shot learning dataset, the Animals with Attributes 2 (AWA2) dataset which we make publicly available both in terms of image features and the images themselves. Second, we compare and analyze a significant number of the state-of-the-art methods in depth, both in the classic zero-shot setting but also in the more realistic generalized zero-shot setting. Finally, we discuss in detail the limitations of the current status of the area which can be taken as a basis for advancing it.

1 INTRODUCTION

The paper argues that zero-shot learning progress is difficult to assess because methods lack a shared evaluation protocol and published evaluations can violate zero-shot assumptions. It therefore benchmarks methods systematically while emphasizing disjoint validation, fair metrics, and generalized evaluation.

Zero-shot learning recognizes objects whose instances were absent from training, but progress is difficult to quantify without an established evaluation protocol.
The benchmark evaluates methods across methods, datasets, and evaluation protocols, focusing on how auxiliary information connects seen and unseen classes.
A unified protocol requires hyperparameter tuning on validation classes disjoint from training classes, since tuning on test classes violates the zero-shot assumption.
The paper recommends per-class averaged top-1 accuracy for imbalanced datasets and evaluation on generalized zero-shot learning, where seen classes join the search space.
Pre-training feature extractors on datasets containing zero-shot test classes violates the zero-shot idea because feature extraction is part of training.

2 RELATED WORK

Prior zero-shot learning methods transfer information from seen to unseen classes through attributes, embeddings, compatibility functions, generative models, or transductive use of unlabeled information. The paper situates its evaluation across these methodological families and related concerns about restrictive evaluation settings.

Early attribute-based methods predict image attributes and infer labels by matching those predictions to class attributes, but intermediate attribute prediction can introduce domain shift.
Compatibility methods learn mappings between image features and semantic class representations, using linear or nonlinear functions and ranking-based objectives.
Other approaches embed visual and semantic features into a shared intermediate space or construct classifiers through mixtures of seen classes or phantom classes.
Generative methods model class-conditional distributions and learn to derive unseen-class distributions from semantic representations.
Transductive approaches use visual or semantic information from unseen classes without labels, while generalized zero-shot learning permits both seen and unseen classes at test time.
The paper differs from earlier evaluations by comparing more methods across datasets and evaluation settings within a broader unified analysis.

3 EVALUATED METHODS

The evaluated methods span compatibility learning, nonlinear embeddings, attribute-classifier pipelines, hybrid models, and generative approaches. They differ in how they map images and semantic information, optimize compatibility, or synthesize class representations.

Compatibility learning: Compatibility learning predicts the class with maximum compatibility between image and class embeddings, using bilinear functions and ranking or structured-SVM objectives.
Compatibility learning: ALE weights higher-ranked classes more strongly through α_i = 1/i, whereas SJE emphasizes the top-ranked class and requires scoring all classifiers during prediction.
Linear embedding models: ESZSL and SAE explicitly regularize linear embedding models, with ESZSL admitting a convex closed-form solution and SAE additionally reconstructing image embeddings.
Nonlinear compatibility: LATEM learns piecewise-linear mappings whose selected matrix represents a latent visual characteristic, while CMT uses a two-layer neural network to map images into a semantic word space.
Attribute classifiers: DAP predicts attributes with probabilistic classifiers before combining attribute scores into class predictions, while IAP first predicts seen-class probabilities and derives attribute probabilities indirectly.
Hybrid and generative models: SSE, CONSE, and SYNC form hybrid models using mixtures or graph-based combinations of seen-class information, whereas GFZSL models each class conditionally as a Gaussian distribution.

4 DATASETS

The benchmark covers coarse- and fine-grained attribute datasets plus large-scale ImageNet, spanning different dataset sizes, class counts, granularity, and population imbalance. AWA2 provides a publicly redistributable counterpart to AWA1 with more images.

The benchmark selects aPY, AWA1, SUN, CUB, and ImageNet to cover coarse- and fine-grained datasets from small and medium scales to large scale.
Attribute datasets: aPY contains 32 classes and 64 attributes, while AWA1 contains 50 classes and 30,475 images.
Attribute datasets: AWA2 collects 37,322 freely redistributable images for the same 50 animal classes as AWA1, using the same 85 binary and continuous attributes.
Attribute datasets: AWA2 contains more images than AWA1, and its image-feature distribution follows a similar pattern with slightly more visible clusters for some classes.
ImageNet: ImageNet contains 14 million images from 21K hierarchically related classes and exhibits substantial variation in granularity and class population.
ImageNet: The ImageNet evaluation uses 1K training classes, including 200 validation classes, with test splits defined over the remaining classes.

5 EVALUATION PROTOCOL

The paper defines evaluation protocols that align dataset encodings, splits, and criteria while preventing test-class leakage. It evaluates both standard and generalized zero-shot learning across varied datasets and class distributions.

Protocol components: The evaluation protocol specifies image and class encodings, dataset splits, and criteria for zero-shot and generalized zero-shot learning.The benchmark uses image embeddings and class embeddings across several datasets, with generalized evaluation including both training and test classes.
Dataset splits: Test classes must be absent from the data used to train image-feature extractors, so the paper replaces overlapping standard splits with proposed splits.Standard splits contain test classes among ImageNet’s 1K pre-training classes; proposed splits remove those overlaps while retaining the class counts.
Dataset splits: The benchmark uses disjoint validation classes for hyperparameter search and constructs multiple dataset splits with fixed test classes and separate training and validation classes.For SUN, CUB, AWA1, aPY, and AWA2, the proposed protocol specifies distinct training and validation class counts while keeping test classes fixed.
Dataset splits: ImageNet evaluation includes hierarchical, most-populated, least-populated, and nearly complete-data splits to test generalization across semantic similarity and class-frequency distributions.The least-populated subsets contain approximately 1, 3, and 51 images per class, while the final split covers approximately 20K remaining classes.
Evaluation criteria: Average per-class top-1 accuracy gives each class equal weight, and generalized zero-shot evaluation uses the harmonic mean of seen- and unseen-class accuracies.The harmonic mean is chosen to emphasize high accuracy on both seen and unseen classes rather than allowing much higher seen-class accuracy to dominate an arithmetic mean.

6 EXPERIMENTS

Experiments show that evaluation choices materially affect zero-shot results, while the proposed splits, AWA2 dataset, robustness analyses, and generalized setting expose where methods are reliable or fragile.

6.1 Zero-Shot Learning Experiments: Published results can diverge because implementations, features, hyperparameters, and test splits differ across methods and datasets.Reproductions were nearly identical for several methods but varied for methods affected by non-convex optimization, random sampling, or validation choices.
6.1 Zero-Shot Learning Experiments: The proposed splits reduce AWA1 and AWA2 results substantially because their standard test classes overlap with ImageNet1K pre-training classes.CUB and SUN are less affected because their class overlap with ImageNet1K is smaller.
6.1 Zero-Shot Learning Experiments: Robustness across validation splits is higher on SUN and CUB than on the other datasets, consistent with their balanced class image counts and fine-grained structure.The study varies hyperparameter-validation splits while keeping the test split fixed.
6.1 Zero-Shot Learning Experiments: GFZSL ranks highest across both standard and proposed splits, while SYNC falls from second to seventh and ALE rises from second to first.The authors identify GFZSL and ALE as the most robust methods for attribute datasets.
6.1 Zero-Shot Learning Experiments: AWA1 and AWA2 generally produce similar results, but SJE changes from 65.6% on AWA1 to 61.9% on AWA2 because it selects different hyperparameters.This comparison indicates sensitivity to parameter setting rather than a dataset artifact.
6.1 Zero-Shot Learning Experiments: AWA2 is supported as an AWA1 replacement because cross-dataset results indicate the datasets are sufficiently similar, while training-set choice strongly affects outcomes.The paired t-test reports pvalue= 0.007 for significantly different results when the training set changes.
6.1 Zero-Shot Learning Experiments: ImageNet experiments show that fine-grained subsets with fewer images are harder, and the test-set class nature matters more than the number of classes.The study evaluates ten methods at large scale and compares performance across differently populated class subsets.
6.2 Generalized Zero-Shot Learning Results: Generalized zero-shot learning lowers results because seen classes become distractors, and no method dominates across all ImageNet splits and accuracy measures.On standard datasets, ALE leads the harmonic mean on SUN, CUB, and AWA1, while DEVISE leads AWA2 and CMT* leads aPY.

7 CONCLUSION

The paper presents a unified evaluation of zero-shot learning methods, datasets, and protocols, identifies evaluation flaws, and introduces AWA2 and proposed dataset splits. It finds systematic performance patterns across classic and generalized zero-shot learning while motivating balanced evaluation of seen and unseen classes.

Evaluation scope: The study evaluates 13 state-of-the-art methods across six datasets under a unified protocol covering classic and generalized zero-shot learning.The evaluation spans SUN, CUB, AWA1, AWA2, aPY, and ImageNet.
Main findings: Generative models and compatibility learning frameworks outperform independent object or attribute classifiers and other hybrid models in classic zero-shot learning.The reported advantage concerns the classic zero-shot setting.
Evaluation protocol: Proposed splits remove ImageNet1K overlap with test classes, while disjoint training and validation class splits are required for valid parameter tuning.The paper identifies standard splits and tuning procedures that can violate zero-shot assumptions.
Dataset contribution: AWA2 contains 50 classes, inherited attributes from AWA1, and 37,322 different images; evaluated methods perform similarly on AWA2 and AWA1, whose statistical consistency test indicates compatibility.AWA2 is introduced as a publicly redistributable dataset alternative to AWA1.
Generalized zero-shot learning: Generalized zero-shot learning yields significantly lower accuracy than classic zero-shot learning, but relative model comparisons remain similar across the two settings.The generalized setting includes training classes in the search space.
Generalized zero-shot learning: The harmonic mean of seen- and unseen-class accuracy is proposed as a unified GZSL measure because it encourages performance on both types of samples.This measure is intended to reflect a setting closer to real-world recognition.