Source-linked AI summary
Zero-Shot Learning -- The Good, the Bad and the Ugly
Yongqin Xian, Bernt Schiele, Zeynep Akata
TL;DR
Zero-shot learning lacks an agreed benchmark, making published progress difficult to compare and allowing protocol violations such as pre-training on test classes. The paper unifies evaluation protocols and data splits, then analyzes state-of-the-art methods in zero-shot and generalized zero-shot settings. It finds that compatibility-learning frameworks generally have an edge and that generalized evaluation must account for performance on training classes.
Problem
Zero-shot learning lacks a unified benchmark, while published evaluations can be incomparable or violate the zero-shot assumption through test-class pre-training or tuning.
Method
The paper evaluates ten methods across five datasets and multiple splits using unified protocols, per-class accuracy, disjoint validation classes, and both zero-shot settings.
Results
Compatibility-learning frameworks have an edge over independent classifiers and hybrid models, while generalized evaluation shows that training-class accuracy matters alongside test-class accuracy.
Takeaways & Limitations
Zero-shot benchmarks should use carefully chosen splits, disjoint validation classes, and generalized evaluation that includes training classes in the search space.
Takeaways & Limitations
Direct Attribute Prediction is included primarily because it is historically widespread, despite performing poorly relative to compatibility-learning frameworks.
Abstract
from arXiv · showhide
Due to the importance of zero-shot learning, the number of proposed approaches has increased steadily recently. We argue that it is time to take a step back and to analyze the status quo of the area. The purpose of this paper is three-fold. First, given the fact that there is no agreed upon zero-shot learning benchmark, we first define a new benchmark by unifying both the evaluation protocols and data splits. This is an important contribution as published results are often not comparable and sometimes even flawed due to, e.g. pre-training on zero-shot test classes. Second, we compare and analyze a significant number of the state-of-the-art methods in depth, both in the classic zero-shot setting but also in the more realistic generalized zero-shot setting. Finally, we discuss limitations of the current status of the area which can be taken as a basis for advancing it.
1. Introduction
The paper addresses rapid growth in zero-shot learning by benchmarking methods, datasets, and evaluation protocols under unified, realistic settings. It emphasizes rare-class evaluation and protocol choices that preserve the zero-shot assumption.
- Zero-shot learning recognizes objects whose instances were absent from training, but progress is difficult to quantify without an established evaluation protocol.
- The benchmark evaluates methods across linear, nonlinear, independent-classifier, and hybrid approaches that connect observed and unobserved classes through auxiliary information.
- Evaluation uses multiple splits and datasets because fine-grained rare-object classes require expert-labeled data and should receive primary attention.
- The unified protocol requires disjoint validation classes, favors average per-class top-1 accuracy for imbalanced data, excludes test classes from feature pre-training, and includes generalized zero-shot learning.
2. Related Work
Related work spans zero-shot and generalized zero-shot learning methods and prior evaluations. This paper extends those efforts with a broader benchmark covering methods, datasets, splits, and robustness.
- Zero-shot learning methods use intermediate attribute classifiers, mixtures of seen-class proportions, or direct compatibility-learning frameworks.
- Generalized zero-shot learning evaluates both seen and unseen classes at test time, addressing the need to reject unknown objects while recognizing known ones.
- Prior evaluations compared fewer methods, datasets, or splits, whereas this work evaluates ten methods on five datasets across zero-shot and generalized settings.
- The benchmark adds statistical significance and robustness tests, making the evaluation more extensive than prior work.
3. Evaluated Methods
The paper evaluates zero-shot methods across compatibility learning, attribute classifiers, and hybrid models, covering both standard and generalized settings. These methods differ in how they map images and class information or combine predictions from seen classes.
- Zero-shot learning assigns test images to unseen classes, while generalized zero-shot learning permits assignment to both seen and unseen classes.
- Learning Linear Compatibility: Linear compatibility methods learn a mapping between given image and class embeddings, with ESZSL adding regularization to the risk objective.
- Learning Linear Compatibility: DEVISE, ALE, and SJE use ranking-based objectives that emphasize compatibility with the correct class, with ALE weighting higher-ranked items more strongly.
- Learning Nonlinear Compatibility: Nonlinear compatibility methods add multiple mappings or neural nonlinearities, with LATEM selecting among matrices for different visual characteristics and CMT using novelty detection.
- Learning Intermediate Attribute Classifiers: DAP learns probabilistic attribute classifiers and combines their scores for class prediction, whereas hybrid methods represent images and semantic classes through mixtures or aligned model spaces.
- Hybrid Models: CONSE combines semantic embeddings using predicted training-class probabilities, while SYNC aligns real and phantom classes through a weighted bipartite graph minimizing distortion.
4. Datasets and Evaluation Protocol
The paper defines a unified zero-shot learning benchmark spanning datasets, embeddings, splits, and evaluation criteria. It removes ImageNet overlap from proposed test splits, emphasizes rare-class performance, and evaluates generalized zero-shot learning with balanced seen/unseen accuracy.
- Benchmark design: The benchmark covers datasets, image and class embeddings, dataset splits, and evaluation criteria for zero-shot and generalized zero-shot learning.The selected datasets include coarse- and fine-grained attribute datasets plus large-scale ImageNet.
- Dataset splits: The proposed splits exclude test classes appearing among ImageNet 1K classes used to pre-train ResNet, avoiding inflated accuracy from feature pre-training overlap.Standard splits contain overlapping classes, including 1 of 50 CUB test classes and 6 of 72 SUN test classes.
- Dataset splits: The proposed splits also include training-class images at test time to evaluate generalization across both seen and unseen classes.ImageNet splits additionally vary by hierarchical distance and class population, including 2-hop, 3-hop, and population-based subsets.
- Evaluation criteria: Average per-class top-1 accuracy gives equal class-level weighting, preventing densely populated classes from dominating evaluation.The metric averages correct predictions independently for each class before averaging across classes.
- Evaluation criteria: In generalized zero-shot learning, harmonic mean combines seen- and unseen-class accuracies because the goal is high performance on both groups.The evaluation search space includes training and test classes, making this setting more practical.
5. Experiments
The experiments compare zero-shot methods across reproduced, unified, and proposed splits, then assess robustness, ImageNet scale, and generalized zero-shot performance. Results show that split design, class population, and whether seen classes are included materially affect rankings and accuracy.
- Zero-Shot Learning Results: The study reproduces prior results, evaluates common and proposed train/validation/test splits, and analyzes method robustness and rankings across multiple datasets.Experiments cover SUN, CUB, AWA, aPY, and ImageNet, followed by generalized zero-shot evaluation.
- Reproducing Results: 74.5% versus 54.5% for SSE and 64.3% versus 57.3% for ESZSL reveals that using 10 rather than 72 SUN test classes changes reported accuracy substantially.The comparison identifies an irregularity in published evaluations relative to the standard SUN split.
- Reproducing Results: 62.8% versus 59.1% for SYNC on SUN shows that features from a pre-trained model whose data intersect test classes can inflate results relative to ImageNet-pretrained features.The paper therefore proposes splits whose test classes do not belong to ImageNet1K used for ResNet pre-training.
- Standard and Proposed Splits: On the proposed split, ALE, DEVISE, and SJE rank highest, while DAP, CMT, and CONSE rank lowest, indicating stronger performance from max-margin compatibility methods in this evaluation.The best method changes across datasets and splits: for example, ALE leads SUN and aPY, SYNC leads CUB, and SJE leads AWA on PS.
- ImageNet Results: ImageNet accuracy is higher for most-populated than least-populated classes, drops sharply from 1K to 5K classes, and remains poor across all 20K categories.The paper reports approximately 6.6M images for the 5K subset versus approximately 1,624 for the 1K subset.
- Generalized Zero-Shot Learning Results: Generalized zero-shot results are significantly lower because training classes enter the search space, creating a trade-off between test-class and training-class performance.Compatibility methods perform well on test classes, whereas independent attribute or object classifiers perform well on training classes; harmonic mean ranks ALE best on SUN, CUB, and AWA, and CMT* best on aPY.
6. Conclusion
The evaluation finds compatibility learning superior to independent classifiers and hybrid models, while emphasizing sanitized splits and disjoint validation for zero-shot evaluation.
- Compatibility learning frameworks outperform independent object or attribute classifiers and hybrid models in the evaluation.
- Standard zero-shot splits can overlap with feature-learning classes, motivating new splits that preserve class disjointness.
- Disjoint training and validation class splits are necessary when tuning parameters for zero-shot learning.
- Generalized zero-shot learning includes training classes in the evaluation search space and remains an area for future research.