Source-linked AI summary
Generating Visual Representations for Zero-Shot Classification
Maxime Bucher, Stéphane Herbin, Frédéric Jurie
TL;DR
Zero-shot classification must classify categories described semantically when exemplar images are unavailable, while prior embedding approaches limit discriminative classification and generalized evaluation. The paper learns a conditional generator from seen classes to create unseen-class training features, enabling standard discriminative classifiers. Experiments with four generative models and five datasets validate the approach with state-of-the-art results on ZSC and GZSC.
Problem
Zero-shot classification lacks exemplar images for some categories, preventing direct training of discriminative classifiers and causing bias toward seen classes in generalized zero-shot classification.
Method
The method learns a conditional generator from seen-class examples and semantic descriptions, generates artificial training features for unseen classes, and trains discriminative classifiers.
Results
The approach achieves state-of-the-art performance on ZSC and GZSC across four standard datasets, with experiments using four generative models and five datasets.
Takeaways & Limitations
Generating unseen-class training examples turns zero-shot classification into a standard supervised learning problem and enables classifier learning in visual feature space.
Takeaways & Limitations
The paper addresses the inductive setting and does not address transductive zero-shot classification, where unseen target data are available without labels.
Abstract
from arXiv · showhide
This paper addresses the task of learning an image clas-sifier when some categories are defined by semantic descriptions only (e.g. visual attributes) while the others are defined by exemplar images as well. This task is often referred to as the Zero-Shot classification task (ZSC). Most of the previous methods rely on learning a common embedding space allowing to compare visual features of unknown categories with semantic descriptions. This paper argues that these approaches are limited as i) efficient discrimi-native classifiers can't be used ii) classification tasks with seen and unseen categories (Generalized Zero-Shot Classification or GZSC) can't be addressed efficiently. In contrast , this paper suggests to address ZSC and GZSC by i) learning a conditional generator using seen classes ii) generate artificial training examples for the categories without exemplars. ZSC is then turned into a standard supervised learning problem. Experiments with 4 generative models and 5 datasets experimentally validate the approach, giving state-of-the-art results on both ZSC and GZSC.
1. Introduction and related works
Zero-shot classification lacks exemplar images for some categories, limiting direct use of discriminative classifiers and creating bias when seen and unseen classes are jointly tested. The paper instead generates feature representations for unseen classes from semantic descriptions, then trains a discriminative classifier.
- Problem: Zero-shot classification uses semantic descriptions to classify categories without training examples, alongside seen categories with available exemplars.Attributes provide semantic information for relating unseen categories to seen classes.
- Limitations of prior work: Embedding-based approaches select the class with the highest compatibility score, but do not use unseen-category semantics during compatibility learning.Their discriminative decision is concentrated in the final arg max selection.
- Limitations of prior work: Discriminative classifiers cannot be learned directly for zero-shot categories because no images are available for them.This motivates generating training data before classifier learning.
- Proposed approach: The proposed approach learns a feature generator from seen classes, generates artificial features for unseen classes, and trains a standard discriminative classifier on the resulting data.The generator maps attributes or text descriptions to image features; the classifier is learned from artificially generated training examples.
- Generalized zero-shot classification: The method addresses generalized zero-shot classification by using artificial training examples for both seen and unseen classes, avoiding bias toward seen classes.Experiments report state-of-the-art performance on both ZSC and GZSC across four standard datasets, with scaling insights for ImageNet.
- Proposed approach: Inference in visual feature space is motivated by the claim that visual features are more readily separable, especially with discriminative deep features.This contrasts with inference in abstract attribute or embedding spaces.
2. Approach
The approach learns a classifier when data exist only for seen classes by generating unseen-class examples from semantic representations, then combining them with seen data for supervised learning. It uses conditional generators, including four studied model strategies, to support both ZSC and the homogeneous treatment of seen and unseen classes.
- Problem: The paper targets inductive ZSC, where empirical data are available only for seen classes and unseen classes are specified through semantic representations.
- Scope: The approach is restricted to the inductive setting and does not address transductive ZSC because target data availability is considered a practical constraint.
- Core approach: The method artificially generates training data for unseen classes from seen classes and their semantic representations, then trains a discriminative class predictor.
- Core approach: Generated unseen examples can be combined with seen data as one homogeneous supervised problem, enabling Generalized Zero-Shot Classification.
- Conditional generation: The generator learns a conditional random process that maps a semantic description and random prior sample z to generated data bx = G(a, z; w).
- Generator strategies: Four generator strategies are studied: GMMN, conditional adversarial models, denoising auto-encoders, and adversarial auto-encoders.
- Generator strategies: GMMN matches real and generated conditional distributions with differentiable kernel statistics, whereas AC-GAN uses a learned discriminative discrepancy measure.
- Generator strategies: For the autoencoder generators, semantic representations condition the decoder, which produces novel examples from the semantic description and latent code.
3. Experiments
The experiments compare four generative models, evaluate the selected approach on generalized, large-scale, and classical zero-shot tasks, and report state-of-the-art performance across these settings.
- Comparing the generative models: The study compares four generative models on regular Zero-Shot classification and uses validation performance to select a model for subsequent experiments.The validation results are reported as mean accuracy, and the comparison is summarized in Table 1.
- Comparing the generative models: GMMN outperforms the other three models on average, with a 5% improvement on aP&Y, and is more computationally stable than the adversarial alternatives.The authors consequently choose GMMN for the remaining experiments.
- Generalized Zero-Shot Classification: In Generalized Zero-Shot Classification, the protocol evaluates seen-only, unseen-only, and combined seen-and-unseen class predictions, with the combined setting using generated features for all classes.The combined setting is harder because test images may belong to any class and the candidate set is larger.
- Generalized Zero-Shot Classification: In the hardest u →a setting, the generative approach achieves accuracy 30% higher than state-of-the-art methods on AwA and 10% higher on CUB.The authors attribute this to learning a Softmax classifier that discriminates both seen and unseen classes, avoiding the scoring-function bias toward seen classes.
- Large-Scale Zero-Shot Classification: On ImageNet, the model achieves state-of-the-art performance for each reported configuration and improves the best competitors by 5% on Flat-Hit 1 in the 2-hop scenario.The experiment uses 1,000 seen classes and 20,345 unseen classes represented with skip-gram embeddings.
4. Conclusions
The paper reformulates ZSC and GZSC by generating artificial examples for exemplar-free categories, converting the tasks into standard supervised classification. Experiments validate this approach across four generative models and five datasets.
- A conditional generator learned from seen data produces artificial training examples for categories without exemplars.
- The formulation addresses limitations involving discriminative classifiers and bias in Generalized Zero-Shot Classification.
- Experiments with 4 generative models and 5 datasets give state-of-the-art performance.