Source-linked AI summary
Feature Generating Networks for Zero-Shot Learning
Yongqin Xian, Tobias Lorenz, Bernt Schiele, Zeynep Akata
TL;DR
The paper addresses generalized zero-shot learning under severe imbalance between seen and unseen classes, where labeled unseen-class examples are unavailable. It generates CNN features conditioned on class semantics using a WGAN with classification loss, and reports improved state-of-the-art performance across five datasets in both ZSL and GZSL.
Problem
Generalized zero-shot learning is hindered by prediction bias toward seen classes when training uses only labeled seen-class data.
Method
f-CLSWGAN generates CNN features conditioned on class-level semantic information using Wasserstein loss regularized by classification loss.
Results
The approach consistently improves upon the state of the art across five datasets in both zero-shot and generalized zero-shot learning.
Takeaways & Limitations
Generated unseen-class features enable training softmax classifiers for generalized zero-shot learning and can support other classifiers, CNN features, and class embeddings.
Abstract
from arXiv · showhide
Suffering from the extreme training data imbalance between seen and unseen classes, most of existing state-of-the-art approaches fail to achieve satisfactory results for the challenging generalized zero-shot learning task. To circumvent the need for labeled examples of unseen classes, we propose a novel generative adversarial network (GAN) that synthesizes CNN features conditioned on class-level semantic information, offering a shortcut directly from a semantic descriptor of a class to a class-conditional feature distribution. Our proposed approach, pairing a Wasserstein GAN with a classification loss, is able to generate sufficiently discriminative CNN features to train softmax classifiers or any multimodal embedding method. Our experimental results demonstrate a significant boost in accuracy over the state of the art on five challenging datasets -- CUB, FLO, SUN, AWA and ImageNet -- in both the zero-shot learning and generalized zero-shot learning settings.
1. Introduction
The paper targets zero-shot learning, where labeled examples for some classes are unavailable, and proposes generating CNN features for unseen classes instead of relying on insufficient synthetic images. Its f-CLSWGAN model conditions feature generation on class-level information and improves performance across ZSL and GZSL settings.
- Zero-shot learning requires classification without labeled training examples for certain classes.
- Synthetic images are not accurate enough to improve image classification, motivating direct generation of CNN features for unseen classes.
- The proposed f-CLSWGAN generates CNN features for unseen classes using an attribute-conditional adversarial network.
- The paper compares generative models for feature generation in both zero-shot and generalized zero-shot learning, where test classes include unseen and seen labels.
- Across five datasets, the approach improves upon the state of the art in both ZSL and GZSL settings.
- The framework supports different deep CNN features and class-level auxiliary information, including sentence, attribute, and word2vec embeddings.
2. Related work
The related work reviews GANs and zero-shot learning, then positions this paper as a feature-generation approach for generalized zero-shot learning. Unlike image-generation methods that are unsuitable for classifier training, the proposed method combines WGAN with classification regularization and compares favorably with GMMN.
- Generative Adversarial Networks: GAN research includes extensions for deep convolutional generation, factorized image generation, and additional latent-information objectives.
- Feature Generation: The paper reports that state-of-the-art GAN-generated images are not ready to serve as classifier training data.
- Feature Generation: The proposed architecture generates CNN features directly and combines WGAN loss with classification loss to make them discriminative.
- ZSL and GZSL: In zero-shot learning, training and test classes are disjoint, so supervised learning methods cannot directly use labeled examples from every test class.
- Feature Generation: For GZSL, the method generates unseen-class CNN features and improves over GMMN in feature generation on most datasets for both ZSL and GZSL.
3. Feature Generation & Classification in ZSL
The method generates CNN features from class embeddings and noise, then uses them with seen data to train ZSL or GZSL classifiers. f-CLSWGAN adds a classification loss to conditional Wasserstein feature generation so synthesized features are discriminative.
- Feature Generation: The generator maps Gaussian noise and a class embedding to a CNN feature, enabling synthesis for unseen classes from their semantic embeddings.After training on seen-class features, the generator can resample noise to produce arbitrarily many features for each unseen class.
- Feature Generation: Conditional WGAN training incorporates class embeddings into both generator and discriminator, while gradient penalty regularizes the discriminator along real-generated interpolations.The Wasserstein terms approximate distribution distance, and the penalty enforces unit gradient norm along straight-line segments.
- Feature Generation: f-CLSWGAN minimizes classification loss on generated features because conditional WGAN generation alone does not guarantee features suitable for discriminative classification.A linear softmax classifier pretrained on real seen-class features supplies the negative-log-likelihood regularizer, weighted by a classifier hyperparameter.
- Classification: Generated unseen-class features are combined with real seen-class data to train multimodal embedding models or softmax classifiers.Embedding methods learn compatibility between image features and class embeddings; softmax training uses synthetic data for ZSL and combines synthetic with seen data for GZSL.
4. Experiments
Experiments evaluate f-CLSWGAN across ZSL and GZSL benchmarks, generative-model variants, feature encoders, conditioning signals, and feature-generation settings. The method improves performance broadly, with especially large GZSL gains and stronger results from ResNet features and sentence conditioning.
- Experimental protocol: Experiments cover ZSL and GZSL on CUB, FLO, SUN, AWA, and ImageNet, including comparisons, robustness analyses, and image-versus-feature generation.The protocol also examines CNN architectures, semantic embeddings, training stability, and the number of generated features.
- ZSL results: 61.5% versus 54.9% on CUB, 71.2% versus 53.4% on FLO, 62.1% versus 58.1% on SUN, and 69.9% versus 65.6% on AWA in ZSL accuracy.These comparisons are against results obtained without feature generation.
- GZSL results: 49.7% versus 34.4% on CUB, 65.6% versus 21.9% on FLO, 39.4% versus 26.3% on SUN, and 59.6% versus 27.5% on AWA in GZSL harmonic mean.Feature generation improves unseen-class accuracy while maintaining seen-class accuracy, reducing the seen/unseen imbalance.
- Generative-model comparisons: f-CLSWGAN is the best-performing generative model in almost all evaluated ZSL and GZSL cases, outperforming f-GAN, f-WGAN, and f-GMMN comparisons.The classification loss helps f-CLSWGAN recover where f-WGAN performs lower than f-GMMN.
- Feature-generation analysis: Increasing generated unseen-class features from 1 to 100 raises accuracy from 28.2% to 56.5% on CUB and from 37.9% to 66.5% on FLO.The reported model ordering is f-GAN < f-WGAN < f-GMMN < f-CLSWGAN on CUB, with f-WGAN and f-GMMN reversed on FLO.
5. Conclusion
The framework generates CNN features for unseen classes and uses them for classification across architectures, classifiers, embeddings, datasets, and both ZSL and GZSL settings. It also motivates GZSL as an auxiliary testbed for evaluating generative models alongside image inspection.
- f-xGAN generates CNN features followed by classification to tackle generalized zero-shot learning.f-CLSWGAN adapts conditional GANs to feature generation and adds a classification loss so generated features are better suited for classification.
- Generating unseen-class features enables softmax classifiers to operate in the generalized zero-shot setting.The framework also supports multimodal classifiers used for zero-shot learning.
- The framework integrates with GoogleNet and ResNet features, multiple classifiers, several class embeddings, and datasets ranging from fine-grained to large-scale.Supported embeddings include sentence, attribute, and Word2Vec representations; datasets include CUB, FLO, SUN, AWA, and ImageNet.
- GZSL tasks are proposed as an auxiliary evaluation method for generative models, complementing manual inspection of generated image pixels.The paper argues that image inspection is tedious and prone to errors, and reports that model ordering in ZSL and GZSL follows the comparison between GAN and WGAN.