PaperlayerView exact revision on arXiv

Source-linked AI summary

Leveraging the Invariant Side of Generative Zero-Shot Learning

Jingjing Li, Mengmeng Jin, Ke Lu, Zhengming Ding, Lei Zhu, Zi Huang

arXiv:1904.04092v1cs.CV

TL;DR

Conventional ZSL handles unseen samples indirectly through visual-semantic mappings despite limited semantic information. LisGAN directly synthesizes unseen features with conditional GANs, soul-sample regularization, and cascade classification, outperforming prior state-of-the-art methods on five benchmarks.

  • Problem

    Conventional ZSL handles unseen samples indirectly through visual-semantic mappings, while limited semantic information restricts performance.

  • Method

    LisGAN uses conditional WGANs to synthesize unseen features from semantic-conditioned noise, regularizes them toward class soul samples, and applies a cascade classifier.

  • Results

    LisGAN outperforms previous state-of-the-art methods with remarkable advances on five popular benchmarks.

  • Takeaways & Limitations

    The approach combines generative unseen-feature synthesis with invariant soul-sample regularization and coarse-to-fine recognition for zero-shot learning.

  • Takeaways & Limitations

    The study fixes conditional WGAN and focuses its ablation analysis on soul-sample regularization and related design choices.

Abstract

from arXiv · show

Conventional zero-shot learning (ZSL) methods generally learn an embedding, e.g., visual-semantic mapping, to handle the unseen visual samples via an indirect manner. In this paper, we take the advantage of generative adversarial networks (GANs) and propose a novel method, named leveraging invariant side GAN (LisGAN), which can directly generate the unseen features from random noises which are conditioned by the semantic descriptions. Specifically, we train a conditional Wasserstein GANs in which the generator synthesizes fake unseen features from noises and the discriminator distinguishes the fake from real via a minimax game. Considering that one semantic description can correspond to various synthesized visual samples, and the semantic description, figuratively, is the soul of the generated features, we introduce soul samples as the invariant side of generative zero-shot learning in this paper. A soul sample is the meta-representation of one class. It visualizes the most semantically-meaningful aspects of each sample in the same category. We regularize that each generated sample (the varying side of generative ZSL) should be close to at least one soul sample (the invariant side) which has the same class label with it. At the zero-shot recognition stage, we propose to use two classifiers, which are deployed in a cascade way, to achieve a coarse-to-fine result. Experiments on five popular benchmarks verify that our proposed approach can outperform state-of-the-art methods with significant improvements.

1. Introduction

The paper identifies limits in indirect visual-semantic mapping for zero-shot learning and proposes LisGAN, which combines conditional generation, soul-sample regularization, and cascade recognition. Experiments on five datasets report improvements over state-of-the-art methods.

  • Conventional ZSL handles unseen samples indirectly through visual-semantic mappings, while limited semantic information restricts performance.
  • GAN-based ZSL must address diversity from limited or similar attributes and ensure generated samples remain related to real samples and semantics.
  • LisGAN uses conditional GANs with random-noise inputs and class semantic descriptions, while supervised discriminator classification preserves inter-class discrimination.
  • Soul samples represent category-level semantic characteristics, and generated samples are regularized to remain close to soul samples from the same class.
  • A cascade of two classifiers uses high-confidence unseen predictions as references to fine-tune recognition results.
  • Extensive experiments on five widely used datasets report remarkable improvements over state-of-the-art methods.

2. Related Work

Related work frames ZSL as recognition of unseen categories using seen visual data and semantic features. It introduces GANs as a direct-generation alternative, with LisGAN illustrated as a conditional WGAN using soul samples and confidence-based refinement.

  • 2.1. Zero-Shot Learning: Zero-shot learning recognizes unseen objects using visual features from seen classes and semantic features covering seen and unseen classes.
  • 2.1. Zero-Shot Learning: Generalized zero-shot recognition differs from conventional ZSL by classifying samples from both seen and unseen categories.
  • 2.1. Zero-Shot Learning: Conventional ZSL methods learn a mapping function, whereas GAN-based approaches directly generate unseen samples from attributes and convert ZSL into supervised learning.
  • 2.1. Zero-Shot Learning: LisGAN uses a conditional WGAN, multiple soul samples for each class, and high-confidence unseen samples to fine-tune final predictions.
  • 2.2. Generative Adversarial Nets: GANs train a generator and discriminator adversarially; in this paper, the generator produces sample features rather than image pixels.
  • 2.2. Generative Adversarial Nets: WGANs improve training stability and reduce mode-collapse problems, while conditional GANs incorporate additional information into generation.

3. The Proposed Method

LisGAN conditionally generates visual features from semantic descriptions and random noise, then regularizes them with class-specific soul samples. A cascade of classifiers uses generated unseen features for zero-shot prediction.

  • Overall framework: LisGAN trains a conditional GAN whose generator synthesizes visual features from semantic descriptions and random noise, while the discriminator distinguishes real from generated features.The discriminator also incorporates supervised classification and Wasserstein objectives during adversarial training.
  • Soul samples: Multiple soul samples model the multi-view nature of real images and are intended to alleviate view-induced domain shift.The formulation allows different visual views within a category to have multiple invariant representations.
  • Soul samples: Soul samples represent category characteristics and regularize each generated feature toward at least one same-class soul sample.The method defines soul samples from category representations and uses them to constrain generated features.
  • Training regularization: The two regularizations operate at individual and group levels, respectively, encouraging generated features to match real samples and generated soul samples to match real class structure.LR1 addresses individual samples, whereas LR2 addresses clusters.
  • Unseen prediction: After training on seen classes, the generator synthesizes unseen-class features from unseen semantic attributes, converting zero-shot learning into supervised classification.A softmax classifier is trained on generated features and applied to real unseen features.
  • Unseen prediction: LisGAN uses two classifiers in cascade: the first estimates confidence, and the second leverages confidently classified samples to predict other unseen samples.The second classifier may be a softmax, SVM, or training-free NNC.

4.1. Datasets

The evaluation uses five datasets spanning objects, birds, flowers, and scenes, with established zero-shot splits and CNN features paired with semantic descriptions.

  • Datasets: aPaY contains 15,339 images across 32 categories, using 20 PASCAL classes as seen and 12 Yahoo classes as unseen.Each category has a 64-dimensional attribute vector.
  • Datasets: AwA contains 30,475 images from 50 animal classes, each associated with 85 numeric attributes.The classes align with Osherson’s class/attribute matrix.
  • Datasets: CUB contains 11,788 images from 200 bird species, with Wikipedia articles and 312 binary attributes organized into 28 attribute groupings.The dataset extends CUB-200 and includes scientific-classification organization.
  • Datasets: FLO contains 8,189 images from 102 flower categories with substantial scale, pose, lighting, within-class, and between-class variation.The study uses the semantic descriptions provided by Reed et al.
  • Evaluation setup: The study follows prior zero-shot splits, extracts 2048-dimensional ResNet-101 CNN features, and uses dataset attributes or 1024-dimensional RNN descriptions for FLO.Table 1 reports dataset statistics and split settings.

4.2. Implementation and Compared Methods

The experiments use a conditional WGAN-based implementation with specified architectural and training settings, compare against representative ZSL methods, and evaluate top-1 accuracy under conventional and generalized protocols.

  • Implementation: The GAN uses an MLP generator with a 4,096-unit hidden layer and a discriminator with adversarial and n-way classification branches.The generator conditions noise on semantic descriptions; the discriminator distinguishes real from fake features and predicts sample categories.
  • Compared Methods: The compared methods include representative embedding, semantic, generative, and state-of-the-art approaches such as DAP, ALE, GAZSL, and f-CLSWGAN.The full comparison includes DAP, CONSE, SSE, DeViSE, SJE, ESZSL, ALE, SYNC, SAE, DEM, GAZSL, and f-CLSWGAN.
  • Evaluation Protocol: Conventional ZSL reports average per-class top-1 accuracy on unseen classes, whereas GZSL evaluates seen and unseen classes together.For GZSL, possible labels are searched over the combined seen and unseen label space.

4.3. Zero-shot Learning

LisGAN achieves strong conventional zero-shot performance across five datasets, outperforming existing methods on four evaluations and improving over prior state of the art on the remaining dataset.

  • Results: 2.6% improvement over the state-of-the-art method is reported on aPaY, with additional gains of 2.4%, 1.5%, and 2.4% on AWA, CUB, and FLO.The method achieves the best result on four of the five evaluations and a state-of-the-art result on the last dataset.
  • Results: GAN-based methods generally outperform embedding methods in the reported zero-shot learning comparisons.The paper attributes this pattern to directly converting ZSL into supervised learning through generated visual features rather than using an indirect embedding route.

4.4. Generalized Zero-shot Learning

In generalized zero-shot learning, LisGAN reports stronger performance on unseen samples and harmonic mean than existing methods, with gains across all five datasets and explanations tied to invariant-side regularization and cascade classification.

  • Evaluation: Generalized zero-shot learning evaluates both seen and unseen samples by searching categories from the union of seen and unseen labels.The experiments use separate seen-class training and test splits.
  • Results: LisGAN performs better than existing methods, with significantly stronger results on unseen samples and harmonic mean in generalized zero-shot learning.The authors describe the method as dependable across both seen and unseen classes.
  • Evaluation: The reported harmonic mean is used to assess balance between seen and unseen performance and is described as more stable regarding outliers than arithmetic or geometric means.The mean is defined as (2*Unseen*Seen)/(Unseen+Seen).
  • Results: Up to 2.8%, 2.7%, 1.9%, 2.7%, and 0.8% harmonic-mean improvements are reported on aPaY, AwA, CUB, FLO, and SUN, respectively.The average improvement over the five datasets is 2.2%.
  • Analysis: The authors attribute performance gains to soul-sample regularization and a cascade classifier that supports coarse-to-fine recognition.Soul samples relate generated features to real samples and semantic descriptions, while the cascade uses the first classifier’s results to strengthen the second.

4.5. Model Analysis

Model analysis examines class-wise accuracy, hyperparameter sensitivity, training stability, and component ablations, showing category-specific gains, stable training, and contributions from the main design components.

  • 4.5.1 Class-wise Accuracy: Accuracy on aPaY categories “tvmonitor”, “donkey” and “jetski” is boosted around 10% against f-CLSWGAN.The confusion-matrix comparison reports generally better accuracy on most categories, while some unseen categories remain difficult.
  • 4.5.2 Parameter Sensitivity: Parameter analysis varies the supervised-loss weight, soul-sample regularization weight, sample entropy threshold, and synthesized samples per class.The paper fixes β = 10 for the Lipschitz constraint and sets the sample entropy threshold below the median of all entropies.
  • 4.5.3 Model Stability: The model reaches stable results with 30 epochs despite small fluctuations as training epochs increase.The analysis reports zero-shot and generalized zero-shot results in terms of testing error.
  • 4.5.4 Ablation Analysis: The model uses a conditional WGAN as a fixed basis while focusing analysis on soul-sample regularization and the cascade classifier.The authors treat conditional WGAN as an established infrastructure and analyze the added components through ablation.
  • 4.5.4 Ablation Analysis: The ablation settings indicate that supervised loss, soul-sample regularization, multiple soul samples, and cascade classification each contribute to the framework.The supervised loss supports discriminative generated features, multiple soul samples address multi-view domain shift, and the cascade improves accuracy through sample entropy.

5. Conclusion

LisGAN uses conditional WGANs and soul-sample regularization to synthesize unseen samples, then applies a cascade classifier for recognition. Experiments on five benchmarks report improvements over previous state-of-the-art methods.

  • LisGAN synthesizes fake unseen samples from random noises conditioned on class semantic descriptions using a conditional WGAN.
  • Soul-sample regularization keeps each generated sample close to real samples and corresponding semantic descriptions.
  • A cascade classifier fine-tunes recognition accuracy at the zero-shot recognition stage.
  • Extensive experiments on five popular benchmarks report remarkable advances over previous state-of-the-art methods.
Loading 1904.04092v1…