PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent Embedding Feedback and Discriminative Features for Zero-Shot Classification

Sanath Narayan, Akshita Gupta, Fahad Shahbaz Khan, Cees G. M. Snoek, Ling Shao

arXiv:2003.07833v2cs.CV

TL;DR

Zero-shot methods synthesize unseen-class features from semantic embeddings, but semantic-consistency constraints are not retained across feature synthesis and classification. This paper applies a semantic embedding decoder throughout the pipeline, using iterative feedback and discriminative feature transformation, and reports improvements over existing methods on six benchmarks.

  • Problem

    Existing methods enforce semantic consistency during training but discard that constraint during feature synthesis and classification.

  • Method

    A VAE-GAN framework uses a semantic embedding decoder at all stages, with feedback for iterative feature refinement and discriminative transformation for classification.

  • Results

    The approach outperforms existing methods on all six object and action recognition datasets and achieves absolute GZSL object-recognition gains of 4.6%, 7.1%, 1.7%, and 3.1% on CUB, FLO, SUN, and AWA.

  • Takeaways & Limitations

    Semantic embedding information remains useful beyond training: feedback supports feature synthesis, while decoder embeddings and visual features are used to reduce category ambiguities during classification.

  • Takeaways & Limitations

    Prior GAN-based approaches can encounter mode collapse that decreases generated-feature diversity, while some semantic-consistency modules are used only during training.

Abstract

from arXiv · show

Zero-shot learning strives to classify unseen categories for which no data is available during training. In the generalized variant, the test samples can further belong to seen or unseen categories. The state-of-the-art relies on Generative Adversarial Networks that synthesize unseen class features by leveraging class-specific semantic embeddings. During training, they generate semantically consistent features, but discard this constraint during feature synthesis and classification. We propose to enforce semantic consistency at all stages of (generalized) zero-shot learning: training, feature synthesis and classification. We first introduce a feedback loop, from a semantic embedding decoder, that iteratively refines the generated features during both the training and feature synthesis stages. The synthesized features together with their corresponding latent embeddings from the decoder are then transformed into discriminative features and utilized during classification to reduce ambiguities among categories. Experiments on (generalized) zero-shot object and action classification reveal the benefit of semantic consistency and iterative feedback, outperforming existing methods on six zero-shot learning benchmarks. Source code at https://github.com/akshitac8/tfvaegan.

1 Introduction

The paper addresses zero-shot and generalized zero-shot recognition by extending semantic consistency beyond training into feature synthesis and classification. Its approach uses feedback and discriminative feature transformation, with experiments spanning six object and action benchmarks.

  • Zero-shot learning classifies images or videos into unseen categories without corresponding visual examples during training, while generalized zero-shot learning also includes seen categories at test time.
  • Existing GAN-based methods synthesize unseen-class features from class-specific semantic embeddings, but auxiliary semantic-consistency modules are typically discarded after training.
  • The method uses a semantic embedding decoder throughout training, feature synthesis, and classification within a VAE-GAN framework.
  • A feedback module transforms decoder embeddings to modulate generator representations during training and feature synthesis, while discriminative transformation uses decoder embeddings with visual features during classification.
  • The approach outperforms existing methods on six datasets, including four object-recognition and two video action-recognition benchmarks.

2 Related Work

Related work develops zero-shot classifiers through semantic compatibility, transductive information, and GAN-based feature synthesis. The paper positions its contribution as the first feedback loop for iterative feature refinement in generalized zero-shot recognition across images and videos.

  • Earlier zero-shot image methods learn semantic embedding classifiers or compatibility functions between semantic and visual feature spaces.
  • Some inductive approaches use only labelled seen-class data, whereas transductive approaches additionally leverage unlabelled unseen-class data through label propagation.
  • GAN-based methods synthesize unseen-class image or video features and use them with seen features to train zero-shot classifiers.
  • Prior feedback methods improved applications including classification, image-to-image translation, and super-resolution, but this paper introduces feedback for iterative feature synthesis in generalized zero-shot recognition.
  • Zero-shot action recognition in videos has received less attention than zero-shot image classification, motivating evaluation across both modalities.

3 Method

TF-VAEGAN extends a VAE-GAN zero-shot recognition pipeline with semantic embedding reconstruction and feedback during feature synthesis, while using decoder embeddings for classification. The method synthesizes unseen-class features from semantic embeddings and transforms visual features into discriminative representations for ZSL and GZSL classifiers.

  • 3 Method: TF-VAEGAN learns to synthesize unseen-class features from seen-class features and class-specific semantic embeddings, then trains ZSL/GZSL classifiers with synthesized and real features.The classifiers map visual features, or transformed visual features, to unseen classes in ZSL and to seen or unseen classes in GZSL.
  • 3.1 Preliminaries: f-VAEGAN: The VAE-GAN combines an encoder, generator, and discriminator: the encoder produces latent noise, the generator synthesizes features, and the discriminator distinguishes real from synthesized features.The VAE uses KL divergence and reconstruction losses, while the GAN uses a WGAN loss.
  • 3.2 Overall Architecture: The semantic embedding decoder reconstructs embeddings from visual or synthesized features and enforces semantic cycle-consistency with an ℓ1 reconstruction loss.This constraint is intended to keep generated features aligned with the embeddings that generated them.
  • 3.3 Semantic Embedding Decoder: For classification, latent decoder embeddings are concatenated with real or synthesized visual features to form transformed discriminative features.The resulting representations are used to learn final ZSL and GZSL classifiers.
  • 3.4 Feedback Module: A feedback module transforms the decoder’s latent embedding and feeds it into the generator to iteratively refine synthesized features during training and feature synthesis.The feedback additively modulates generator-layer outputs, and the generator reuses the same noise and semantic embedding in the next sub-iteration.

4 Experiments

Experiments evaluate TF-VAEGAN across four object-recognition datasets and multiple ZSL/GZSL settings, including fine-tuned and transductive variants. The method consistently outperforms the f-VAEGAN baseline and existing methods, while ablations and visualizations support contributions from feedback and discriminative feature transformation.

  • Experimental Setup: Four object-recognition datasets—CUB, FLO, SUN, and AWA—are evaluated using standard splits, protocols, and class embeddings.The datasets contain 200, 102, 717, and 50 categories, respectively, with 2048-dimensional ResNet-101 visual features.
  • State-of-the-art Comparison: TF-VAEGAN outperforms f-VAEGAN on all four datasets in inductive and transductive ZSL settings.Inductive ZSL scores are 64.9%, 70.8%, 66.0%, and 72.2% on CUB, FLO, SUN, and AWA; transductive T1 scores are 74.7%, 92.6%, 70.9%, and 92.1%.
  • State-of-the-art Comparison: TF-VAEGAN improves over f-VAEGAN across fine-tuned inductive and transductive settings for both ZSL and GZSL.For FT-IN GZSL, harmonic-mean gains over f-VAEGAN are 1.8%, 4.3%, 3.2%, and 1.5% on CUB, FLO, SUN, and AWA.
  • Ablation Study: On CUB, Feedback and T-feature each improve over the baseline, while their combination produces the strongest ZSL and GZSL results.The final TF-VAEGAN gains 3.7% for ZSL and 4.6% for GZSL over the inductive baseline.
  • Ablation Study: t-SNE visualizations show improved inter-class grouping and fewer misclassifications for fine-grained CUB categories compared with the baseline.The comparison focuses on Cactus Wren, Winter Wren, Sage Thrasher, and Northern Waterthrush.
  • Generalization Capabilities: Integrating the contributions into f-CLSWGAN yields TF-CLSWGAN performance above vanilla f-CLSWGAN on all datasets for both ZSL and GZSL.This comparison evaluates the generalization of the proposed contributions beyond the VAE-GAN architecture.
  • Feature Visualization: Feedback-synthesized features invert to images that are generally semantically closer to ground-truth images than baseline synthesized features.The qualitative feature-visualization experiment uses four example flower classes from FLO.

5 (Generalized) Zero-Shot Action Recognition

TF-VAEGAN generalizes to zero-shot action recognition using the same underlying I3D video features as prior work. On HMDB51 and UCF101, it performs favorably against existing methods in both ZSL and GZSL settings.

  • TF-VAEGAN uses I3D features and the same evaluation protocols as CEWGAN to assess generalization to zero-shot action recognition.The goal is to improve performance without relying on improved video features.
  • TF-VAEGAN performs favorably against existing methods on both HMDB51 and UCF101 for ZSL and GZSL action recognition.
  • 33.0% ZSL and 37.6% GZSL are achieved by TF-VAEGAN on HMDB51, compared with 30.2% and 36.1% for CEWGAN.

6 Conclusion

The proposed VAE-GAN framework uses a semantic embedding decoder throughout training, feature synthesis, and classification. Feedback modulates generator representations, while discriminative feature transformation uses decoder embeddings during classification.

  • The method deploys a semantic embedding decoder at all stages of the zero-shot learning framework: training, feature synthesis, and classification.
  • The feedback module transforms decoder latent embeddings and uses them to modulate the generator's latent representations during training and feature synthesis.
  • A discriminative feature transformation combines decoder latent embeddings with corresponding features during classification.
  • Experiments on six datasets report favorable performance compared with existing methods.

A Quantitative Results

Ablations on CUB show that using decoder outputs in the feedback module and adopting the alternate training strategy progressively improve ZSL and GZSL classification performance.

  • Feedback design choices: 61.4% ZSL and 53.3% GZSL are obtained by the TwoStage+D feedback configuration on CUB.
  • Feedback design choices: 62.0% ZSL and 53.8% GZSL are obtained when feedback input comes from the semantic embedding decoder in TwoStage+Dec.
  • Feedback design choices: 62.8% ZSL and 54.8% GZSL are achieved by Our Feedback, which combines TwoStage+Dec with an alternate training strategy.
  • Feedback design choices: The ablation results identify decoder-based feedback and the alternate training strategy as improvements over the original TwoStage+D setup.

B.1 Feature Visualization Comparison

Feature visualization compares images inverted from ground-truth, baseline-synthesized, and feedback-synthesized features. Feedback features generally produce images that are semantically closer to the ground truth across flower examples.

  • Implementation details: The image generator inverts feature instances into 64x64 images using a fully connected layer followed by five upconvolutional blocks.
  • Visualization: For each flower class, visualizations include a ground-truth image and inversions from its real, baseline-synthesized, and feedback-synthesized features.
  • Visualization: Feedback-synthesized features generally yield inversions that are semantically closer to the ground-truth image than baseline-synthesized features.
  • Visualization: Feedback improves flower color, petal shape, bud shape, and structural details in the qualitative comparisons.

B.2 Classification Performance Comparison

The section qualitatively compares TF-VAEGAN with baseline f-VAEGAN on confusing categories from the CUB and Oxford Flowers datasets. Ground-truth instances, baseline predictions, and proposed-method predictions are arranged for visual comparison.

  • The comparison examines five most confusing categories and five image instances per category on both CUB and Oxford Flowers.Categories are selected with respect to baseline f-VAEGAN performance.
  • For each dataset, the top row shows variations of ground-truth class instances, while the second and third rows show baseline and proposed-method predictions.
  • Green boxes mark correct classification predictions and red boxes mark incorrect predictions in the qualitative comparisons.
Loading 2003.07833v2…