PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantics Disentangling for Generalized Zero-Shot Learning

Zhi Chen, Yadan Luo, Ruihong Qiu, Sen Wang, Zi Huang, Jingjing Li, Zheng Zhang

arXiv:2101.07978v5cs.CV

TL;DR

GZSL methods can rely on visual features that are not semantically aligned with shared attributes, limiting transfer to unseen classes. SDGZSL generates unseen visual features with a conditional VAE and factorizes them into independent semantic-consistent and semantic-unrelated representations. Across canonical and generalized zero-shot learning tasks, the paper reports stronger generalization from the semantic-consistent features.

  • Problem

    Existing GZSL approaches may use visual features containing information unrelated to annotated attributes, which limits semantic-visual alignment for unseen-class transfer.

  • Method

    SDGZSL combines conditional-VAE visual feature generation with relation-guided factorization into semantic-consistent and semantic-unrelated representations, using total correlation to encourage independence.

  • Results

    Experiments on four benchmark datasets report that semantic-consistent features are more generalizable for canonical and generalized zero-shot learning.

  • Takeaways & Limitations

    Separating semantic-consistent information from unrelated visual content improves the reported use of visual representations for zero-shot classification.

Abstract

from arXiv · show

Generalized zero-shot learning (GZSL) aims to classify samples under the assumption that some classes are not observable during training. To bridge the gap between the seen and unseen classes, most GZSL methods attempt to associate the visual features of seen classes with attributes or to generate unseen samples directly. Nevertheless, the visual features used in the prior approaches do not necessarily encode semantically related information that the shared attributes refer to, which degrades the model generalization to unseen classes. To address this issue, in this paper, we propose a novel semantics disentangling framework for the generalized zero-shot learning task (SDGZSL), where the visual features of unseen classes are firstly estimated by a conditional VAE and then factorized into semantic-consistent and semantic-unrelated latent vectors. In particular, a total correlation penalty is applied to guarantee the independence between the two factorized representations, and the semantic consistency of which is measured by the derived relation network. Extensive experiments conducted on four GZSL benchmark datasets have evidenced that the semantic-consistent features disentangled by the proposed SDGZSL are more generalizable in tasks of canonical and generalized zero-shot learning. Our source code is available at https://github.com/uqzhichen/SDGZSL.

1. Introduction

GZSL addresses classification when test classes include both seen and unseen categories, but extracted visual features can contain information unrelated to annotated semantics. SDGZSL disentangles visual features into semantic-consistent and semantic-unrelated representations using relation learning and total correlation, improving reported GZSL performance.

  • GZSL evaluates models when both seen and unseen images must be identified, unlike conventional ZSL, which assumes only unseen test classes.
  • Pre-trained visual features can include salient information unrelated to annotated attributes, weakening semantic-visual alignment and harming transfer to unseen classes.
  • SDGZSL factorizes visual features into semantic-consistent and semantic-unrelated latent representations, using a relation module for semantic consistency and a total correlation penalty for independence.
  • The framework is reported to substantially improve GZSL performance over directly using entangled features from pre-trained CNNs.
  • Experiments on four benchmark datasets report that SDGZSL performs better than state-of-the-art methods, with an exception on AWA.

2. Related Work

Prior GZSL work includes generative models that synthesize visual features from semantic information and disentanglement methods that separate feature types. SDGZSL instead enforces independence between semantic-consistent and semantic-unrelated representations while integrating generation and disentangling end to end.

  • Generative GZSL methods synthesize unseen-class visual features from seen data and semantic representations, commonly using GANs, VAEs, or both.
  • Total correlation is used in disentanglement research to encourage independence, while this work applies it between two representations rather than across dimensions within one representation.
  • DLFZRL separates semantic, non-semantic, and non-discriminative features, whereas SDGZSL filters non-semantic features and combines generation with disentangling in an end-to-end model.

3. Proposed Approach

SDGZSL factorizes visual features into semantic-consistent and semantic-unrelated representations for GZSL. It combines relation-based semantic alignment, reconstruction, conditional generation, and total-correlation regularization before classifying seen and generated unseen representations.

  • Semantics Disentanglement: SDGZSL factorizes each visual feature into semantic-consistent h_s and semantic-unrelated h_n representations using an encoder-decoder architecture.The encoder maps visual features to a latent representation, while the decoder reconstructs the original feature from both factors.
  • Semantic-Visual Relationship Learning: A relation network learns h_s by maximizing compatibility with the corresponding class semantic embeddings.Matched representation-embedding pairs receive compatibility target 1, while mismatched pairs receive target 0.
  • Disentanglement by Total Correlation Penalty: A total correlation penalty encourages independence between h_s and h_n by contrasting jointly encoded representations with representations formed through batch-wise permutation.A discriminator estimates whether its input follows the independent distribution, and the permuted representation is used for contrast.
  • Conditional Generation: A conditional VAE models visual features conditioned on class embeddings so that representations for unseen classes can be generated from semantic information.The inference network produces latent variables from visual features and embeddings, while the generator uses them to model the conditioned visual distribution.
  • Training and Inference: After training, generated unseen semantic-consistent representations are combined with seen features to train a supervised Softmax classifier.The model generates unseen representations from Gaussian noise and unseen semantic embeddings, then applies supervised classification to seen and unseen objects.

4. Experiments

Experiments on four benchmark datasets evaluate SDGZSL against existing methods, conventional zero-shot learning, retrieval, ablations, hyper-parameters, and disentangled representations. The results generally favor semantic-consistent representations and the full model, while visualizations and retrieval analyses examine why.

  • Comparison with State-of-the-art Methods: SDGZSL consistently outperforms compared methods on GZSL except on AWA, and surpasses DLFZRL on all reported datasets.The comparison includes recent embedding-based and generative methods using pre-trained or fine-tuned ResNet101 features.
  • Conventional Zero-shot Learning Results: SDGZSL performs better than all other methods on CUB and FLO in conventional zero-shot learning, while DLFZRL and E-PGN are slightly higher on aPaY and AWA, respectively.
  • Zero-shot Retrieval Results: The disentangling module significantly boosts zero-shot image retrieval over cVAE when retrieving 100%, 50%, and 25% of unseen-class images across aPaY, AWA, CUB, and FLO.Figure 4 compares cVAE and SDGZSL using mean average precision.
  • Model Analysis: The full model performs best when both the relation network and total-correlation penalty are applied, while semantic-consistent hs improves GZSL and semantic-unrelated hn performs worse than h.Using both hs and hn together yields performance similar to cVAE.
  • Model Analysis: On CUB, semantic-consistent hs is more discriminative than semantic-unrelated hn, although hn retains discriminative patterns from features not annotated in the attributes.The t-SNE visualization uses 50 unseen classes.

5. Conclusion

The paper concludes that SDGZSL factorizes visual features into independent semantic-consistent and semantic-unrelated representations within a conditional VAE. Across four image-classification datasets, the approach consistently outperforms other state-of-the-art methods.

  • SDGZSL factorizes ResNet101 visual features into independent semantic-consistent and semantic-unrelated representations using a relation module and total-correlation term.The disentangling encoder-decoder is incorporated into a conditional VAE and trained end-to-end.
  • Across four image-classification datasets, SDGZSL consistently performs better than other state-of-the-art methods.The model transfers generation learned on seen classes to synthesize missing visual samples for unseen classes.

A. Total Correlation Approximation

The total-correlation approximation estimates dependence between the two disentangled representations through a discriminator. The resulting density ratio is used to approximate total correlation.

  • A discriminator Disϕ(h) approximates τ(y = 1 | h), allowing the density ratio between dependent and independent representation pairs to be estimated.
  • The estimated density ratio is then used to approximate the total correlation between hs and hn.

B. Implementation

The implementation uses fully connected encoder, decoder, and relation modules with LeakyReLU, ReLU, and dropout components. Training uses Adam and specifies ranges for the principal loss weights.

  • The encoder and decoder use one and two fully connected layers, while each layer is followed by LeakyReLU and dropout.Their hidden-unit sizes are [l + m] for the encoder and [2048, 2048] for the decoder.
  • The relation module uses two fully connected layers, and the relation, total-correlation, and discriminator weights are set between 0.1 and 5.
  • The cVAE uses a five-layer MLP encoder and a three-layer MLP decoder, with Adam optimization using β1 = 0.9, β2 = 0.999, and batch size 64.

C. Class-wise Analysis

SDGZSL improves class-wise recognition of unseen AWA classes over a standard cVAE, reducing several confusions between visually similar categories.

  • C. Class-wise Analysis: Almost all 10 unseen AWA classes achieve higher accuracy with SDGZSL than with the standard cVAE.The comparison uses confusion matrices for unseen test samples.
  • C. Class-wise Analysis: 41% of horse samples misclassified as sheep with cVAE falls to 9% with SDGZSL.
  • C. Class-wise Analysis: 31% of sheep samples misclassified as cow with cVAE falls to 3% with SDGZSL, while rat-to-hamster confusion changes from 35% to 35%.

D. Comparison with Traditional Methods

The paper evaluates semantic-consistent representations with traditional embedding methods and reports improved GZSL performance, while confusion matrices compare cVAE and SDGZSL predictions.

  • D. Comparison with Traditional Methods: Figure 8 compares cVAE and SDGZSL confusion matrices, with ground-truth classes on the vertical axis and predictions on the horizontal axis.
  • D. Comparison with Traditional Methods: Traditional embedding methods perform poorly in GZSL, especially on unseen classes, where samples tend to be misclassified into seen classes.The paper attributes this to simple embedding functions failing to clearly distinguish seen and unseen class domains.
  • D. Comparison with Traditional Methods: Training ALE with semantic-consistent representations instead of original visual features boosts performance by a large margin.The improvement supports transferring visual-semantic relationships from seen to unseen classes.
Loading 2101.07978v5…