PaperlayerView exact revision on arXiv

Source-linked AI summary

TransZero: Attribute-guided Transformer for Zero-Shot Learning

Shiming Chen, Ziming Hong, Yang Liu, Guo-Sen Xie, Baigui Sun, Hao Li, Qinmu Peng, Ke Lu, Xinge You

arXiv:2112.01683v1cs.CVcs.AI

TL;DR

ZSL requires transferable, discriminative visual features because existing methods often overlook attribute localization and transferability across seen and unseen classes. TransZero uses an attribute-guided Transformer with feature augmentation, attribute-guided localization, and visual-semantic embedding. Experiments on three benchmark datasets demonstrate the superiority of the approach.

  • Problem

    Existing ZSL methods neglect transferability and discriminative attribute localization, limiting visual representations for visual-semantic interaction.

  • Method

    TransZero combines a feature augmentation encoder, a visual-semantic decoder, and a visual-semantic embedding network to refine and localize visual features.

  • Results

    Experiments on three popular benchmark datasets demonstrate the superiority of TransZero, which achieves new state-of-the-art performance.

  • Takeaways & Limitations

    TransZero refines visual features and provides attribute-level localization for discriminative visual embedding representations in ZSL.

  • Takeaways & Limitations

    The method is evaluated under ZSL settings with disjoint seen and unseen classes and unlabeled unseen-class samples.

Abstract

from arXiv · show

Zero-shot learning (ZSL) aims to recognize novel classes by transferring semantic knowledge from seen classes to unseen ones. Semantic knowledge is learned from attribute descriptions shared between different classes, which act as strong priors for localizing object attributes that represent discriminative region features, enabling significant visual-semantic interaction. Although some attention-based models have attempted to learn such region features in a single image, the transferability and discriminative attribute localization of visual features are typically neglected. In this paper, we propose an attribute-guided Transformer network, termed TransZero, to refine visual features and learn attribute localization for discriminative visual embedding representations in ZSL. Specifically, TransZero takes a feature augmentation encoder to alleviate the cross-dataset bias between ImageNet and ZSL benchmarks, and improves the transferability of visual features by reducing the entangled relative geometry relationships among region features. To learn locality-augmented visual features, TransZero employs a visual-semantic decoder to localize the image regions most relevant to each attribute in a given image, under the guidance of semantic attribute information. Then, the locality-augmented visual features and semantic vectors are used to conduct effective visual-semantic interaction in a visual-semantic embedding network. Extensive experiments show that TransZero achieves the new state of the art on three ZSL benchmarks. The codes are available at: \url{https://github.com/shiming-chen/TransZero}.

1 Introduction

ZSL transfers semantic knowledge from seen to unseen classes, but existing methods struggle to represent fine-grained attributes, transfer visual features, and localize discriminative regions. TransZero addresses these challenges with an attribute-guided Transformer and visual-semantic embedding network.

  • ZSL recognizes unseen classes without training samples by exploiting semantic relatedness between disjoint seen and unseen label spaces.
  • Attention-based ZSL methods learn region embeddings but neglect visual-feature transferability and discriminative attribute localization.
  • Global visual features miss fine-grained information because discriminative class evidence may occur in only a few attribute-specific regions.
  • TransZero reduces entangled relationships among region features and localizes object attributes to improve transferability and represent discriminative regions.
  • Its feature augmentation encoder addresses cross-dataset bias and region-geometry entanglement, while its visual-semantic components support attribute-guided interaction.
  • TransZero achieves new state-of-the-art performance on three ZSL benchmarks and qualitatively refines visual features for fine-grained part localization.

2 Related Work

Prior ZSL work maps visual features to semantic domains, while Transformer architectures provide a basis for modeling region relationships. TransZero combines these directions to learn transferable, attribute-localized visual representations and visual-semantic interactions.

  • Early ZSL methods learn mappings between global visual features and semantic domains to transfer knowledge from seen to unseen classes.
  • Transformer success in language and vision is associated with self-supervision and self-attention for modeling complex relationships.
  • TransZero’s architecture combines an attribute-guided Transformer with a visual-semantic embedding network.
  • The proposed Transformer reduces entangled region relationships and learns attribute localization for discriminative region representations.

3 Proposed Method

TransZero addresses ZSL, where unseen classes lack training samples and semantic vectors transfer knowledge across disjoint seen and unseen label spaces. Its architecture combines an attribute-guided Transformer with a visual-semantic embedding network for classification.

  • ZSL trains on seen classes and predicts unseen classes without available unseen-class training samples.
  • Class semantic vectors with attribute descriptions support knowledge transfer between seen and unseen classes.
  • Conventional ZSL predicts only unseen classes, whereas generalized ZSL predicts both seen and unseen classes.
  • TransZero refines visual features and localizes object attributes to represent discriminative region features for visual-semantic interaction.
  • TransZero combines an attribute-guided Transformer for locality-augmented visual features with a visual-semantic embedding network for ZSL classification.

Attribute-Guided Transformer

The attribute-guided Transformer improves feature transferability by reducing region-feature entanglement and uses semantic attribute queries to localize relevant image regions. Its decoder then produces locality-augmented visual features.

  • Feature Augmentation Encoder: A feature augmentation encoder addresses cross-dataset bias between ImageNet and ZSL benchmarks.
  • Feature Augmentation Encoder: Flattened grid features entangle representations across image regions, hindering transferability between domains.
  • Feature Augmentation Encoder: Relative geometry features are constructed from pairwise grid positions and subtracted from visual features to produce a more accurate attention map.
  • Feature Augmentation Encoder: The encoder represents packed visual features learned from flattened features through a fully connected layer, ReLU, and dropout.
  • Visual-Semantic Decoder: The decoder uses visual encoder outputs as keys and values and learnable semantic attribute embeddings as queries.
  • Visual-Semantic Decoder: Guided by semantic attribute features, the decoder localizes image regions most relevant to each attribute.
  • Visual-Semantic Decoder: A multi-head self-attention layer and feed-forward network transform attended features into locality-augmented visual features.

Visual-Semantic Embedding Network

The visual-semantic embedding network maps locality-augmented visual features into an attribute-based semantic space. It uses learned attribute vectors to produce attribute scores and a mapped semantic embedding.

  • The network maps locality-augmented visual features into the semantic embedding space using semantic attribute vectors as support.
  • A mapping function matches locality-augmented visual features with semantic attribute information.
  • An embedding matrix projects visual features into the semantic attribute space.
  • ψ(x_i)[a] is an attribute score representing confidence that attribute a is present in image x_i.
  • TransZero obtains a mapped semantic embedding ψ(x_i) from the set of semantic attribute vectors.

Model Optimization

TransZero is optimized with attribute regression, attribute-based cross-entropy, and self-calibration losses. These objectives constrain semantic mapping, class compatibility, and seen-to-unseen probability calibration.

  • TransZero uses attribute regression, attribute-based cross-entropy, and self-calibration losses for optimization.
  • Attribute Regression Loss: Attribute regression minimizes mean squared error between ground-truth attributes and embedded attribute scores.
  • Attribute-Based Cross-Entropy Loss: Attribute-based cross-entropy computes compatibility between visual embeddings and class semantic vectors to favor the corresponding class.
  • Self-Calibration Loss: Self-calibration shifts prediction probability from seen toward unseen classes because regression and cross-entropy losses overfit seen classes.
  • The overall objective combines attribute-based cross-entropy with weighted attribute regression and self-calibration terms.
  • Table 1 reports CZSL and GZSL results across CUB, SUN, and AWA2 for end-to-end and non-end-to-end methods.

Zero-Shot Prediction

After training, TransZero embeds a test instance into the semantic space and applies explicit calibration to predict its label. The candidate label space depends on whether evaluation uses CZSL or GZSL.

  • Zero-Shot Prediction: Explicit calibration then predicts the test label from the embedded instance.
  • Zero-Shot Prediction: C_u corresponds to CZSL, whereas C corresponds to GZSL in the prediction formulation.

4 Experiments

TransZero is evaluated on three ZSL benchmarks using top-1 accuracy for CZSL and seen, unseen, and harmonic-mean metrics for GZSL. It achieves leading fine-grained performance and strong generalized recognition, including high harmonic means on CUB and AWA2.

  • Datasets and Metrics: The experiments cover CUB and SUN as fine-grained benchmarks and AWA2 as a coarse-grained benchmark.CUB contains 200 bird classes and 312 attributes; SUN contains 717 scene classes and 102 attributes.
  • Datasets and Metrics: Top-1 accuracy is measured in CZSL, while GZSL reports seen accuracy S, unseen accuracy U, and harmonic mean H.H is defined as H = (2 × S × U)/(S + U).
  • Conventional Zero-Shot Learning: 76.8% and 65.6% top-1 accuracy are achieved on CUB and SUN in CZSL, respectively.TransZero also obtains 70.1% top-1 accuracy on AWA2.
  • Generalized Zero-Shot Learning: GZSL methods generally score higher on seen than unseen classes for CUB and AWA2, whereas SUN exhibits U > S.The SUN pattern is attributed to its much larger number of seen classes than unseen classes.
  • Generalized Zero-Shot Learning: 68.8% and 70.2% harmonic mean are achieved on CUB and AWA2 in GZSL, respectively.The reported improvements over attention-based methods are at least 1.6%, 3.2%, and 3.1% on CUB, SUN, and AWA2.

Ablation Study

Ablations show that feature augmentation, the visual-semantic decoder, self-calibration, and attribute regression each contribute to TransZero's performance and feature localization.

  • Component Ablations: Removing the feature augmentation encoder drops CUB acc/harmonic mean by 9.5%/12.0% and SUN by 4.4%/8.7%.
  • Component Ablations: Removing feature augmentation from the standard encoder drops CUB acc/harmonic mean by 2.8%/2.3% and SUN by 1.8%/2.3%.
  • Component Ablations: Removing the visual-semantic decoder substantially reduces performance across all datasets.
  • Component Ablations: Self-calibration improves CUB and SUN harmonic mean by 10.7% and 3.4%, respectively, while attribute regression further improves visual-semantic mapping.
  • Feature Visualization: TransZero localizes attribute-specific regions more precisely than AREN, which primarily learns broader region embeddings such as the whole bird body.
  • Feature Visualization: The feature augmentation encoder further improves the quality of unseen visual features beyond the standard encoder.

Hyperparameter Analysis

The analysis examines visual-feature transformations and the attribute-regression loss weight. Large λ_AR values hurt evaluation, while λ_AR = 0.005 gives the best performance; increasing λ_SC favors seen over unseen accuracy.

  • Attribute Regression Weight: The attribute-regression loss weight λ_AR is evaluated over {0.001, 0.005, 0.01, 0.05, 0.1, 0.5}.
  • Feature Visualization: t-SNE compares seen and unseen visual features from the CNN backbone, encoder without FA, encoder, and decoder across 10 randomly selected CUB classes.
  • Attribute Regression Weight: λ_AR = 0.005 produces the best performance, whereas large λ_AR values reduce all evaluation protocols.Large values make the attribute-regression loss dominate and mitigate contributions from other losses.
  • Self-Calibration Weight: Increasing λ_SC raises seen-class accuracy and lowers unseen-class accuracy, while TransZero is insensitive to self-calibration loss in CZSL.The selected setting is λ_SC = 0.3 for all datasets.

5 Conclusion

TransZero is introduced as an attribute-guided Transformer network for ZSL, combining feature refinement, attribute localization, and visual-semantic interaction. Experiments on three benchmark datasets demonstrate the superiority of the approach.

  • TransZero is introduced as an attribute-guided Transformer network for ZSL.
  • Its feature augmentation encoder addresses cross-dataset bias and entangled region-feature relationships to improve visual-feature discriminability and transferability.
  • A visual-semantic decoder learns attribute localization and locality-augmented visual features, which a visual-semantic embedding network interacts with class semantic vectors.
  • Extensive experiments on three popular benchmark datasets demonstrate the superiority of TransZero.
Loading 2112.01683v1…