PaperlayerView exact revision on arXiv

Source-linked AI summary

Attribute Prototype Network for Zero-Shot Learning

Wenjia Xu, Yongqin Xian, Jiuniu Wang, Bernt Schiele, Zeynep Akata

arXiv:2008.08290v4cs.CVcs.LG

TL;DR

Zero-shot learning needs image representations that transfer attributes to unseen classes while localizing visual evidence, but training-class correlations can hinder novel attribute combinations. APN jointly learns global and local features using class-level attributes, with prototype regression and decorrelation, and reports consistent benchmark gains plus improved attribute localization.

  • Problem

    Existing zero-shot representations do not directly provide attribute localization and may rely on correlations among attributes from seen classes, limiting novel configurations.

  • Method

    APN jointly learns global and local image features by regressing and decorrelating attribute prototypes from intermediate features using only class-level attributes and semantic relatedness.

  • Results

    APN consistently improves over state-of-the-art methods across CUB, AWA2, and SUN in zero-shot and generalized zero-shot learning, including harmonic means of 67.2% on CUB, 37.6% on SUN, and 65.5% on AWA2.

  • Takeaways & Limitations

    The learned representations improve zero-shot models and provide attribute-level visual evidence through localized prototype responses without part annotations during training.

  • Takeaways & Limitations

    The method still conflates some correlated attributes, such as white belly and white breast, and zero-shot learning remains sensitive to biased side information and lower accuracy than seen-and-unseen-class training.

Abstract

from arXiv · show

From the beginning of zero-shot learning research, visual attributes have been shown to play an important role. In order to better transfer attribute-based knowledge from known to unknown classes, we argue that an image representation with integrated attribute localization ability would be beneficial for zero-shot learning. To this end, we propose a novel zero-shot representation learning framework that jointly learns discriminative global and local features using only class-level attributes. While a visual-semantic embedding layer learns global features, local features are learned through an attribute prototype network that simultaneously regresses and decorrelates attributes from intermediate features. We show that our locality augmented image representations achieve a new state-of-the-art on three zero-shot learning benchmarks. As an additional benefit, our model points to the visual evidence of the attributes in an image, e.g. for the CUB dataset, confirming the improved attribute localization ability of our image representation.

1 Introduction

Visual attributes enable semantic transfer in zero-shot learning, but standard image representations do not directly support attribute localization and may exploit correlations tied to seen classes. APN addresses this by jointly learning localized, decorrelated attributes from intermediate features using only class-level supervision.

  • Motivation: Visual attributes provide semantic knowledge transfer from seen to unseen classes, motivating locality-aware image representations for zero-shot learning.Locality denotes associating an image region with a visual attribute.
  • Motivation: Existing attention and attribute-classifier approaches can learn correlations among training attributes, limiting recognition of novel attribute configurations.The paper illustrates this with frequently co-occurring attributes such as yellow crown and yellow belly.
  • Approach: APN learns local features by injecting intermediate-layer losses that regress and decorrelate visual attributes.The framework uses weak supervision and semantic relatedness rather than human-annotated associations between local features and attributes.
  • Approach: The model uses only class-level attributes and their semantic relatedness as supervision for learning localized attribute features.No human-annotated correspondence between local features and visual attributes is provided during training.
  • Contributions: APN consistently improves over the state of the art on CUB, AWA2, and SUN in both zero-shot and generalized zero-shot settings.The authors also report qualitative attribute localization without part annotations and better part detection than a recent weakly supervised method.

2 Related work

Prior work studies semantic embeddings, generative feature synthesis, attention, and prototype learning, but APN focuses on spatial attribute prototypes with explicit semantic meaning. This targets attribute and part localization while improving the locality of zero-shot image representations.

  • Zero-shot learning: Zero-shot learning methods commonly learn compatibility functions between image representations and class embeddings, while recent work also synthesizes features for unseen classes.These approaches focus primarily on visual-semantic embeddings or generative models.
  • Zero-shot learning: Image representation learning remains relatively underexplored in zero-shot learning, with prior methods improving locality mainly through learned visual attention.APN instead improves locality by learning a prototype network that localizes attributes.
  • Prototype learning: Prototype networks typically classify by distances to learned class prototypes, whereas APN learns spatial prototypes associated with input patches for attributes.This shifts prototype learning from sample-based class decisions toward attribute-linked spatial features.
  • Prototype learning: Prior latent attention or prototype methods often assign semantic meaning post hoc, limiting attribute or part localization ability.The cited comparison notes that one prior method can localize only two parts.
  • Locality and representation learning: Local features are widely used in representation learning and fine-grained tasks, while this work focuses on spatial feature configuration rather than feature channels.The paper defines a local feature as one encoded from a local image region.
  • Method overview: APN combines an Image Encoder, BaseMod, and ProtoMod to learn global classification features alongside local attribute prototypes and similarity maps.The end-to-end framework is presented as the paper’s locality-oriented representation-learning architecture.

3 Attribute Prototype Network

The Attribute Prototype Network jointly learns global discriminative features and local attribute-aware features for zero-shot recognition. It combines visual-semantic embedding with prototype-based attribute regression, decorrelation, and localization using class-level supervision.

  • Problem: ZSL transfers knowledge from seen to unseen classes using class embeddings such as attribute vectors.The model predicts unseen classes from image representations and known class embeddings.
  • Base Module: The Image Encoder produces spatial feature maps, while BaseMod applies global average pooling to obtain discriminative global features.For f(x) ∈ R^(H×W×C), pooling over spatial locations yields g(x) ∈ R^C.
  • Base Module: A visual-semantic embedding layer maps global features into attribute space and trains compatibility with the corresponding class embedding.The backbone and embedding layer are jointly optimized using cross-entropy classification loss.
  • Prototype Module: ProtoMod learns attribute prototypes from local features, producing similarity maps whose maxima associate each attribute with its closest image region.Upsampling these maps to input resolution enables attribute localization.
  • Prototype Module: Attribute regression uses class-level attribute vectors to make local features encode semantic attributes without human local-feature-to-attribute annotations.The regression objective minimizes mean squared error between predicted and ground-truth attribute vectors.
  • Prototype Module: Attribute decorrelation encourages competition across unrelated attribute groups and sharing within related groups to reduce reliance on incidental correlations.The method groups semantically related attributes, such as attributes describing the same body part.
  • Joint Learning and Inference: The full objective jointly combines classification, regression, decorrelation, and compactness losses, and the learned features support both ZSL and GZSL inference.The representation can also be used with another ZSL model, such as ABP.

4 Experiments

Experiments evaluate APN on three ZSL benchmarks and compare its recognition and localization performance with baselines and state-of-the-art methods. APN consistently improves zero-shot recognition, generalized zero-shot harmonic mean, and weakly supervised part and attribute localization.

  • Experimental setup: APN is evaluated on CUB, AWA2, and SUN using zero-shot and generalized zero-shot learning settings.CUB contains 11,788 images and 312 attributes, SUN 14,340 images and 102 attributes, and AwA 37,322 images and 85 attributes.
  • Ablation study: 2.0% (CUB), 3.5% (AWA2), and 1.6% (SUN) ZSL accuracy improvements over BaseMod result from adding APN losses.The largest gains come from attribute regression and attribute decorrelation losses, which add locality to the representation.
  • Zero-shot recognition: 67.2% on CUB, 37.6% on SUN, and 65.5% on AWA2 are APN’s generalized zero-shot harmonic means.The reported gains over state-of-the-art non-generative methods indicate improved balance between seen- and unseen-class performance.
  • Generative models: APN features improve generative models, with APN+ABP gains over ABP∗ of 2.6% on CUB and 5.3% on AWA in ZSL.In GZSL, the corresponding gains are 2.7% on CUB and 2.3% on SUN.
  • Part localization: 22.1% higher PCP than BaseMod (52.8% vs 30.7%) demonstrates improved weakly supervised part localization.The largest localization improvements occur for the leg and head.
  • Part localization: 52.8% PCP remains below SPDA-CNN’s 73.6%, although APN requires no part annotations during training.Under the same bounding-box size, APN also improves over SGMA, reaching 79.2% versus 61.5% on average.
  • Qualitative localization: APN produces more concentrated and diverse attention maps and more accurate attribute-level boxes than BaseMod and BaseMod+CAM.For example, APN localizes the black wing to the wing region while BaseMod+CAM places it on the head.
  • Limitations: The decorrelation loss reduces but does not eliminate attribute confusion, including occasional conflation of white belly and white breast.This issue is identified as a challenging direction for future research.

5 Conclusion

The attribute prototype network jointly learns global and local features for zero-shot learning, improving representation locality and localization while consistently outperforming prior methods.

  • APN jointly learns global and local features for zero-shot learning.Local features regress attributes and decorrelate prototypes with regularization.
  • APN consistently improves over the state of the art on three zero-shot learning benchmarks.The benchmarks are CUB, AWA2, and SUN, evaluated in both zero-shot and generalized zero-shot settings.
  • APN representations improve over finetuned ResNet representations when used with feature-generating models.
  • APN accurately localizes attributes in images, with part localization accuracy significantly exceeding a weakly supervised zero-shot localization model.

Broader Impact

Zero-shot learning seeks to recognize novel classes from class-level descriptions rather than annotated examples. The paper highlights attribute localization and decorrelation as useful directions, while noting limitations in accuracy and side-information quality.

  • Zero-shot learning aims to recognize objects in novel classes using class-level descriptions instead of dedicated annotated training data.
  • The attribute prototype network uses local features and decorrelated attribute prototypes to distinguish frequently co-occurring attributes.The paper gives yellow foreheads and yellow crowns in birds as an example of correlated attributes.
  • Zero-shot learning remains less accurate than models trained on both seen and unseen classes, limiting applicability where high accuracy and confidence are required.The passage names self-driving cars as an example of such a setting.
  • Zero-shot generalization depends substantially on side-information quality, so biased descriptions of similarity between seen and unseen classes can harm generalization.
Loading 2008.08290v4…