Source-linked AI summary
Semantic-Guided Multi-Attention Localization for Zero-Shot Learning
Yizhe Zhu, Jianwen Xie, Zhiqiang Tang, Xi Peng, Ahmed Elgammal
TL;DR
Zero-shot learning seeks recognition of classes with few or no training images, while existing approaches often neglect discriminative visual features needed to distinguish fine-grained classes. SGMA weakly supervises semantic-guided multi-attention localization and jointly learns global and local features with embedding softmax and class-center triplet losses. The proposed method outperforms state-of-the-art methods; on CUB, it improves the H score by 6.7% (48.5% v.s. 41.8%).
Problem
Zero-shot learning seeks recognition of classes with few or no training images, while existing approaches often neglect discriminative visual features needed to distinguish fine-grained classes.
Source: §1 · p. 1Method
SGMA weakly supervises semantic-guided multi-attention localization and jointly learns global and local features with embedding softmax and class-center triplet losses.
Source: §1 · p. 2Results
The proposed method outperforms state-of-the-art methods; on CUB, it improves the H score by 6.7% (48.5% v.s. 41.8%).
Source: §5 · p. 9, §4.4 · p. 9Takeaways & Limitations
The experiments support the significance of discriminative parts for zero-shot object recognition.
Source: §5 · p. 9Inference assumes unseen-class semantic descriptions can be represented as a linear combination of seen-class descriptions.
Source: §3.4 · p. 6
Abstract
from arXiv · showhide
Zero-shot learning extends the conventional object classification to the unseen class recognition by introducing semantic representations of classes. Existing approaches predominantly focus on learning the proper mapping function for visual-semantic embedding, while neglecting the effect of learning discriminative visual features. In this paper, we study the significance of the discriminative region localization. We propose a semantic-guided multi-attention localization model, which automatically discovers the most discriminative parts of objects for zero-shot learning without any human annotations. Our model jointly learns cooperative global and local features from the whole object as well as the detected parts to categorize objects based on semantic descriptions. Moreover, with the joint supervision of embedding softmax loss and class-center triplet loss, the model is encouraged to learn features with high inter-class dispersion and intra-class compactness. Through comprehensive experiments on three widely used zero-shot learning benchmarks, we show the efficacy of the multi-attention localization and our proposed approach improves the state-of-the-art results by a considerable margin.
1 Introduction
Zero-shot learning uses semantic descriptions to recognize classes with few or no training examples, but existing approaches often underemphasize discriminative visual features. The paper proposes jointly learning semantic-guided object parts and global and local features for zero-shot recognition.
1 Introduction
- Zero-shot learning targets recognition of classes with few or no training images, a setting relevant to objects with scarce visual data.The paper identifies costly collection and annotation of large labeled datasets as a challenge.Source: §1 · p. 1
- Existing approaches often neglect discriminative visual features, and whole-image features may miss subtle differences between fine-grained classes.The paper argues that multiple part regions, such as a bird’s head and tail, can be crucial for distinguishing classes.Source: §1 · p. 2
- The proposed weakly supervised model jointly discovers object regions and learns feature representations under semantic-description guidance.The paper presents this as the first end-to-end joint optimization of multi-attention localization and global and local feature learning for zero-shot learning.Source: §1 · p. 2
- A multi-attention loss encourages compact, diverse attention maps, while embedding softmax and class-center triplet losses supervise global and local features.The paper describes the combined losses as promoting inter-class dispersion and intra-class compactness.Source: §1 · p. 2
- Experiments and analysis on three zero-shot learning datasets demonstrate performance on both part detection and zero-shot learning.The contribution summary characterizes the method’s performance as excellent.Source: §1 · p. 2
2 Related Work
Prior zero-shot approaches use semantic attributes, visual-semantic embeddings, or generated features, while localization methods range from annotated parts to weakly supervised attention. SGMA distinguishes itself by using semantic guidance to find multiple finer parts and jointly training localization with zero-shot recognition.
2 Related Work
- Zero-shot methods include attribute-based prediction, visual-semantic embedding approaches such as ALE and ESZSL, nonlinear embeddings such as LatEm and DEM, and methods that generate unseen-class features.The related-work discussion identifies SCoRe and LDF as end-to-end approaches addressing discriminative feature learning; LDF adds a zoom network.Source: §2 · p. 2
- Unlike LDF’s square crop of the whole object, SGMA’s multi-attention localization can find multiple finer discriminative regions, such as a head or tail.Figure 1 depicts attention maps and crops feeding part images and original images to separate CNNs.Source: §2 · p. 3, Figure 1 · p. 3
- Part-box methods use human annotations, whereas SGMA learns part attention weakly supervised and uses semantic guidance to optimize localization jointly with zero-shot recognition.The paper contrasts this with approaches that use classification-loss supervision for attention localization.Source: §2 · p. 3
3 Method
SGMA predicts labels for unseen-class images from semantic descriptions by learning global and local visual features. It discovers discriminative parts and combines embedding and class-center objectives to train the model end to end.
3 Method
- Unseen-class prediction uses labeled examples from seen classes and semantic representations for the disjoint unseen classes.The task is to predict an unseen image’s class label from its image and the set of unseen-class semantic representations.Source: §3 · p. 3
- Three modules generate attention maps, crop discriminative parts, and jointly learn global and local features for zero-shot recognition.The joint feature subnet uses the original image alongside the cropped parts.Source: §3 · p. 3
3.1 Multi-Attention Subnet
- Global average pooling supplies channel descriptors for part-specific attention weights, which weight feature maps to produce attention maps.The sigmoid-gated weights are intended to emphasize channels associated with each part and downweight irrelevant channels.Source: §3.1 · p. 4
- Compactness and diversity losses encourage concentrated attention maps to focus on distinct discriminative regions.The diversity loss discourages other maps from activating strongly at the same location and uses a maximum-margin design to reduce sensitivity to noise.Source: §3.1 · pp. 4–5
3.2 Region Cropping Subnet
- A cropping network predicts square-region parameters from attention maps, enabling differentiable cropping of the corresponding image parts.The predicted parameters specify the square’s center and side length; the cropped region is resized to the original image size.Source: §3.2 · p. 5
3.3 Joint Feature Learning Subnet
- The joint feature subnet processes the original image and cropped parts through separate VGG19 backbones to learn global and local features.Embedding softmax encourages inter-class distinction, while class-center triplet loss encourages within-class compactness.Source: §3.3 · p. 5
Embedding Softmax Loss
- Embedding softmax treats visual-semantic compatibility scores as logits, with late fusion summing scores from the original image and part images.The fusion strategy discards the extra dimension-reduction layer used after feature concatenation in a cited approach.Source: Embedding Softmax Loss · pp. 5–6
Class-Center Triplet Loss
- Class-center triplet loss encourages intra-class compactness using trainable class centers and avoids the triple sampling required by naive triplet loss.Feature normalization places points on a unit hypersphere to make setting the margin easier.Source: Class-Center Triplet Loss · p. 6
4 Experiment
Experiments evaluate the method's implementation, datasets, weakly supervised part localization, zero-shot classification, and ablations. The model outperforms prior methods on fine-grained zero-shot benchmarks, while component tests examine localized regions and joint-loss supervision.
4 Experiment
- The experiments use two attention maps because more maps cause attended regions to overlap severely and hardly improve zero-shot performance.Source: §4 · p. 6
4.1 Implementation Details and Model Initialization
- Implementation uses VGG19, 448 × 448 inputs for attention maps, and 224 × 224 inputs for joint feature embedding; initialization uses channel clustering and attention-derived pseudo-labels.Source: §4.1 · p. 7
4.2 Datasets and Experiment Settings
- Experiments cover fine-grained CUB and FLO and coarse-grained AwA; evaluation uses Mean Class Accuracy and both standard and proposed zero-shot splits.The proposed split avoids test classes used to train the feature extractor in the standard split.Source: §4.2 · p. 7
4.3 Part Detection Results
- Part detection improves by 35.7% over random crops on average; it reaches 61.5% versus 79.1% for annotation-trained SPDA-CNN, while removing multi-attention loss lowers performance to 47.6%.Detected regions roughly align with semantic parts, including bird heads and legs, animal heads and rear bodies, and flower stamens, pistils, and petals.Source: §4.3 · p. 8, Table 1 · p. 8
4.4 Zero-Shot Classification Results
- The method improves state-of-the-art zero-shot results by 3.4% on CUB SS, 3.5% on CUB PS, and 5.0% on FLO; shared CNN weights reduce complexity but lower CUB-PS and AWA-PS scores by 3.6% and 2.7%.Source: §4.4 · p. 8, Table 2 · p. 8
- In generalized zero-shot learning, the model outperforms prior methods on H; on CUB, H improves by 6.7% (48.5% vs. 41.8%).Source: §4.4 · p. 9, Table 3 · p. 9
4.5 Ablation Study
- Combining the original image with localized parts improves baseline performance by 5.4% (65.2% vs. 59.8%); random parts often hurt, while combined joint-loss inferences improve the baseline by 3.9%.Source: §4.5 · p. 9
5 Conclusion
The conclusion presents discriminative-part localization as important for zero-shot recognition. It summarizes a semantic-guided multi-attention model that jointly learns global and local features, and reports better performance than state-of-the-art methods.
5 Conclusion
- The method uses semantic-guided multi-attention localization to detect discriminative object parts and jointly learn global and local features end to end.Source: §5 · p. 9
- Multi-attention loss favors compact, diverse attention, while embedding softmax and class-center triplet losses supervise the image and discovered-part features.Source: §5 · p. 9
- Experiments on three benchmarks show that the proposed method outperforms state-of-the-art methods.Source: §5 · p. 9