Source-linked AI summary
MSDN: Mutually Semantic Distillation Network for Zero-Shot Learning
Shiming Chen, Ziming Hong, Guo-Sen Xie, Wenhan Yang, Qinmu Peng, Kai Wang, Jian Zhao, Xinge You
TL;DR
ZSL must transfer latent visual–attribute semantics from seen to unseen classes, but global alignment and unidirectional attention capture these relationships incompletely. MSDN uses mutually teaching attribute→visual and visual→attribute attention with semantic distillation, achieving significant gains and state-of-the-art performance across CUB, SUN, and AWA2.
Problem
ZSL needs to infer intrinsic semantic knowledge between visual and attribute features on seen classes for transfer to unseen classes, beyond global alignment or limited unidirectional attention.
Method
MSDN combines attribute→visual and visual→attribute attention sub-networks with a semantic distillation loss that makes them learn collaboratively and teach each other.
Results
MSDN achieves significant performance gains over counterparts and new state-of-the-art performance on the CUB, SUN, and AWA2 benchmarks.
Takeaways & Limitations
Mutually distilling semantic representations enables knowledge transfer between visual and attribute features for zero-shot recognition.
Takeaways & Limitations
The method assumes class semantic vectors are available for seen and unseen classes and that training data are drawn from seen classes.
Abstract
from arXiv · showhide
The key challenge of zero-shot learning (ZSL) is how to infer the latent semantic knowledge between visual and attribute features on seen classes, and thus achieving a desirable knowledge transfer to unseen classes. Prior works either simply align the global features of an image with its associated class semantic vector or utilize unidirectional attention to learn the limited latent semantic representations, which could not effectively discover the intrinsic semantic knowledge e.g., attribute semantics) between visual and attribute features. To solve the above dilemma, we propose a Mutually Semantic Distillation Network (MSDN), which progressively distills the intrinsic semantic representations between visual and attribute features for ZSL. MSDN incorporates an attribute$\rightarrow$visual attention sub-net that learns attribute-based visual features, and a visual$\rightarrow$attribute attention sub-net that learns visual-based attribute features. By further introducing a semantic distillation loss, the two mutual attention sub-nets are capable of learning collaboratively and teaching each other throughout the training process. The proposed MSDN yields significant improvements over the strong baselines, leading to new state-of-the-art performances on three popular challenging benchmarks, i.e., CUB, SUN, and AWA2. Our codes have been available at: \url{https://github.com/shiming-chen/MSDN}.
1. Introduction
ZSL transfers semantic knowledge from seen to unseen classes, but prior global-alignment and unidirectional-attention methods inadequately capture intrinsic visual–attribute semantics. MSDN addresses this with mutually teaching attention sub-networks and semantic distillation.
- ZSL transfers semantic knowledge from seen classes to unseen classes using shared semantic descriptions such as attributes.
- Prior methods either align global image features with class semantics or use unidirectional attention that captures limited visual–attribute alignments.
- MSDN combines attribute→visual attention for attribute-based visual features with visual→attribute attention for visual-based attribute features.
- The two attention sub-networks teach each other during training to distill intrinsic semantic representations for transfer from seen to unseen classes.
- MSDN achieves significant gains over counterparts on the CUB, SUN, and AWA2 benchmarks.
2. Related Work
Prior ZSL work includes embedding, generative, and common-space approaches, while MSDN applies mutual semantic distillation through two attention sub-networks. Its design is motivated by the need to transfer richer intrinsic semantic information between visual and attribute domains.
- Embedding-based ZSL maps visual features into semantic space but can overfit seen classes under generalized ZSL.
- MSDN’s two sub-networks learn complementary representations that calibrate each other to discover intrinsic visual–attribute semantics.
- Generative ZSL learns semantic→visual mappings to generate unseen-class visual features and convert ZSL into conventional classification.
- Knowledge distillation transfers information by matching teacher and student class probabilities or feature representations beyond conventional supervised targets.
- MSDN adapts teacher–student distillation into mutual learning between attribute→visual and visual→attribute attention sub-networks.
3. Mutually Semantic Distillation Network
MSDN learns intrinsic semantic representations through two complementary attention sub-nets that map between visual regions and semantic attributes. A semantic distillation loss aligns their predictions so they learn collaboratively for knowledge transfer from seen to unseen classes.
- Attribute→Visual Attention Sub-net: MSDN uses an attribute→visual attention sub-net to localize image regions relevant to each semantic attribute.It produces attribute-based visual features from regional visual features and semantic attribute vectors.
- Attribute→Visual Attention Sub-net: Each attribute-based visual feature captures visual evidence for its corresponding attribute and is mapped into semantic space as an attribute score.The resulting embedding contains one score for each attribute, representing confidence that the attribute appears in the image.
- Visual→Attribute Attention Sub-net: The visual→attribute attention sub-net attends to semantic attributes for each image region to learn complementary visual-based attribute representations.These representations are designed to align with the attribute-based visual features and calibrate them.
- Visual→Attribute Attention Sub-net: MSDN maps visual-based attribute features into semantic space and further projects them into the K-dimensional semantic attribute space.The projection uses a learnable matrix to match the dimensionality of class semantic vectors.
- Model Optimization: Both attention sub-nets use attribute-based cross-entropy with self-calibration, while semantic distillation aligns their class posterior probabilities for mutual learning.The final prediction fuses the embeddings from both sub-nets with combination coefficients and explicit calibration.
4. Experiments
MSDN is evaluated on three ZSL benchmarks using conventional and generalized settings, component ablations, and qualitative visualizations. Results show strong performance from mutually distilled attention sub-nets, with complementary representations and carefully balanced combination coefficients improving transfer.
- Conventional Zero-Shot Learning: MSDN achieves 76.1% and 65.8% top-1 accuracy on CUB and SUN in conventional zero-shot learning.The method also obtains 70.1% on the coarse-grained AWA2 dataset.
- Generalized Zero-Shot Learning: MSDN reaches 68.1% and 67.7% harmonic mean on CUB and AWA2 in generalized zero-shot learning.It also improves harmonic mean by at least 3.7% over attention-based methods on SUN.
- Ablation Studies: Single attention sub-nets improve over the baseline, while semantic distillation further raises harmonic mean by 5.4% on CUB and 4.8% on SUN.The ablations compare visual→attribute and attribute→visual attention with and without semantic distillation.
- Ablation Studies: The full model improves acc/Harmonic mean over the baseline by 18.7%/19.0% on CUB and 11.0%/10.8% on SUN.Using only JSD or ℓ2 limits the distillation capacity, whereas combining complementary embeddings improves feature representations.
- Qualitative Results: Attention maps show complementary attribute localizations, while t-SNE visualizations compare baseline and attention-based features across seen and unseen classes.The two sub-nets can identify different semantic attributes, such as “under tail white” and “shape chicken-like marsh.”
- Qualitative Results: MSDN performs best with combination coefficients (α1, α2) of (0.9,0.1) on CUB and (0.7,0.3) on SUN.Performance decreases when either coefficient is set too small or too large because the two feature types are complementary.
5. Conclusion and Discussion
MSDN distills semantic representations mutually between visual and attribute features for zero-shot learning. Its attention sub-networks learn collaboratively through semantic distillation, with experiments showing superiority on three benchmarks.
- MSDN uses attribute→visual and visual→attribute attention sub-networks to learn attribute-based visual and visual-based attribute features.
- A semantic distillation loss aligns the sub-networks’ class posterior probabilities to encourage mutual learning.
- MSDN distills intrinsic semantic representations between visual and attribute features for effective zero-shot knowledge transfer.
- Experiments on CUB, SUN, and AWA2 show MSDN’s superiority over strong baselines.