Source-linked AI summary
Transferable Contrastive Network for Generalized Zero-Shot Learning
Huajie Jiang, Ruiping Wang, Shiguang Shan, Xilin Chen
TL;DR
GZSL must recognize source and target classes, but existing methods often overfit source classes and learn little about targets. TCN explicitly transfers knowledge from source images to similar target classes through contrastive learning, and experiments on five benchmark datasets show superiority for GZSL, with class-similarity quality limiting performance on coarse-grained data such as APY.
Problem
Existing GZSL approaches often overfit source classes because they ignore target classes during learning, despite target images lacking labeled training data.
Method
TCN encodes images and class semantics, contrasts their fused representations, and transfers knowledge from source images to similar target classes.
Results
Experiments on five benchmark datasets show the superiority of TCN for GZSL.
Takeaways & Limitations
Explicit knowledge transfer can address source-class overfitting and improve the model's ability to learn knowledge about target classes.
Takeaways & Limitations
TCN depends on reasonable class similarities; rough similarities in coarse-grained APY can degrade knowledge transfer and performance.
Abstract
from arXiv · showhide
Zero-shot learning (ZSL) is a challenging problem that aims to recognize the target categories without seen data, where semantic information is leveraged to transfer knowledge from some source classes. Although ZSL has made great progress in recent years, most existing approaches are easy to overfit the sources classes in generalized zero-shot learning (GZSL) task, which indicates that they learn little knowledge about target classes. To tackle such problem, we propose a novel Transferable Contrastive Network (TCN) that explicitly transfers knowledge from the source classes to the target classes. It automatically contrasts one image with different classes to judge whether they are consistent or not. By exploiting the class similarities to make knowledge transfer from source images to similar target classes, our approach is more robust to recognize the target images. Experiments on five benchmark datasets show the superiority of our approach for GZSL.
1. Introduction
The paper frames GZSL as a response to source-class overfitting when test images may come from either source or target classes. TCN addresses this by explicitly transferring knowledge from source images to similar target classes while combining discriminative and transferable contrastive properties.
- GZSL allows test samples to come from either source or target classes, unlike the stronger target-only assumption in traditional ZSL.
- Existing approaches often overfit source classes because they ignore target classes during model learning and face projection domain shift.
- TCN learns target classifiers from similar source images, such as using horse images to learn the target class zebra.
- TCN explicitly contrasts images with class semantics and transfers knowledge from source classes to similar target classes.
- The method combines discriminative learning to distinguish classes with transferable learning to improve robustness on target classes.
- Experiments on five benchmark datasets are reported as showing the superiority of the proposed approach.
2. Related Work
The related work describes semantic information, visual-semantic transformations, recognition strategies, and methods designed to reduce source-class overfitting. TCN is positioned as a latent-space approach that fuses image and semantic information for contrastive learning rather than relying only on distance minimization.
- Semantic Information: Attributes and word vectors are the main semantic representations used to relate source and target classes for knowledge transfer.Attributes are described as accurate but labor-intensive, whereas word vectors reduce annotation effort but contain textual noise.
- Visual-Semantic Transformations: Visual-semantic transformation methods are grouped into visual-to-semantic, semantic-to-visual, and latent-space embeddings.
- Visual-Semantic Transformations: Latent-space methods encode visual and semantic representations into a shared space for image recognition.
- Visual-Semantic Transformations: TCN belongs to latent-space embedding but fuses image and semantic information for contrastive learning instead of only minimizing their latent-space distance.
- Zero-Shot Recognition: Zero-shot recognition commonly uses distance-based nearest-neighbor methods or classifier-based methods, while TCN uses contrastive values.
- Zero-Shot Recognition: Generative approaches address source-class overfitting by generating target-class image features, whereas TCN is described as easier to learn and complementary to them.
3. Approach
TCN learns to contrast fused image and class-semantic representations while explicitly transferring knowledge from source images to similar target classes. Its loss combines discriminative learning on source classes with a transferable property, and recognition selects the class with the largest contrastive value.
- Information Fusion: TCN fuses image and class-semantic information, then learns to judge whether each fusion is consistent rather than relying on a fixed distance.A CNN encodes images, an MLP encodes class semantics, and element-wise multiplication produces the fused feature.
- Contrastive Learning: The contrastive network computes a learned contrastive value for each image–class pair from the fused feature.The function is designed to assess consistency between an image and a specific class.
- Contrastive Learning: TCN combines discriminative and transferable properties so source supervision distinguishes classes while the model generalizes toward target classes.The full loss is L = LD + αLT, where α controls the relative importance of the two properties.
- Class Similarity: To reduce source-class overfitting, TCN transfers knowledge from source images to similar target classes despite having no labeled target images.Source–target similarities are estimated by reconstructing each source-class semantic representation from target classes; reconstruction coefficients represent similarity.
- Recognition: Zero-shot recognition assigns an image to the target class with the largest contrastive value, while GZSL compares all source and target classes.The framework defines separate decision rules for ZSL and GZSL over their respective candidate class sets.
4. Experiment
Experiments across five benchmark datasets evaluate TCN for ZSL and GZSL, including comparisons with state-of-the-art methods, baseline ablations, and visual analyses of knowledge transfer. Results indicate strong transfer to target classes, while performance depends on the quality of class similarities.
- Datasets and Settings: Experiments use APY, AWA1, AWA2, CUB, and SUN, with image features from ResNet101 and class attributes as semantic information.The semantic transformation and contrastive learning components are implemented with fully connected neural networks.
- Performance on ZSL and GZSL: TCN achieves the best zero-shot performance on three datasets and is comparable to the best approach on SUN.The paper attributes its fine-grained CUB performance to contrastive discrimination and effective transfer between similar classes, while APY performance is lower because source-target relations are weak.
- Performance on ZSL and GZSL: GZSL evaluation measures target-class accuracy ts, source-class accuracy tr, and harmonic mean H across APY, AWA1, AWA2, CUB, and SUN.The evaluation addresses settings where test samples may come from either source or target classes.
- Importance of Knowledge Transfer: Removing the knowledge-transfer term produces only small ZSL improvements but significant GZSL improvements for TCN.The comparison supports explicit knowledge transfer as a way to address source-class overfitting and learn target-class knowledge.
- Importance of Knowledge Transfer: TCN performs best on CUB when α equals 0.01, while increasing α lowers source-class performance as the model emphasizes target classes.The source-class loss supports discrimination and the target-class loss supports transfer, so their balance affects robustness.
- Visualization of Class Similarities: Class similarities transfer source knowledge to similar target classes, such as leopard to bobcat and killer whale to blue whale, seal, walrus, and dolphin.The approach is more robust when these relations are reasonable, but rough similarities in coarse-grained APY can degrade transfer and performance.
5. Conclusion
The paper proposes TCN for GZSL, using contrastive learning and explicit transfer from source images to similar target classes. It reports that this approach addresses source-class overfitting across five benchmark datasets.
- TCN contrasts images with class semantics to judge their consistency.
- Its discriminative property supports classification across different classes.
- Its transferable property improves robustness to target classes.
- Explicitly transferring knowledge from source images to similar target classes addresses source-class overfitting in GZSL.Experiments on five benchmark datasets report superiority of the proposed approach.