Source-linked AI summary
Contrastive Embedding for Generalized Zero-Shot Learning
Zongyan Han, Zhenyong Fu, Shuo Chen, Jian Yang
TL;DR
GZSL must recognize both seen and unseen classes despite limited labeled data and the original visual feature space’s limited discriminative ability. CE-GZSL combines feature generation with embedding, maps real and synthetic features into a new space, and uses contrastive embedding with class-wise and instance-wise supervision. Across five benchmarks, it outperformed state-of-the-art methods on three datasets and achieved competitive results on two.
Problem
GZSL includes both seen and unseen test classes, while feature-generation methods operate in an original visual feature space described as lacking discriminative ability.
Method
CE-GZSL combines feature generation with an embedding model and uses contrastive embedding to exploit class-wise and instance-wise supervision.
Results
The method outperformed state-of-the-art methods on three of five benchmark datasets and achieved competitive results on the other two.
Takeaways & Limitations
The framework provides a GZSL approach that performs final classification in a new embedding space using both real and synthetic features.
Abstract
from arXiv · showhide
Generalized zero-shot learning (GZSL) aims to recognize objects from both seen and unseen classes, when only the labeled examples from seen classes are provided. Recent feature generation methods learn a generative model that can synthesize the missing visual features of unseen classes to mitigate the data-imbalance problem in GZSL. However, the original visual feature space is suboptimal for GZSL classification since it lacks discriminative information. To tackle this issue, we propose to integrate the generation model with the embedding model, yielding a hybrid GZSL framework. The hybrid GZSL approach maps both the real and the synthetic samples produced by the generation model into an embedding space, where we perform the final GZSL classification. Specifically, we propose a contrastive embedding (CE) for our hybrid GZSL framework. The proposed contrastive embedding can leverage not only the class-wise supervision but also the instance-wise supervision, where the latter is usually neglected by existing GZSL researches. We evaluate our proposed hybrid GZSL framework with contrastive embedding, named CE-GZSL, on five benchmark datasets. The results show that our CEGZSL method can outperform the state-of-the-arts by a significant margin on three datasets. Our codes are available on https://github.com/Hanzy1996/CE-GZSL.
1. Introduction
GZSL addresses recognition of both seen and unseen classes, but existing approaches face seen-class bias or lack discriminative visual features. The proposed hybrid framework combines feature generation with embedding, while contrastive embedding adds instance-wise supervision to class-wise supervision.
- Problem: GZSL extends zero-shot learning by testing on both seen and unseen classes, intensifying the challenge of seen-class bias.Conventional ZSL tests only unseen classes, whereas GZSL includes both class types.
- Problem: Feature generation methods synthesize unseen-class training features, but their original visual feature space is considered suboptimal for GZSL classification.The paper attributes this limitation to the space’s distance from semantic information and lack of discriminative ability.
- Framework: The hybrid GZSL framework maps real seen and synthetic unseen features into a new embedding space for final classification.Classification is performed in the new embedding space rather than the original visual feature space.
- Contrastive Embedding: Contrastive embedding uses class-wise and instance-wise supervision, unlike traditional semantic embedding, which relies only on class-wise supervision.It distinguishes a positive sample or descriptor from many negatives from different classes using contrastive loss.
- Evaluation: The method was evaluated on five benchmark datasets and reportedly outperformed state-of-the-art methods on three while remaining competitive on two.These evaluations are presented as three contributions alongside the hybrid framework and contrastive embedding.
2. Related Work
Prior zero-shot learning methods embed visual samples and semantic descriptors, while newer GZSL methods generate unseen-class features for supervised classification. This paper identifies the original visual feature space as suboptimal and proposes a hybrid framework combining feature generation with embedding.
- Early zero-shot learning methods transfer recognition from seen to unseen classes through a shared semantic space containing descriptors for both class types.
- Embedding-based approaches map visual samples and semantic descriptors into a common embedding space for recognition.
- Feature-generation methods learn conditional generative models, such as VAEs and GANs, to synthesize unseen-class visual features and reduce data imbalance.
- Using real seen-class examples and synthesized unseen-class examples, these methods convert GZSL into a standard supervised classification problem.
- Generative Flows have also been introduced into zero-shot learning, achieving good performance on both generalized and conventional ZSL.
- Because the original visual feature space lacks discriminative ability, the paper integrates feature generation with an embedding model in a hybrid GZSL framework using contrastive embedding.
3. Contrastive Embedding for GZSL
The framework combines feature generation with embedding, then uses contrastive learning to exploit both instance-wise and class-wise supervision for GZSL.
- Problem definition: GZSL provides labeled training instances only from seen classes, while test instances may come from both seen and unseen classes.
- Hybrid framework: Feature generation synthesizes unseen-class features, but the original visual feature space remains poorly structured for GZSL classification.
- Hybrid framework: The hybrid framework maps real seen and synthetic unseen features into a new embedding space and performs classification there.
- Contrastive embedding: The final CE-GZSL objective replaces semantic embedding with combined instance-level and class-level contrastive losses.
- Contrastive embedding: Instance-level contrastive embedding distinguishes a same-class positive from K different-class negatives using a nonlinear projection head.
- Contrastive embedding: Class-level contrastive embedding uses comparator network F to select the correct semantic descriptor among S seen-class descriptors.
D min G,E,H,F V (G, D) + Lins
The complete method jointly trains generation, embedding, projection, and comparison components, then classifies using real seen and synthetic unseen embeddings.
- Complete framework: The method learns generator G and discriminator D, embedding function E, projection H, and comparator F as components of the hybrid framework.
- GZSL classification: Unseen features are generated from semantic descriptors and Gaussian noise, then embedded with E into the classification space.
- GZSL classification: A softmax classifier is trained on real seen samples and synthetic unseen samples after both are represented in the same embedding space.
4. Experiments
Experiments evaluate CE-GZSL on five benchmark datasets under conventional and generalized ZSL protocols, compare its components and embedding spaces, and analyze synthesis and embedding dimensions. CE-GZSL achieves strong benchmark performance, while combining class-wise and instance-wise supervision improves results.
- Comparison with SOTA: CE-GZSL achieves the best unseen-class accuracy on four datasets and the best harmonic mean on AWA2, CUB, and FLO.On CUB, it is the first state-of-the-art GZSL method to exceed 60.0 on both U and H.
- Comparison with SOTA: In conventional ZSL, CE-GZSL performs best on CUB and second best on AWA1 and FLO.On CUB, it is the only one of ten recent methods exceeding 70.0.
- Component Analysis: The new embedding space outperforms the original semantic descriptor space, and contrastive embedding outperforms semantic embedding in both spaces.These comparisons assess the hybrid framework using generation with alternative embedding models and classification spaces.
- Component Analysis: Using both instance-level and class-level supervision improves U and S, producing better H than using either supervision type alone.Either supervision component alone remains competitive with state-of-the-art GZSL methods.
- Hyper-Parameter Analysis: Performance increases with more synthesized unseen-class examples, with best results at 1,800, 2,400, 300, 600, and 100 examples for AWA1, AWA2, CUB, FLO, and SUN, respectively.The evaluation covers five datasets and measures GZSL using seen accuracy S, unseen accuracy U, and harmonic mean H.
- Hyper-Parameter Analysis: Increasing embedding dimension improves performance significantly, but CE-GZSL uses dimension 2,048 to trade off performance against computational cost.Temperature changes produce only slight variation in H across datasets.
5. Conclusion
The paper presents CE-GZSL, a hybrid framework that combines embedding and generation models for GZSL. It maps real and synthetic visual samples into an embedding space and uses contrastive embedding to exploit class-wise and instance-wise supervision.
- CE-GZSL integrates an embedding model with a generation model for generalized zero-shot learning.
- The framework maps real and synthetic visual samples into an embedding space for final supervised GZSL classification.
- Contrastive embedding uses both class-wise and instance-wise supervision, the latter usually neglected in existing GZSL research.
- CE-GZSL achieves state-of-the-art performance on three benchmark datasets and second-best performance on two.