Source-linked AI summary
Episode-based Prototype Generating Network for Zero-Shot Learning
Yunlong Yu, Zhong Ji, Zhongfei Zhang, Jungong Han
TL;DR
Zero-shot learning must recognize unseen classes without visual instances, but existing models can struggle to generalize across disjoint seen and unseen domains and generative methods can be unstable. The paper introduces episode-based training with a prototype-generating network that synthesizes semantic-conditioned visual prototypes, and reports state-of-the-art results on four benchmarks across traditional and generalized ZSL.
Problem
Zero-shot learning recognizes unseen classes without visual training instances, while existing approaches may not generalize across disjoint seen and unseen domains and generative methods can suffer instability and mode collapse.
Method
The method trains through episodes that mimic zero-shot tasks and uses a Prototype Generating Network with adversarial visual-semantic modeling and a parameter-economic Multi-modal Cross-Entropy Loss.
Results
The proposed approach achieves state-of-the-art performance on four benchmarks under both traditional ZSL and generalized ZSL tasks.
Takeaways & Limitations
Repeated episodic experience on mimetic unseen classes is reported to generalize to real unseen classes, while the model aligns visual-semantic consistency efficiently.
Takeaways & Limitations
Existing generative approaches still struggle in generalized ZSL because of training instability and mode collapse, motivating the proposed framework.
Abstract
from arXiv · showhide
We introduce a simple yet effective episode-based training framework for zero-shot learning (ZSL), where the learning system requires to recognize unseen classes given only the corresponding class semantics. During training, the model is trained within a collection of episodes, each of which is designed to simulate a zero-shot classification task. Through training multiple episodes, the model progressively accumulates ensemble experiences on predicting the mimetic unseen classes, which will generalize well on the real unseen classes. Based on this training framework, we propose a novel generative model that synthesizes visual prototypes conditioned on the class semantic prototypes. The proposed model aligns the visual-semantic interactions by formulating both the visual prototype generation and the class semantic inference into an adversarial framework paired with a parameter-economic Multi-modal Cross-Entropy Loss to capture the discriminative information. Extensive experiments on four datasets under both traditional ZSL and generalized ZSL tasks show that our model outperforms the state-of-the-art approaches by large margins.
1. Introduction
The paper addresses zero-shot recognition of unseen classes without visual training instances by introducing episode-based training and a prototype-generating model. It reports state-of-the-art performance on four benchmarks under traditional and generalized ZSL.
- Zero-Shot Learning recognizes unseen classes without visual instances during training, relying on class semantics instead.
- Existing visual-semantic models may fail to generalize across disjoint seen and unseen domains, while generative methods face instability and mode collapse in generalized ZSL.
- Episode-based training repeatedly splits data into class-exclusive support and refining sets, mimicking fake zero-shot tasks and accumulating experience on mimetic unseen classes.
- The Prototype Generating Network synthesizes class-level visual prototypes from semantic prototypes and combines adversarial visual-semantic modeling with an efficient Multi-modal Cross-Entropy Loss.
- Extensive experiments on four benchmarks achieve state-of-the-art performance on both traditional ZSL and generalized ZSL tasks.
2. Related Work
Prior ZSL work commonly synthesizes visual features from class-level semantics or models visual-semantic relationships, while episode-based training has been used mainly for few-shot learning. This paper distinguishes its approach through class-level prototype generation and episodes that mimic zero-shot classification.
- Generative ZSL: Generative ZSL methods synthesize visual features from class-level semantic features, sometimes using WGANs with classification networks.
- Generative ZSL: The proposed model synthesizes class-level visual prototypes from semantic prototypes without extra noise and jointly models visual prototype generation with class semantic inference.
- Generative ZSL: Unlike cycle-consistent approaches, the model uses separable bidirectional mappings integrated by a discriminator and classification network to align visual-semantic interactions.
- Episode-based learning: Episode-based training is established in few-shot learning, but few studies apply it to ZSL.
- Episode-based learning: Each proposed episode mimics a zero-shot task requiring a base visual-semantic interaction model to predict unseen classes, rather than voting across models.
3. Methodology
The methodology trains a zero-shot model through simulated episodes and uses a Prototype Generating Network to align visual and semantic modalities while synthesizing class-level visual prototypes. A multi-modal cross-entropy objective and refining stage improve discriminative classification and adaptation to unseen classes.
- Episode-based training: Each episode splits seen categories into disjoint support and refining sets, simulating a zero-shot classification task during training.The support set trains a base model, while the refining set evaluates and updates its unseen-class prediction behavior.
- Prototype Generating Network: The Prototype Generating Network synthesizes visual prototypes from class semantic prototypes and includes a semantic inference network for visual-to-semantic mapping.G maps class semantics into visual prototypes, while F projects image features into the class semantic space.
- Adversarial alignment: An adversarial framework integrates the two modality mappings using two generators and one discriminator to align visual-semantic interactions.The discriminator receives real image-feature/semantic pairs and fake synthesized-prototype/projected-semantic pairs.
- Discriminative learning: The multi-modal cross-entropy loss combines visual features, class semantics, and one-hot labels to preserve discriminative information in both spaces.Visual samples are compared with generated class visual prototypes through inner-product affinities, while semantic-space probabilities use F(x) and the semantic prototype buffer.
- Refining and prediction: During refining, G is updated using predicted class distributions on refining instances, with Euclidean distance reported because it mostly outperforms cosine distance.At test time, instances are classified by searching for the nearest generated visual prototype using a predefined distance metric.
4. Experiments
Experiments evaluate E-PGN on four datasets under traditional and generalized ZSL, comparing it with existing methods and analyzing its episode strategy, loss components, and distance metric. E-PGN improves both traditional and generalized ZSL performance, while episode composition and multi-modal loss materially affect results.
- Datasets and evaluation: Experiments use four benchmarks and evaluate average per-class top-1 accuracy T for traditional ZSL, plus unseen accuracy u, seen accuracy s, and harmonic mean H for generalized ZSL.The benchmarks include AwA1, AwA2, CUB, and FLO.
- State-of-the-art comparison: E-PGN improves T over prior results from 71.1% to 74.4% on AwA1, 70.4% to 73.4% on AwA2, 71.1% to 72.4% on CUB, and 83.9% to 85.7% on FLO.These comparisons are reported for the four datasets in the traditional ZSL setting.
- State-of-the-art comparison: E-PGN also improves generalized-ZSL H from 63.9% to 64.6% on AwA2 and from 54.3% to 56.2% on CUB, while reaching 71.2% on AwA1 and 76.5% on FLO.The reported improvements are accompanied by gains over competing approaches across the evaluated metrics.
- State-of-the-art comparison: E-PGN improves generalized-ZSL H by raising unseen accuracy while maintaining seen accuracy at a high level, unlike approaches that trade lower seen accuracy for higher unseen accuracy.The authors relate this robustness to progressive episode training paired with the effective base model.
- Episode-based paradigm: Episode training generally outperforms PGN without episodes, while selecting ten mimetic unseen classes usually beats five and larger selections can reduce performance.The authors attribute the decline to fewer classes remaining for base-model training and poorer initialization for mimetic-unseen prediction.
- Component ablations: The full model achieves the best results on fourteen of sixteen metrics, and removing MCE substantially degrades performance on three of four datasets.The ablation indicates that the three calibration terms complement one another and that MCE contributes significantly to classification performance.
- Loss and distance analyses: MCE outperforms traditional CE on AwA1, AwA2, and CUB, performs comparably on FLO, and Euclidean distance generally outperforms Cosine distance.The authors associate MCE's advantage with encoding class semantic information in the classification module.
5. Conclusion
The paper introduces episode-based training in which episodes mimic zero-shot tasks, then combines this framework with a generative model and multi-modal cross-entropy loss. Across four benchmarks, the resulting model achieves state-of-the-art performance with large margins.
- Training paradigm: Episode-based training uses repeated fake zero-shot tasks to accumulate experience predicting mimetic unseen classes that generalizes to real unseen classes.The approach is designed to enhance model adaptability for zero-shot learning.
- Model: The proposed generative model aligns visual-semantic consistency and uses a parameter-economic multi-modal cross-entropy loss.The model is trained under the episode-based framework.
- Results: Comprehensive results on four benchmark datasets show that the model achieves new state-of-the-art performance and beats competitors by large margins.The conclusion reports this outcome without specifying individual dataset metrics.