Source-linked AI summary
Multimodal Prototypical Networks for Few-shot Learning
Frederik Pahde, Mihai Puscas, Tassilo Klein, Moin Nabi
TL;DR
Few-shot learning is challenging because novel classes have very limited labeled data, a setting in which finetuning is susceptible to overfitting. The paper generates visual-space features from text to enrich class prototypes, then classifies using visual data at test time. It reports outperforming state-of-the-art single-modal and multimodal few-shot methods on CUB-200 and Oxford-102.
Problem
Few-shot learning must classify novel classes from very limited samples, a setting where finetuning is susceptible to overfitting.
Source: §1 · p. 1Method
A text-conditional GAN maps textual descriptions into visual feature representations to enrich prototypes, with nearest-neighbor classification at test time using visual data only.
Source: §3.2 · p. 3, §3.1 · p. 3, §3.3 · p. 4Results
Takeaways & Limitations
The reported results support nearest-neighbor classification in a visual representation space enriched with cross-modal generated features for these two datasets.
Source: §3.2 · p. 3, §6 · p. 8Training uses multimodal data, but testing is single-modal and uses images from novel classes.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
Although providing exceptional results for many computer vision tasks, state-of-the-art deep learning algorithms catastrophically struggle in low data scenarios. However, if data in additional modalities exist (e.g. text) this can compensate for the lack of data and improve the classification results. To overcome this data scarcity, we design a cross-modal feature generation framework capable of enriching the low populated embedding space in few-shot scenarios, leveraging data from the auxiliary modality. Specifically, we train a generative model that maps text data into the visual feature space to obtain more reliable prototypes. This allows to exploit data from additional modalities (e.g. text) during training while the ultimate task at test time remains classification with exclusively visual data. We show that in such cases nearest neighbor classification is a viable approach and outperform state-of-the-art single-modal and multimodal few-shot learning methods on the CUB-200 and Oxford-102 datasets.
1. Introduction
Few-shot learning addresses the overfitting and accuracy problems that arise when novel concepts have very limited training data. This work uses text-conditioned feature generation to improve visual prototypes and reports gains over prior few-shot methods on two datasets.
1. Introduction
- Few-shot learning studies learning new concepts from very limited training data, where finetuning can overfit and limited samples can yield poor accuracy.The introduction contrasts this with deep learning’s usual reliance on large training sets.Source: §1 · p. 1
- Prior feature-generation approaches can produce useful hallucinated samples, but their diversity is bounded by the samples used to train the generator.The paper proposes multimodal data as a way to broaden generated-sample diversity.Source: §1 · p. 2
- A text-conditional GAN maps textual descriptions into visual feature representations, shifting visual prototypes toward multimodal prototypes for novel classes.The proposed nearest-neighbor inference uses visual data only.Source: §1 · p. 2
- The method combines real and generated features for nearest-neighbor classification and outperforms baselines and state-of-the-art multimodal and image-only few-shot methods by a large margin on CUB-200 and Oxford-102.Source: §1 · p. 2
2. Related Work
Related work spans few-shot methods that address limited data through learning strategies, feature generation, or additional data, alongside multimodal methods that use text for visual tasks.
2.1. Few-Shot Learning
- Few-shot research includes verification, regularization, additional related data, feature hallucination, meta-learning, and attention-based classification.Matching networks use attention over labeled-example embeddings [39], while prototypical networks classify using class prototypes [37].Source: §2.1 · pp. 2–3
- Prototypical networks learn an embedding where each class is represented by its sample mean, then classify by the nearest prototype [37].Other few-shot approaches include meta-learning, attention, and strategies aimed at limiting overfitting.Source: §2.1 · pp. 2–3
2.2. Multimodal Learning
- Multimodal work includes text-image alignment, text-conditioned image or feature generation, and zero-shot classification using textual descriptions.In few-shot learning, Pahde et al. [26] generate additional training images from text and use self-paced sample selection.Source: §2.2 · p. 3
3. Multimodal Prototypical Networks
The method uses text available during training to enrich sparse novel-class visual representations, then classifies visual test samples by their nearest multimodal prototype. A text-conditional GAN maps descriptions into a pretrained visual embedding space, and generated features are combined with real visual features.
3.1. Preliminaries
- Novel-class training includes paired images, descriptions, and labels, while testing uses only images from novel classes.The label space is divided into base classes with sufficient data and underrepresented novel classes.Source: §3.1 · p. 3
3.2. Nearest Neighbor in Visual Embedding Space
- Nearest-neighbor classification assigns each visual test sample to the class of its closest prototype in a pretrained discriminative visual embedding space.With few novel-class samples, the visual prototypes may only roughly approximate the true class means.Source: §3.2 · pp. 3–4
3.3. Cross-modal Feature Generation
- A text-conditional GAN maps descriptions into the pretrained visual feature space to generate additional features that compensate for sparse novel-class training data.Its discriminator uses reconstruction and auxiliary classification losses, and adversarial optimization focuses the generator on class-discriminative features.Source: §3.3 · pp. 4–5
3.4. Multimodal Prototype
- The method combines real visual and text-generated feature prototypes using a weighted average, then classifies visual test samples with the updated prototypes.The weighting factor controls the combination, and generated features can be produced repeatedly.Source: §3.4 · p. 5
4. Experiments
Experiments evaluate cross-modal few-shot classification on CUB-200 and Oxford-102 against image-only and multimodal baselines. The method outperforms the compared approaches across shot settings, while generated features improve prototype-based classification over a strong image-only baseline.
4.1. Datasets
- CUB-200 uses 150 base and 50 novel classes for 50-way evaluation, while Oxford-102 uses 82 base and 20 novel classes for 20-way evaluation, with 1–20 novel-class images.Both datasets provide ten short textual descriptions per image.Source: §4.1 · p. 5
4.2. Implementation Details
- A modified ResNet-18 produces visual embeddings, and a text-conditional GAN maps text embeddings into 256-dimensional visual features averaged across ten descriptions per image.The image encoder is trained on base classes; the GAN uses conditioning augmentation and an auxiliary classification loss.Source: §4.2 · pp. 5–6
- Classification uses cosine nearest-prototype prediction, averaging visual and text-derived prototypes with λ = 1 and updating the generator over ten iterations.The evaluation also includes image-only prototypes, text-only generated prototypes in a zero-shot setting, and the combined multimodal prototype.Source: §4.2 · p. 6, §4.3 · p. 6
4.3. Results
- The comparisons use 600 randomly sampled few-shot episodes and report average top-5 accuracy with 95% confidence intervals.The multimodal comparison includes the approach in, identified as the only existing multimodal few-shot method to the authors’ knowledge.Source: §4.3 · p. 6
- Across every n-shot scenario, the method outperforms the strong baselines and the other multimodal few-shot approach; gains are larger on CUB-200 than Oxford-102, where it still beats state of the art.The authors also report narrower confidence intervals than the baselines.Source: §4.3 · pp. 6–7
4.4. Comparison to Single-modal Methods
- The image-only baseline already reaches state-of-the-art accuracy, while text-conditioned features improve results in both 1- and 5-shot settings and shift prototypes toward more correctly classified test samples.Figure 3 visualizes prototypes and unseen-image retrieval in a 5-way, 1-shot CUB-200 setting.Source: §4.4 · p. 7, Figure 3 · p. 7
5. Analysis
The analysis tests how text availability and multimodal prototype shifts relate to classification accuracy. More text helps most when image support is scarce, while larger prototype shifts accompany larger class-level gains.
5.1. Reducing Textual Data
- Accuracy gains from text are larger with fewer images, increase as more descriptions become available, and are never harmed by adding text.The second description gives the largest gain in each n-shot setting.Source: §5.1 · p. 8, Figure 4 · p. 8
5.2. Impact of Prototype Shift
- Classes with more-shifted prototypes tend to gain more accuracy; the most-shifted prototype yields an average per-class top-5 accuracy gain of about 15%.On average, multimodal adjustment is not harmful, and the authors suggest generated features carry complementary information that simulates unseen novel-class samples.Source: §5.2 · p. 8, Figure 5 · p. 8
6. Conclusion and Future Work
The paper addresses few-shot learning by enriching visual features with cross-modal generation and classifying unseen samples against multimodal prototypes. On CUB-200 and Oxford-102, it reports outperforming strong baselines and state-of-the-art single-modal and multimodal methods.
6. Conclusion and Future Work
- The method enriches the visual feature space using data from another modality and classifies unseen samples by their nearest multimodal class prototype.The authors plan to explore jointly optimizing the learned representation on multimodal data.Source: §6 · p. 8
Appendix A. Extended Quantitative Results
The appendix extends quantitative comparisons across classification metrics, datasets, and few-shot settings. It reports that the method generally improves accuracy and reduces variance, with generated features having less impact as visual data increases.
Appendix A. Extended Quantitative Results
- Top-1 and top-3 accuracies in the 50-way comparisons follow the same trends as the reported top-5 accuracy.Table 3 compares multimodal few-shot approaches and baselines on CUB-200 and Oxford-102 for n ∈ {1, 2, 5, 10, 20}.Source: Appendix A · p. 11, Table 3 · p. 11
- Across most additional k-way settings, the method improves accuracy and decreases variance, but generated features matter less when more visual data are available.The comparisons use k ∈ {5, 10, 20} and n ∈ {1, 2, 5, 10, 20}.Source: Appendix A · p. 11, Table 4 · p. 12
Appendix B. Visualization of Embedding Space
The appendix visualizes few-shot embedding spaces with t-SNE and shows how multimodal prototypes are positioned relative to class clusters and unseen samples.
Appendix B. Visualization of Embedding Space
- Across k-way visualizations, multimodal prototypes often move toward their class-cluster centers relative to image-only prototypes.The plots use colors for class membership and show prototypes as triangles or crosses, with unseen test samples as dots.Source: Appendix B · p. 12, Figure 6 · p. 13, Figure 8 · p. 13
Appendix C. Retrieval Results
Text-conditioned generated feature vectors retrieve mostly same-class unseen test images in a 5-way 1-shot setting, supporting nearest-neighbor classification. Figure 10 shows the captions, retrieved images, and corresponding training images.
Appendix C. Retrieval Results
- The 10-way and 50-way t-SNE figures show image-only prototypes, updated multimodal prototypes, and unseen test samples.Source: Figure 7 · p. 13, Figure 9 · p. 13
- Most retrieved unseen test images are from the correct class when using text-conditioned generated features in 5-way 1-shot learning.The authors report that these features are close to unseen same-class images, facilitating nearest-neighbor classification.Source: Appendix C · p. 13, Figure 10 · p. 14
- Retrieval uses cosine distance between generated feature vectors conditioned on training-image textual descriptions.Source: Appendix C · p. 13
- Figure 10 pairs one caption per training image with the top-five retrieved unseen test images and the corresponding training image; box colors mark correct versus wrong classes.Source: Figure 10 · p. 14, Appendix C · p. 14