PaperlayerView exact revision on arXiv

Source-linked AI summary

Baby steps towards few-shot learning with multiple semantics

Eli Schwartz, Leonid Karlinsky, Rogerio Feris, Raja Giryes, Alex M. Bronstein

arXiv:1906.01905v2cs.CV

TL;DR

Few-shot learning from a few visual examples remains challenging, while the verbal descriptions accompanying new objects have received less attention. The paper combines multiple, richer semantic cues in a few-shot approach and reports improved 1-shot results on miniImageNet and CUB.

  • Problem

    Few-shot methods still fall short of fully supervised learning, and the semantic descriptions accompanying visual examples have received comparatively little attention in few-shot research.

    Source: §1 · pp. 1–2

  • Method

    The paper proposes a multiple-semantics few-shot setting and multi-branch architecture using category labels, attributes, and natural-language descriptions alongside images.

    Source: §1 · p. 2

  • Results

    In 1-shot, the model improves the best previously reported result by 2.3% on miniImageNet and 5.2% on CUB.

    Source: §4.2 · p. 5

  • Takeaways & Limitations

    Semantic information also improves over visual-only features in 5-shot, although its importance decreases as more examples become available.

    Source: §4.2 · p. 5

    The experiments touch only a small part of the possible approaches for combining descriptive and multiple semantics.

    Source: §5 · p. 6

Abstract

from arXiv · show

Learning from one or few visual examples is one of the key capabilities of humans since early infancy, but is still a significant challenge for modern AI systems. While considerable progress has been achieved in few-shot learning from a few image examples, much less attention has been given to the verbal descriptions that are usually provided to infants when they are presented with a new object. In this paper, we focus on the role of additional semantics that can significantly facilitate few-shot visual learning. Building upon recent advances in few-shot learning with additional semantic information, we demonstrate that further improvements are possible by combining multiple and richer semantics (category labels, attributes, and natural language descriptions). Using these ideas, we offer the community new results on the popular miniImageNet and CUB few-shot benchmarks, comparing favorably to the previous state-of-the-art results for both visual only and visual plus semantics-based approaches. We also performed an ablation study investigating the components and design choices of our approach.

1 Introduction

Few-shot learning remains difficult for AI, and its best methods still trail fully supervised learning. This work explores whether multiple, richer semantics can make few-shot learning more like infant learning, proposing a benchmark, protocol, and multi-branch architecture.

1 Introduction

  • Few-shot learning remains substantially below fully supervised methods trained with many examples, despite extensive research.The introduction cites ImageNet as an example of fully supervised training.Source: §1 · p. 1
  • Prior semantic approaches used category labels or prescribed attributes, while this work proposes multiple and richer semantic information.The paper identifies natural-language descriptions alongside labels and attributes as available semantics, and reports favorable comparisons on miniImageNet and CUB.Source: §1 · p. 2
  • The authors propose Few-Shot Learning with Multiple and Complex Semantics (FSL-MCS), a benchmark and protocol, and a multi-branch network architecture.The setting combines few images with multiple semantic explanations, including names, attributes, and descriptions.Source: §1 · p. 2

2 Related Work

Related work surveys metric, meta-learning, and generative or augmentation-based approaches to few-shot learning, alongside zero-shot methods that use semantics for unseen classes. It notes that semantic information has also been combined with few-shot methods.

2.1 Few-Shot Learning

  • Metric-learning methods embed examples in a learned space and classify new instances by proximity, with variants using graph neural networks or learned similarity functions.Some approaches optimize the embedding for few-shot tasks, and semantic information has been combined with metric learning.Source: §2.1 · p. 3
  • Meta-learning trains on few-shot episodes to learn strategies for adapting to new tasks, including metric-based classifiers and gradient-based approaches.The section describes Matching Networks, prototype-based methods, and gradient-based methods such as MAML.Source: §2.1 · p. 3
  • Generative and augmentation-based methods create additional examples from few-shot data or use external data to support learning.Examples include data synthesis and transformations; label and attribute semantics have also been used to train example-synthesis networks.Source: §2.1 · p. 3

2.2 Zero-Shot Learning

  • Zero-shot learning supplies semantics for unseen classes and relates semantic and visual representations through mappings or a shared embedding space.Other approaches generate visual samples from the supplied semantics for classifier training.Source: §2.2 · p. 3

3 Method

The model combines visual prototypes with attention-weighted semantic prototypes and classifies queries by their distance to the combined prototype. Training first pre-trains the CNN, then freezes its backbone while jointly training the semantic branches through episodic meta-learning.

3 Method

  • Training pre-trains the CNN on base categories, freezes its backbone, then meta-trains the full model episodically with support images, queries, and category semantics.The approach adds multiple semantic branches, cross-conditioning and cascaded merging, and intermediate branch losses.Source: §3 · p. 3, §3.1 · p. 4
  • Each category prototype combines a visual prototype with attention-weighted semantic prototypes, while branch-specific cross-entropy losses supervise visual and semantic outputs.Query probabilities are computed by applying softmax to negative distances between query features and prototypes.Source: §3 · p. 4

3.1 Implementation details

  • The experiments use DenseNet-121, 512-dimensional visual and semantic outputs, 300-unit hidden layers, scalar sigmoid attention outputs, and 0.7 dropout in semantic branches.The semantic backbone and attention networks are two-layer MLPs.Source: §3.1 · p. 4
  • miniImageNet descriptions come automatically from WordNet, while CUB category embeddings average 10 randomly sampled descriptions from [38].Training uses only the few-shot dataset's training categories.Source: §3.1 · p. 4

4 Experimental results

The experiments evaluate the multiple-semantics approach on miniImageNet and CUB, report improved prior 1-shot results, and examine semantic-branch and backbone choices.

4.2 Performance evaluation

  • The multiple-semantics model exceeds prior best 1-shot results by 2.3% on miniImageNet and 5.2% on CUB.The reported improvements are for the most challenging 1-shot scenario.Source: Table 2 · p. 5, §4.2 · p. 5

4.1 Datasets

  • miniImageNet uses 64/16/20 train/validation/test classes and label/WordNet descriptions; CUB uses labels, attributes, and descriptions across standard species splits.Testing uses 1,000 random episodes for miniImageNet and CUB.Source: §4.1 · p. 5

4.2 Performance evaluation

  • In 5-shot, semantics still improve over SimpleShot by +0.6% on miniImageNet and +1.6% on CUB, while their importance decreases as visual evidence grows.The table caption notes that visual information is more reliable in 5-shot.Source: §4.2 · p. 5, Table 2 · p. 5

Semantics Effect

  • On 1-shot miniImageNet, visual features plus labels improve 0.9% over SimpleShot; descriptions alone add only +0.3% over labels, and a label-only ensemble adds +0.2%.These comparisons are reported in the Table 3 ablation study.Source: Semantics Effect · p. 5, Table 3 · p. 6
  • Per-branch task-specific cross-entropy losses improve 1.4% over baseline in the 1-shot miniImageNet ablation.The authors describe this as evidence of the component's importance.Source: Semantics Effect · p. 5, Table 3 · p. 6
  • The final miniImageNet configuration improves 2.2% over baseline by conditioning the description branch on prior inputs; reusing semantics in extra branches does not help.Adding the self-attending description as a third branch gives +2% over baseline; conditioning with all prior inputs gives the maximal +2.2%.Source: Semantics Effect · p. 6, Table 3 · p. 6

Backbone Architecture

  • A stronger backbone substantially improves 5-shot performance but not 1-shot, where the model relies more on semantic branches.The authors report that visual features are more important in 5-shot.Source: Backbone Architecture · p. 6, Table 4 · p. 6

5 Summary & conclusions

The paper extends few-shot learning with multiple semantic explanations—names, attributes, and descriptions—alongside image examples. The authors describe their experiments as preliminary and outline several directions for combining semantic and visual information.

5 Summary & conclusions

  • The proposed approach pairs few image examples with category names, attributes, and natural-language descriptions to make few-shot learning more like the setting used by human infants.The authors note that their experiments explore only a small part of the possible approaches to combining descriptive and multiple semantics.Source: §5 · p. 6
  • Future work could use instance-level semantics, task-wide attention, nonlinear prototype combinations, or semantic-conditioned metrics for comparing prototypes and query features.The authors also suggest combining logits from semantic and visual branches and exploring alternative semantic sources and prototype-attention combinations.Source: §5 · p. 6
  • Further semantic sources could include category hierarchies and multimodal inputs such as audio, smell, touch, or taste.Source: §5 · p. 7

Related papers

Loading 1906.01905v2…