PaperlayerView exact revision on arXiv

Source-linked AI summary

Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions

Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, Fei Sha

arXiv:1812.03664v6cs.LGcs.CV

TL;DR

Few-shot embedding transfer is task-agnostic even though different UNSEEN classification tasks require different discriminative features. The paper adapts support embeddings jointly with a set-to-set function, selecting a Transformer as FEAT, and reports state-of-the-art performance across standard and extended few-shot settings.

  • Problem

    Task-agnostic embeddings learned on SEEN classes may not highlight the most discriminative features for a specific UNSEEN target task.

  • Method

    FEAT adapts target-task support embeddings with a set-to-set function, using Transformer self-attention to produce task-specific representations.

  • Results

    FEAT achieves state-of-the-art performance on benchmarks, with superiority generalizing to cross-domain, transductive, and generalized few-shot classification.

  • Takeaways & Limitations

    Task-specific embedding spaces leverage relationships among target training instances to produce more discriminative instance representations.

  • Takeaways & Limitations

    The baseline task-agnostic formulation learns only the embedding function, leaving its representation independent of the target task.

Abstract

from arXiv · show

Learning with limited data is a key challenge for visual recognition. Many few-shot learning methods address this challenge by learning an instance embedding function from seen classes and apply the function to instances from unseen classes with limited labels. This style of transfer learning is task-agnostic: the embedding function is not learned optimally discriminative with respect to the unseen classes, where discerning among them leads to the target task. In this paper, we propose a novel approach to adapt the instance embeddings to the target classification task with a set-to-set function, yielding embeddings that are task-specific and are discriminative. We empirically investigated various instantiations of such set-to-set functions and observed the Transformer is most effective -- as it naturally satisfies key properties of our desired model. We denote this model as FEAT (few-shot embedding adaptation w/ Transformer) and validate it on both the standard few-shot classification benchmark and four extended few-shot learning settings with essential use cases, i.e., cross-domain, transductive, generalized few-shot learning, and low-shot learning. It archived consistent improvements over baseline models as well as previous methods and established the new state-of-the-art results on two benchmarks.

1. Introduction

Few-shot methods transfer embeddings learned on SEEN classes to UNSEEN tasks, but task-agnostic features may not distinguish the specific target classes. FEAT adapts support embeddings with a set-to-set function and selects a Transformer implementation, achieving strong results across extended few-shot settings.

  • Motivation: Few-shot learning uses labeled SEEN classes to recognize UNSEEN classes from only a few exemplars.The target classifier is typically built in a transferred embedding space with simple non-parametric classifiers.
  • Motivation: A common embedding space may emphasize features useful for one target distinction but irrelevant to another, such as cat–dog versus cat–tiger.Current approaches are agnostic to downstream target tasks and may de-emphasize task-specific discriminative features.
  • Proposed approach: FEAT adapts all support embeddings jointly with a set-to-set function, producing task-specific embeddings that form class prototypes for nearest-neighbor classification.The adaptation contextualizes instances within the target support set rather than applying an unchanged embedding function.
  • Proposed approach: Transformer is selected from several set-to-set approximators because it is parameter-efficient and best satisfies the desired transformation properties.The evaluated alternatives include Bi-LSTM, DeepSets, GCN, and Transformer, with contextualization, permutation invariance, interpolation, and extrapolation among the desired properties.
  • Results: FEAT achieves superior performance against strong baselines across varied extended few-shot learning tasks.The paper reports evaluations spanning multiple extended settings and state-of-the-art results on two benchmarks.

2. Related Work

Related few-shot work either constructs target classifiers through meta-learning or learns generalizable embeddings for simple classifiers. FEAT follows the embedding-based line but adapts embeddings to each target task instead of assuming SEEN-class embeddings are universally discriminative.

  • Classifier construction: Meta-learning methods optimize classifiers for rapid adaptation or directly meta-predict classifiers from new-task data.
  • Embedding-based methods: Embedding-based methods learn generalizable instance representations and use simple classifiers such as nearest-neighbor rules to reduce overfitting with few labels.
  • FEAT: FEAT differs by transforming SEEN-class embeddings for each target task so they better align with the task’s required discrimination.

3. Learning Embedding for Task-agnostic FSL

The task-agnostic FSL baseline learns an instance embedding from SEEN-class episodes and classifies UNSEEN-task test instances with nearest neighbors. Its limitation is that only the embedding function is optimized, leaving the representation independent of the target task.

  • Problem setup: Standard FSL represents a task as an M-shot N-way problem with a small support set and seeks a classifier for its UNSEEN classes.
  • Training: Additional labeled data from non-overlapping SEEN classes are used to synthesize training episodes for learning the classifier.The sampled episodes classify test instances into SEEN classes during training.
  • Training: Training uses synthesized few-shot tasks and returns the learned embedding function together with the set function.
  • Task-agnostic embedding: The baseline classifier maps each instance with an embedding function and applies nearest-neighbor classification in the resulting space.
  • Task-agnostic embedding: Only the embedding function is learned under the baseline objective, so the resulting representation is termed task-agnostic.

4. Adapting Embedding for Task-specific FSL

The proposed method transforms support embeddings jointly with a permutation-invariant set-to-set function, creating task-specific representations for nearest-neighbor classification. FEAT instantiates this transformation with self-attention and trains it with a contrastive objective that preserves category-wise similarity.

  • Adapting to task-specific embeddings: Task-specific adaptation contextualizes each support embedding using the other instances in the target set, enabling co-adaptation that instance-wise functions cannot provide.
  • Adapting to task-specific embeddings: The adapted set is permutation-invariant, and nearest neighbors are computed using the transformed support embeddings.
  • Learning the adaptation: The method can combine different task-agnostic embedding functions and similarity measures, while optimizing both the embedding and set-transformation functions on synthesized SEEN tasks.
  • Set-to-set alternatives: Bi-LSTM, DeepSets, and GCN provide alternative set-to-set transformations with different mechanisms for sequence dependence, aggregation, or relation propagation.
  • Transformer adaptation: FEAT uses Transformer self-attention to refine each instance embedding in context while naturally satisfying the desired set-transformation properties.
  • Contrastive learning: The contrastive objective encourages adapted instances to be closer to same-class centers than to other-class centers, preserving category-wise similarity.
  • Implementation: The embedding backbone uses ConvNet, ResNet, or WideResNet variants, with optional same-class averaging before set transformation.

5. Experiments

Experiments evaluate FEAT across standard and extended few-shot settings, comparing embedding-adaptation functions, interpolation and extrapolation across classification ways, parameter efficiency, and qualitative adaptation behavior. FEAT consistently performs strongly, with Transformer-based adaptation providing effective task-specific embeddings and broad generalization.

  • Standard Few-Shot Classification: FEAT outperforms instance-embedding baselines and previous methods on MiniImageNet and TieredImageNet benchmarks.On TieredImageNet, embedding-adaptation approaches improve over ProtoNet in almost all cases, with FEAT achieving the best performances among the approaches.
  • Embedding-Adaptation Models: FEAT consistently improves ProtoNet and other embedding-adaptation approaches across backbone settings, without additional bells and whistles.The Transformer models rich interactions between instances, supporting expressive embedding adaptation.
  • Way Interpolation and Extrapolation: FEAT achieves similar performance across interpolation and extrapolation to classification tasks with N={5, 10, 15, 20} classes.Deep Sets performs well in interpolation but degrades in extrapolation, whereas GCN shows the opposite pattern and Bi-LSTM performs worst in both.
  • Parameter Efficiency: FEAT introduces the fewest additional parameters among the evaluated set-to-set functions while achieving the best performances across multiple aspects.This parameter-efficiency observation holds with both ConvNet and ResNet backbones.
  • Ablation and Qualitative Analysis: FEAT improves post-adaptation embeddings in three of four qualitative tasks by separating support embeddings from clutter and better fitting test-category samples.In the negative example, adaptation pushes the “Golden Retriever” and “Lion” embeddings too close together, degrading performance.
  • Extended Few-Shot Settings: FEAT generalizes across domains, improves transductive few-shot learning, and consistently outperforms methods or baselines across three extended task settings.In domain generalization, FEAT improves real-world classification despite seeing support data from Clipart; it also outperforms previous semi-supervised methods in the transductive protocol.

6. Discussion

FEAT adapts embeddings to each target task instead of relying on a common, task-agnostic embedding space. Its task-specific representations support strong benchmark performance and generalization to several extended few-shot settings.

  • FEAT transforms instance embeddings with a set-to-set function to customize the embedding space for each target classification task.The adaptation uses self-attention to incorporate relationships among target task training instances.
  • FEAT achieves state-of-the-art performance on benchmarks and generalizes to cross-domain, transductive, and generalized few-shot classification.

A. Details of Baseline Methods

The paper describes nearest-neighbor and prototype-based few-shot baselines, then compares several set-to-set adaptation functions. FEAT uses a Transformer whose self-attention produces contextualized, permutation-invariant adapted embeddings and extends naturally to transductive FSL.

  • Details of Baseline Methods: Matching Network classifies each test instance using the most similar training instance, with cosine similarity scaled by a temperature parameter γ.The temperature is tuned carefully and strongly influences training when optimizing from pre-trained weights.
  • Details of Baseline Methods: ProtoNet averages same-class embeddings into prototypes when M > 1 and classifies test instances by similarity to the nearest class center.Pre-averaging class instances before adaptation makes class embeddings more precise and facilitates downstream adaptation.
  • Set-to-Set Transformations: The paper evaluates Bi-LSTM, DeepSets, GCN, and Transformer as alternative set-to-set embedding adaptation functions.These implementations differ in how they contextualize or propagate relationships among instances in a task set.
  • Set-to-Set Transformations: Bi-LSTM outputs depend on input order, whereas set-based adaptation seeks permutation-invariant representations.The paper applies adaptation only to the support set, yielding a fully inductive setting.
  • Transformer: FEAT uses Transformer self-attention to transform each instance embedding using contextual instances while preserving permutation invariance.The Transformer matches queries to keys, weights values by proximity, and adds the transformed input residually; its flexibility also supports transductive FSL.

C. Implementation Details

The experiments use four few-shot datasets with separate training, validation, and evaluation class splits, three image-embedding backbones, pre-training, and a shallow Transformer configuration.

  • Backbones: The implementation considers ConvNet, ResNet, and WRN backbones for instance embedding, resizing input images to 84 × 84 × 3.
  • Datasets: The study investigates MiniImageNet, TieredImageNet, CUB, and OfficeHome, splitting each dataset into non-overlapping training, validation, and evaluation classes.
  • Pre-training: The backbone is additionally pre-trained to classify SEEN classes with cross-entropy loss and image augmentations.Random crop, color jittering, and random flipping are used during this stage.
  • Transformer Configuration: FEAT uses hidden dimension d′ = 64 with ConvNet and 640 with ResNet or WRN, dropout 0.5, and a shallow one-layer Transformer.The shallow configuration gives the best overall empirical performance.
  • Optimization: Optimization uses Adam for ConvNet and Nesterov-accelerated stochastic gradient descent for ResNet and WRN.The initial learning rates are 0.002 for ConvNet and 0.001 for ResNet/WRN, with SGD weight decay 5e-4 and momentum 0.9.

D.1. Main Results

FEAT performs strongly across the paper’s main few-shot benchmarks and backbones. It ranks best among evaluated methods on MiniImageNet, remains superior on TieredImageNet 1-shot tasks, and achieves top-tier CUB results.

  • Main Results: FEAT achieves the best performance among popular methods and baselines on MiniImageNet.
  • Main Results: FEAT achieves higher results than current state-of-the-art approaches on MiniImageNet with a WRN backbone and retains superiority on TieredImageNet 1-shot tasks.
  • Main Results: On CUB, embedding adaptation assists few-shot classification, DeepSets outperforms Bi-LSTM, and Transformer-based FEAT obtains top-tier results.The reported comparison covers 5-way 1-shot and 5-shot classification with a ConvNet backbone.

D.2. Ablation Studies

Ablations examine FEAT’s embedding adaptation, set-function behavior, prototype construction, Transformer capacity, prediction metric, and contrastive-loss weighting. The analyses support task-specific adapted embeddings and a shallow Transformer design.

  • FEAT’s standard results are reported across MiniImageNet, TieredImageNet, and CUB using multiple backbones and 10,000 test trials for implementation methods.The cited tables report mean accuracy with 95% confidence intervals.
  • Task-specific embeddings after adaptation improve few-shot classification over pre-adapted, task-agnostic embeddings.
  • FEAT is evaluated for interpolation and extrapolation across 5-, 10-, 15-, and 20-way tasks after training on fixed-way tasks.The evaluation varies the number of classes while training uses 5-shot 20-way or 5-way tasks.
  • Ablations separately examine pre- versus post-averaging, Transformer heads and layers, contrastive-loss weighting, and cosine versus Euclidean prediction.The studies vary prototype placement, Transformer capacity, λ, and similarity measure.

D.3. Few-Shot Domain Generalization

FEAT is evaluated when training and testing involve different visual domains, including classification of real images from support examples drawn from sketches. It also uses unlabeled test instances transductively to adapt embeddings jointly.

  • FEAT generalizes across domains by adapting embeddings when support and test instances have different visual appearances.The cross-domain setup trains on Clipart and evaluates on Clipart or Real World instances.
  • FEAT improves Real World few-shot classification even when support data come only from Clipart.ProtoNet improves over supervised features on Clipart but not on Real World in the described comparison.
  • In transductive FSL, FEAT† augments Transformer adaptation with unlabeled test instances while retaining labeled-instance class centers.FEAT‡ additionally uses adapted unlabeled instances to construct class prototypes.
  • Using more unlabeled test instances further improves FEAT† over standard FEAT, while FEAT‡ gains more from improved class-center estimation.The transductive gain is larger for one-shot than five-shot learning because unlabeled instances are less helpful with more labeled data.

D.5. More Generalized FSL Results

The paper extends FEAT evaluation to generalized few-shot learning and large-scale low-shot learning, covering both seen and unseen classes. Results are reported across mixed class compositions and ImageNet protocols.

  • Generalized FEAT evaluates 1-shot and 5-shot performance on SEEN, UNSEEN, and COMBINED class instances.The evaluation uses a ConvNet backbone on MiniImageNet.
  • In the 5-shot generalized setting, performance improvement mainly comes from UNSEEN tasks.
  • The low-shot evaluation reports top-5 accuracy over all classes on the large-scale ImageNet dataset.
  • Large-scale low-shot learning evaluates few-shot classification across 389 SEEN and 611 UNSEEN ImageNet classes.The setup follows prior-work splits and uses features extracted with a pre-trained ResNet-50.
Loading 1812.03664v6…