PaperlayerView exact revision on arXiv

Source-linked AI summary

Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Difference

Shell Xu Hu, Da Li, Jan Stühmer, Minyoung Kim, Timothy M. Hospedales

arXiv:2204.07305v1cs.CVcs.LG

TL;DR

Few-shot learning seeks effective adaptation from scarce labeled examples, but practical performance depends on choices often controlled in standard evaluations. This paper studies a simple pre-training–meta-training–fine-tuning pipeline with modern architectures and external data, finding strong performance across four FSL benchmarks while identifying scope and resource limitations.

  • Problem

    Few-shot learning needs practical methods for adapting to novel categories with scarce labeled data, while the effects of source data, modern architectures, and deployment-time fine-tuning remain under-studied.

  • Method

    The paper evaluates a simple pipeline that sequentially pre-trains on external data, meta-trains with ProtoNet, and fine-tunes the feature backbone on novel-task support data.

  • Results

    The simple pipeline achieves very competitive performance across four FSL benchmarks, with external data and neural architecture identified as dominant performance factors and fine-tuning important under domain shift.

  • Takeaways & Limitations

    Using external data and modern architectures can be an easy and effective alternative to implementing sophisticated few-shot algorithms, with fine-tuning useful when target tasks differ from pre-training data.

  • Takeaways & Limitations

    The hybrid pipeline is restricted to modalities with large external datasets and may require substantial up-front compute and energy where foundation models do not already exist.

Abstract

from arXiv · show

Few-shot learning (FSL) is an important and topical problem in computer vision that has motivated extensive research into numerous methods spanning from sophisticated meta-learning methods to simple transfer learning baselines. We seek to push the limits of a simple-but-effective pipeline for more realistic and practical settings of few-shot image classification. To this end, we explore few-shot learning from the perspective of neural network architecture, as well as a three stage pipeline of network updates under different data supplies, where unsupervised external data is considered for pre-training, base categories are used to simulate few-shot tasks for meta-training, and the scarcely labelled data of an novel task is taken for fine-tuning. We investigate questions such as: (1) How pre-training on external data benefits FSL? (2) How state-of-the-art transformer architectures can be exploited? and (3) How fine-tuning mitigates domain shift? Ultimately, we show that a simple transformer-based pipeline yields surprisingly good performance on standard benchmarks such as Mini-ImageNet, CIFAR-FS, CDFSL and Meta-Dataset. Our code and demo are available at https://hushell.github.io/pmf.

1. Introduction

The paper examines how pre-training data, neural architecture, and meta-test fine-tuning affect simple few-shot learning pipelines. It proposes a practical pre-training–meta-training–fine-tuning pipeline and reports that external data, modern architectures, and adaptive fine-tuning can produce strong performance.

  • Motivation: Few-shot learning addresses settings where annotated data are scarce or annotation is prohibitively expensive.The paper frames FSL as learning new concepts from few training examples.
  • Research questions: Vision transformers can be combined with simple meta-learners such as ProtoNet, especially when supported by large-scale pre-training data.The paper contrasts this approach with commonly studied small CNN and ResNet architectures.
  • Research questions: The study investigates pre-training data, neural architecture, and meta-test fine-tuning using ProtoNet as a simple few-shot learner.These design choices are examined as practical factors influencing simple few-shot pipelines.
  • Pipeline: The pipeline consists of pre-training, meta-training, and fine-tuning, successively producing class-agnostic, generic, and task-specific feature backbones.The stages are abbreviated P>M>F.
  • Fine-tuning: Fine-tuning is presented as important for out-of-distribution tasks, with validation-based learning-rate selection improving cross-domain few-shot performance.The method addresses disagreement in prior work about whether fixed representations are sufficient at deployment.
  • Conclusion: The resulting baseline surpasses state-of-the-art performance while remaining simple and easy to implement.The paper emphasizes studying pipeline design choices rather than introducing a new few-shot algorithm.

2. Related Work

The related work spans meta-learning, transfer learning, self-supervised learning, cross-domain FSL, and practical ProtoNet-based pipelines. The paper positions its contribution as combining external-data pre-training, modern feature architectures, and support-set fine-tuning in a simple sequential procedure.

  • Few-shot learning: Few-shot learning commonly transfers knowledge from large source data to sparsely annotated target categories through meta-training and meta-testing.Tasks are often organized as K-way-N-shot episodes, although global models covering varied K and N are more practical.
  • Bigger data and architectures: Existing FSL research commonly uses standardized datasets and relatively small architectures, limiting direct assessment of source-data scale and modern transformer feature extractors.The paper explores transformers particularly in combination with foundation-model pre-training.
  • Transfer and self-supervision: Transfer learning and self-supervision pre-train on large source data to reduce the sample complexity of adapting to sparse target tasks.These approaches provide a complementary route to knowledge transfer alongside meta-learning.
  • Pipeline: The paper combines unsupervised external-data pre-training and labeled episodic meta-learning in a sequential P>M>F pipeline using one feature-extractor backbone.The backbone is subsequently adapted to individual tasks through optional fine-tuning.
  • ProtoNet: ProtoNet dynamically constructs class centroids and performs nearest-centroid classification from backbone features, using cosine distance in this work.Because prototypes can be computed for arbitrary class counts, ProtoNet supports various-way-various-shot settings.
  • Fine-tuning: For unseen domains, the method fine-tunes the entire feature backbone using augmented support examples rather than reusing prototypes on the same support set.The support set is the only labeled data available at meta-test time.
  • Fine-tuning: Fine-tuning performance is sensitive to learning rate, motivating automated selection because standard FSL tasks do not provide a validation set for each task.Prior approaches choose the learning rate a priori for every task.

4. Experiments

The experiments evaluate how pre-training, architecture, and fine-tuning affect few-shot classification across standard benchmarks using a simple ProtoNet-centered pipeline.

  • Datasets: Meta-Dataset, miniImageNet, and CIFAR-FS provide the main evaluation benchmarks.Meta-Dataset covers diverse domains; miniImageNet and CIFAR-FS use standard class splits.
  • Evaluation: Performance is measured as average classification accuracy over 600 test episodes or tasks per dataset.miniImageNet and CIFAR-FS use 5-way-1-shot and 5-way-5-shot episodes.
  • Experimental design: Table 1 organizes downstream results by architecture, pre-training algorithm and dataset, and meta-training algorithm.Meta-Dataset scores are averaged across target datasets, whereas miniImageNet and CIFAR-FS results use 5-way-5-shot evaluation.
  • Research questions: The study frames these experiments around three questions: pre-training’s effect, ViT adaptation, and fine-tuning during meta-testing.The broader pipeline-design analysis is introduced as a sequence of questions about practical few-shot learner construction.
  • Experimental design: The experiments vary pre-training regime and neural architecture while using ProtoNet as the standard nearest-centroid meta-test classifier.Results compare episodically trained ProtoNet with configurations lacking a meta-learning step.

1 How does pre-training regime affect FSL?

ImageNet1K pre-training generally improves few-shot performance, and strong unsupervised DINO pre-training can approach supervised pre-training without source labels.

  • Pre-training impact: ImageNet1K pre-training generally provides significant improvement across the evaluated few-shot settings.The comparison is against conventional pipelines without pre-training.
  • Pre-training impact: DINO’s state-of-the-art unsupervised pre-training performs close to supervised pre-training.Supervised pre-training is treated as an unfair upper bound.
  • Unsupervised pre-training: Strong few-shot performance can be achieved without source labels under unsupervised pre-training.The paper notes this despite semantic overlap between some source and target datasets.
  • Unsupervised pre-training: Nearest-centroid classification on frozen pre-trained features performs well with a strong pre-training regime such as DINO.Off-the-shelf foundation-model features can compare favorably with dataset-specific meta-learning.

2 Can state of the art architectures such as ViT be adapted to FSL?

State-of-the-art architectures and stronger pre-trained features substantially improve few-shot learning, while fine-tuning particularly benefits held-out and out-of-distribution domains.

  • Architecture: ViT performs excellently with large pre-training data but trains poorly on smaller meta-train benchmarks without that initialization.With pre-training, ViT outperforms ResNet18 and the ResNet50 baseline across the evaluated settings.
  • Foundation models: Larger pre-training sources and recent architectures make a substantial difference to downstream few-shot performance.The strongest results come from large foundation models such as ViT-base trained on ImageNet21K or YFCC.
  • Few-shot learners: With stronger features, ProtoNet outperforms MetaQDA and MetaOptNet despite those learners improving over direct feature transfer.This suggests comparative conclusions about meta-learners may change in the stronger-feature regime.
  • Scope and assumptions: Class overlap between unsupervised pre-training and meta-testing is likely in practical large-scale experiments.The paper argues this overlap does not hinder benchmarking the ability to construct classifiers from few labels.
  • Fine-tuning: Held-out datasets and domains very different from ImageNet benefit from fine-tuning during meta-testing.Examples include Signs, COCO, Omniglot, and QuickDraw.

3 How to best exploit fine-tuning for meta-testing?

The paper compares simple pipelines across standard and cross-domain benchmarks, emphasizing external pre-training, modern backbones, and fine-tuning. Adaptive fine-tuning further improves the pipeline, especially when target tasks differ from training data.

  • Pipeline design: P>M>F combines pre-training, ProtoNet meta-training, and fine-tuning in a simple sequential pipeline.The pipeline can use different pre-training algorithms and backbone architectures.
  • Benchmark results: The framework outperforms state-of-the-art methods under both within-domain and cross-domain conditions despite being simpler than some competitors.The comparison covers mini-ImageNet, CIFAR-FS, Meta-Dataset, and Broader Study CDFSL.
  • Cross-domain results: The hybrid pipeline outperforms state-of-the-art pure external self-supervision on CDFSL.This result is reported for the cross-domain CDFSL benchmark.
  • Fine-tuning: Adaptive fine-tuning increases the performance margin over sophisticated few-shot learners.The authors recommend fine-tuning when target tasks are less similar to pre-training and meta-training data.

5. Conclusions

The conclusions identify source data and neural architecture as dominant factors in few-shot performance, while fine-tuning with augmentation is important under domain shift. The resulting simple pipelines perform competitively across four benchmarks.

  • Conclusions: Source dataset and neural architecture are dominant factors in few-shot learning performance.The conclusion frames these factors as central to pushing a simple pre-train + ProtoNet pipeline.
  • Conclusions: Fine-tuning the feature backbone with data augmentation is important when training and testing domains differ.This conclusion concerns domain-shifted few-shot tasks.
  • Conclusions: The simple pipelines achieve very competitive performance in four few-shot learning benchmarks.The paper presents this as an empirical verification of the proposed pipeline design.
  • Limitations and future work: The hybrid pipeline is restricted to modalities with large external datasets and can require substantial compute and energy investment.The limitation is especially relevant where pre-trained foundation models do not already exist.
  • Limitations and future work: Adaptive fine-tuning is computationally expensive at meta-test time and may be unsupported on embedded platforms without backpropagation.The authors identify feed-forward representation adaptation as a possible direction for future work.

Supplemental Material

The supplemental material provides additional experimental results, fine-tuning hyperparameter analyses, and visualization of representations before and after ProtoNet meta-training.

  • Supplemental contents: The supplemental material is dated 15 Apr 2022 and identifies the manuscript as arXiv:2204.07305v1.This is bibliographic metadata rather than an experimental result.
  • Supplemental contents: The supplement includes additional results for Table 1 and Table 4 of the main paper.These sections expand the reported benchmark analyses.
  • Supplemental contents: The supplement investigates the impact of hyperparameters in the fine-tuning phase.This analysis focuses specifically on fine-tuning behavior.
  • Supplemental contents: The supplement shows T-SNE plots before and after ProtoNet meta-training.The visualizations provide a representation-level comparison across training stages.

1. Additional results for Meta-Dataset

The additional Meta-Dataset results compare pre-training methods and backbone architectures while holding ProtoNet meta-training fixed. The supplement defines the P > M > F notation and expands the evaluated pre-training variants.

  • Additional results for Meta-Dataset: The additional Meta-Dataset results cover different pre-training methods, ImageNet-domain meta-training, and meta-training across eight specified domains.These results correspond to Tables 1, 2, and 3 of the main paper.
  • Notation: P > M > F denotes pre-training, meta-training, and fine-tuning, respectively.The notation names the sequential stages of the pipeline.
  • Experimental setup: The additional analysis examines pre-training and backbone architecture while keeping ProtoNet meta-training fixed.For example, DINO > PN (ViT-small) denotes DINO pre-training followed by ProtoNet with a ViT-small backbone.
  • Pre-training variants: The evaluated pre-training variants include DINO, BEiT, CLIP, supervised ImageNet-21k, and supervised ImageNet-1k pre-training.The supplement also lists a BEiT plus supervised ImageNet-21k variant.

2. Additional results for miniImageNet and CIFAR-FS

The additional experiments compare pre-training methods and backbone architectures, while the fine-tuning ablation identifies settings used for the fine-tuning phase.

  • Table 4 compares different pre-training methods and backbone architectures on miniImageNet and CIFAR-FS.
  • Fine-tuning hyper-parameters: The fine-tuning stage varies learning rate, gradient-descent steps, and support-set augmentation probability.Learning rate is dominant; the adopted settings are augmentation probability 0.9 and 50 gradient steps.
  • Meta-Dataset comparisons: Tables 1–3 compare pre-training methods and backbone architectures across Meta-Dataset pre-training and meta-training settings.Table 2 restricts meta-training results to ImageNet only, while Table 3 reports Meta-Dataset results more broadly.

4. T-SNE plots: before and after meta-training

T-SNE visualizations show that DINO pre-training produces high-quality representations with semantic clusters across several domains, providing ProtoNet with a useful initialization.

  • Before and after meta-training: DINO pre-training yields high-quality feature representations across multiple domains, including domains that are not necessarily similar to ImageNet.
  • Before and after meta-training: Semantic clusters have already emerged before ProtoNet refinement in the aircraft, CUB, and Omniglot domain examples.These examples are presented in Figures 2, 3, and 4.
  • Before and after meta-training: ProtoNet can refine the initially emerged clusters into much tighter clusters.
Loading 2204.07305v1…