PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?

Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B. Tenenbaum, Phillip Isola

arXiv:2003.11539v2cs.CVcs.LG

TL;DR

Few-shot learning asks how models can adapt quickly to new tasks with limited data, and the paper questions whether sophisticated meta-learning algorithms are necessary. It trains supervised or self-supervised embeddings, fits linear classifiers on them, and adds self-distillation; this simple approach outperforms prior methods on few-shot benchmarks.

  • Problem

    Few-shot learning requires rapid adaptation to new tasks from limited data, motivating research into meta-learning algorithms and the role of learned representations.

  • Method

    The method learns an embedding on the merged meta-training set, trains a linear classifier for each task, and applies self-distillation to the embedding model.

  • Results

    The combined method improves average performance by 3% over previous state-of-the-art results on widely used benchmarks and by more than 7% on Meta-Dataset.

  • Takeaways & Limitations

    Good learned embeddings can be more effective than complicated meta-learning algorithms, while self-supervised representations can perform similarly to supervised ones.

Abstract

from arXiv · show

The focus of recent meta-learning research has been on the development of learning algorithms that can quickly adapt to test time tasks with limited data and low computational cost. Few-shot learning is widely used as one of the standard benchmarks in meta-learning. In this work, we show that a simple baseline: learning a supervised or self-supervised representation on the meta-training set, followed by training a linear classifier on top of this representation, outperforms state-of-the-art few-shot learning methods. An additional boost can be achieved through the use of self-distillation. This demonstrates that using a good learned embedding model can be more effective than sophisticated meta-learning algorithms. We believe that our findings motivate a rethinking of few-shot image classification benchmarks and the associated role of meta-learning algorithms. Code is available at: http://github.com/WangYueFt/rfs/.

1. Introduction

The paper argues that learned representations, rather than elaborate meta-learning algorithms, are central to few-shot classification. It proposes a fixed embedding with a linear classifier and improves it further through self-distillation.

  • Motivation: Few-shot learning evaluates rapid adaptation to new tasks from limited data, commonly using meta-training and meta-testing task distributions.Meta-learning methods include optimization-based and metric-based approaches.
  • Finding: The baseline suggests that good learned representations can be more powerful for few-shot classification than complicated meta-learning algorithms.Unlike transductive fine-tuning, it does not use test-set information and keeps the feature extractor fixed.
  • Baseline: The proposed baseline trains a neural network on merged meta-training data, freezes its penultimate-layer representation, and fits a linear classifier per test task.Training uses either ordinary classification or self-supervised learning.
  • Enhancement: Self-distillation further improves the representation-based baseline.The student and teacher have identical architectures and tasks, and self-distillation is applied to the pretrained network.
  • Results: The combined method improves average performance by 3% over previous state-of-the-art results on widely used benchmarks.On Meta-Dataset, it exceeds previous best results by more than 7% on average.
  • Results: Self-supervised representations achieve similar performance to fully supervised representations, supporting self-supervised embedding learning when labels are unavailable.The paper frames this as learning to learn by learning a good self-supervised embedding.

2. Related works

Related work organizes meta-learning into metric-based and optimization-based approaches, while also examining why these methods adapt quickly. It covers benchmark datasets and knowledge distillation as relevant foundations.

  • Metric-based meta-learning: Metric-based methods embed inputs into fixed-dimensional vectors and learn task-dependent metrics for classifying queries from support examples.Examples include Siamese Networks, Matching Networks, Prototypical Networks, Relation Networks, and TADAM.
  • Optimization-based meta-learning: Optimization-based methods seek learners that acquire task-specific knowledge quickly from few examples while retaining transferable knowledge.MAML searches for parameters that improve substantially after a few gradient steps on a new task.
  • Understanding MAML: Analyses of MAML indicate that feature reuse is the dominant component of its effectiveness, rather than rapid task-specific learning alone.Reptile and related work provide complementary optimization and generalization perspectives.
  • Datasets: Few-shot benchmarks include datasets such as Omniglot, which was designed for one-shot recognition of handwritten characters.These datasets test progress in meta-learning and few-shot learning.
  • Knowledge distillation: Knowledge distillation transfers information from ensembles or teacher models to student models by matching predictive distributions.Sequential distillation extends this framework and has improved teacher-model performance.

3. Method

The method learns an embedding model from merged meta-training data, freezes it at meta-testing, and fits a simple linear classifier separately for each task. Sequential self-distillation can further refine the embedding used for few-shot prediction.

  • 3.1. Problem formulation: Meta-learning evaluates a base learner across held-out tasks, where each task provides limited training data and testing examples.The base learner is trained on each task's training set and evaluated on its testing set.
  • 3.2. Learning embedding model through classification: The method merges meta-training tasks into one classification task and pre-trains an embedding model on the combined data.The learned network is retained through its penultimate layer as a feature extractor.
  • 3.2. Learning embedding model through classification: The default base learner is multivariate logistic regression trained on support embeddings and used to classify query embeddings.Nearest-neighbor classifiers with L-2 or cosine distance are also evaluated.
  • 3.2. Learning embedding model through classification: At meta-testing, the embedding model remains fixed while a base learner is re-initialized and trained separately on each task's training examples.Unlike methods that fine-tune embeddings at test time, this approach carries the pre-trained embedding unchanged into evaluation.
  • 3.3. Sequential self-distillation: Sequential self-distillation trains each generation from knowledge distilled from the preceding generation, then uses the final generation's embedding for meta-testing.Generation 0 uses standard cross-entropy, and the operation can be repeated K times.

4. Experiments

The experiments evaluate the simple embedding-plus-linear-classifier approach across established few-shot benchmarks, using ResNet backbones and comparisons with prior methods. The baseline is competitive or superior, while distillation further improves results.

  • Benchmarks: Experiments cover miniImageNet, tieredImageNet, CIFAR-FS, and FC100, with additional results on Meta-Dataset.The first two derive from ImageNet, while CIFAR-FS and FC100 reorganize CIFAR-100.
  • Setup: The backbone is a ResNet12 with four residual blocks and global average pooling to produce feature embeddings.The architecture uses three convolutional layers per block and max pooling after the first three blocks.
  • Setup: Meta-testing trains an N-way logistic regression classifier on extracted features, with standard image augmentation used during embedding training.The classifier is implemented using scikit-learn.
  • Results: The simple baseline is comparable to MetaOptNet on miniImageNet and outperforms prior work by at least 3% on tieredImageNet.These results are reported as median accuracies over three runs, each averaging 1,000 randomly sampled tasks.
  • Results: Distillation improves the simple baseline by 2–3% on the ImageNet-derived benchmarks.The comparison uses average few-shot classification accuracies with 95% confidence intervals.
  • Results: Fine-tuning on the small support set performs worse than the corresponding pre-trained model in the reported comparison.The cited comparison reports 57.73 ± 0.62 for fine-tuning and 61.1 ± 0.86 for the adopted setup.

4.3. Results on CIFAR derivatives

On CIFAR-FS and FC100, the embedding-based baseline is competitive with strong few-shot methods, while distillation reaches new state-of-the-art results. Ablations show consistent benefits from classifier and feature choices, with further gains from distillation.

  • Results on CIFAR derivatives: The simple baseline is comparable to Prototypical Networks and MetaOptNet on CIFAR-FS and outperforms both on FC100.The distillation version achieves the new state-of-the-art on both datasets.
  • Self-supervised representation learning: Self-supervised embeddings achieve performance comparable to supervised embeddings in the reported CIFAR-derived benchmark comparison.The method trains a label-free ResNet50 embedding on the merged meta-training set using MoCo- and CMC-inspired approaches.
  • Ablation experiments: Logistic regression significantly outperforms nearest neighbours, especially in the 5-shot setting, across the four ablation benchmarks.The ablation compares logistic regression, nearest neighbours, feature normalization, augmentation, and distillation.
  • Ablation experiments: 2% is the consistent 1-shot accuracy improvement from L-2 feature normalization across all four datasets.Normalization places feature embeddings on the unit sphere.
  • Ablation experiments: Distillation adds a further 2% gain even when the other ablation techniques are combined.Support-image augmentation provides only marginal improvement in the same study.
  • Effects of distillation: Few-shot recognition improves during the first two or three sequential distillation generations before accuracy begins decreasing.Feature normalization significantly alleviates the later degradation for logistic regression and nearest neighbours.

4.7. Choice of base classifier

The classifier comparison indicates that logistic regression benefits from feature scale and remains stronger than nearest neighbours as the support set grows. Across backbone comparisons, better embeddings and more training data consistently improve few-shot performance.

  • Choice of base classifier: Logistic regression is clearly better than nearest neighbours in the 1-shot comparison, despite their expected similarity in that regime.The authors attribute the difference to feature scale.
  • Choice of base classifier: After L-2 normalization, logistic regression and nearest neighbours perform similarly in the 1-shot comparison.The normalized variants are denoted LR+Norm and NN+Norm.
  • Backbone comparisons: Results consistently improve with more training data and better backbone networks, supporting the importance of embedding quality.The comparison includes a four-layer ConvNet, ResNet12, and ResNet12 with squeeze-and-excitation modules.
  • Backbone comparisons: The ConvNet baseline outperforms other few-shot or meta-learning models using the same network.This result is reported alongside comparisons across model size and training settings.

5. Results on Meta-Dataset

Meta-Dataset provides a larger and more diverse few-shot evaluation than miniImageNet and tieredImageNet. The simple baselines outperform the best prior reported results on most testing subsets, with a substantial average margin.

  • Benchmark: Meta-Dataset contains more diverse and realistic samples than miniImageNet and tieredImageNet.It is introduced as a benchmark for large-scale few-shot evaluation.
  • Setup: The embedding model is trained solely on the ILSVRC training split using a ResNet-18 backbone.The setup follows Meta-Dataset’s split and uses 128×128 inputs.
  • Results: The simple baselines outperform the best prior results on 9 of 10 testing datasets, often by a large margin.The prior comparison selects the best accuracy across seven methods and three architectures for each testing subset.
  • Results: More than 7% average improvement separates the logistic-regression baseline from previous best results.Self-distillation improves max(LR, SVM) on 7 of the 10 testing subsets.
  • Results: Logistic regression performs better than linear SVM in the authors’ empirical comparison.This observation is reported alongside the Meta-Dataset testing results.

6. Discussion

The paper presents a simple few-shot classification baseline that outperforms current state-of-the-art methods across four widely used benchmarks. Self-distillation further improves performance, while self-supervised embeddings can remain effective without labeled meta-training data.

  • 6. Discussion: Across four widely used few-shot benchmarks, the simple baseline outperforms the current state-of-the-art methods.The authors describe this result as evidence that a simple representation-based approach can perform strongly in few-shot image classification.
  • 6. Discussion: Self-distillation further improves the baseline’s performance by 2-3%.The paper applies self-distillation to the pretrained network and reports this additional boost.
  • 6. Discussion: Self-supervised learning may provide useful embeddings for meta-testing tasks even when meta-training labels are unavailable.The discussion presents this as a possible way to leverage state-of-the-art self-supervised learning methods.
  • 6. Discussion: Meta-Dataset evaluation reports average accuracy with variable numbers of ways and shots, comparing four method variants against the best accuracy from seven prior methods.The evaluation samples 1000 tasks per episode.

A. Architectures

The paper uses ResNet-12 and SEResNet-12 architectures. SEResNet-12 includes a Squeeze-and-Excitation layer, and a dotted component is removed during meta-testing.

  • A. Architectures: Figure 5 presents the ResNet-12 and SEResNet-12 architectures used in the paper.The accompanying text identifies these as the architectures used in the experiments.
  • A. Architectures: In SEResNet-12, “SE, 4” denotes a Squeeze-and-Excitation layer with reduction parameter 4.The notation is defined in the architecture figure description.
  • A. Architectures: The dotted box in the architecture is removed during meta-testing.This design change is specified in the Figure 5 description.

B. More Training Details

Training details vary across the model and benchmark settings. SEResNet-12 follows the ResNet-12 setup, while the 4-layer convnet and feature choice receive additional benchmark-specific specifications.

  • B. More Training Details: SEResNet-12 uses the same training setup as ResNet-12 on all four benchmarks.The setup is described in Section 4.1.
  • B. More Training Details: The 4-layer convnet uses the ResNet-12 training setup on tieredImageNet, CIFAR-FS, and FC100.MiniImageNet instead uses a separate 240-epoch schedule with learning-rate decays at epochs 150, 180, and 210.
  • B. More Training Details: Using the logit layer as the feature gives slightly better miniImageNet accuracy, with an improvement of at most 1%.The paper reports this feature choice in Table 5 for miniImageNet.

C. Unsupervised Learning Details

The unsupervised-learning setup adapts ResNet-50 for 84 × 84 images and trains only on miniImageNet’s meta-training set. It follows established contrastive-learning recipes with two stated changes.

  • C. Unsupervised Learning Details: The first layer of a standard ResNet-50 is adapted to accept 84 × 84 images.This modification matches the image size used in the miniImageNet setup.
  • C. Unsupervised Learning Details: Training uses only the miniImageNet meta-training set and excludes the meta-validation set.The passage explicitly states that the meta-validation set is not used.
  • C. Unsupervised Learning Details: The setup follows the CMC and MoCo training recipes, with 2048 negatives used for each positive sample.The passage identifies the limited total number of miniImageNet images as the reason for using 2048 negatives.
Loading 2003.11539v2…