PaperlayerView exact revision on arXiv

Source-linked AI summary

SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learning

Yan Wang, Wei-Lun Chao, Kilian Q. Weinberger, Laurens van der Maaten

arXiv:1911.04623v2cs.CV

TL;DR

Few-shot learning typically combines convolutional features, meta-learning, and nearest-neighbor classification, but the accuracy of nearest-neighbor baselines without meta-learning remains a central question. The paper evaluates SimpleShot, which applies simple feature transformations before nearest-neighbor classification. Centering plus L2-normalization yields competitive results, including the highest one-shot and five-shot accuracies for three of five miniImageNet architectures.

  • Problem

    The paper examines whether nearest-neighbor baselines without meta-learning can provide competitive few-shot learning accuracy.

    Source: §1 · p. 1

  • Method

    SimpleShot trains convolutional-network features on base classes and applies centering and L2-normalization before Euclidean nearest-neighbor or nearest-centroid classification.

    Source: §1 · p. 1, §3.2 · p. 3

  • Results

    SimpleShot achieves the highest one-shot and five-shot accuracies for three of five network architectures on miniImageNet, while feature transformations consistently improve accuracy across evaluated datasets.

    Source: §3.2 · p. 3

  • Takeaways & Limitations

    SimpleShot reestablishes nearest-neighbor classification as a competitive baseline for few-shot learning.

    Source: §1 · p. 1, §4 · p. 3

Abstract

from arXiv · show

Few-shot learners aim to recognize new object classes based on a small number of labeled training examples. To prevent overfitting, state-of-the-art few-shot learners use meta-learning on convolutional-network features and perform classification using a nearest-neighbor classifier. This paper studies the accuracy of nearest-neighbor baselines without meta-learning. Surprisingly, we find simple feature transformations suffice to obtain competitive few-shot learning accuracies. For example, we find that a nearest-neighbor classifier used in combination with mean-subtraction and L2-normalization outperforms prior results in three out of five settings on the miniImageNet dataset.

1. Introduction

Few-shot learning targets recognition of novel visual classes from few examples, yet current systems commonly combine convolutional features, meta-learning, and nearest-neighbor classification. SimpleShot challenges this approach by showing that feature transformations can make nearest-neighbor baselines highly competitive without meta-learning.

Motivation

  • Few-shot learning systems must reuse knowledge from base classes to recognize novel classes from only a few labeled examples.This setting is important for recognizing the many natural and man-made classes found in the world.Source: §1 · p. 1

Existing approaches

  • Many current few-shot learners combine convolutional-network features, meta-learning, and nearest-neighbor classification.Prior studies suggest meta-learning outperforms vanilla nearest-neighbor classification.Source: §1 · p. 1

SimpleShot

  • SimpleShot challenges the status quo by achieving state-of-the-art performance on popular few-shot benchmarks without meta-learning.The approach applies simple feature transformations before nearest-neighbor classification.Source: §1 · p. 1
  • Mean subtraction and L2-normalization applied to DenseNet features produce very competitive nearest-neighbor results on miniImageNet and tieredImageNet.The observations generalize to other convolutional-network architectures.Source: §1 · p. 1

2. Nearest Neighbors for Few-Shot Learning

SimpleShot trains a convolutional feature extractor on base classes, then classifies novel examples in feature space using Euclidean nearest neighbors or nearest centroids. It evaluates unnormalized, L2-normalized, and centered L2-normalized representations, with centering becoming effective alongside L2-normalization.

Problem setting

  • Few-shot learning uses a base training set and a support set containing K examples for each of C novel classes.This is called the K-shot C-way setting.Source: §2 · p. 2

Feature extraction

  • SimpleShot extracts D-dimensional image features with a convolutional network trained using a linear classifier on the base classes.The network and classifier are trained jointly with cross-entropy loss and stochastic gradient descent.Source: §2 · p. 2

Nearest-neighbor rule

  • In one-shot classification, the nearest-neighbor rule assigns each test image the label of its most similar support example in feature space.The classifier uses a distance measure between extracted features.Source: §2 · p. 2

Nearest-centroid rule

  • In multi-shot settings, SimpleShot averages each class’s support features into a centroid and applies the nearest-neighbor rule to those centroids.Each centroid serves as a one-shot representative for its class.Source: §2 · p. 2

Feature transformations

  • SimpleShot uses Euclidean distance and compares unnormalized, centered, and L2-normalized feature representations.Centering subtracts the mean base-class feature vector, while L2-normalization scales each feature vector to unit norm; centering alone preserves Euclidean distances.Source: §2.1 · p. 2

3. Experiments

The experiments evaluate SimpleShot across multiple datasets, architectures, feature transformations, and few-shot task protocols. Simple feature normalization consistently improves nearest-neighbor performance and yields results comparable with or better than state-of-the-art methods.

Datasets

  • Experiments cover miniImageNet, tieredImageNet, and CIFAR-100, using predefined base, validation, and novel-class splits.The datasets differ in class counts and split construction, including WordNet-based semantic separation for tieredImageNet.Source: §3.1 · p. 2

Evaluation protocol

  • Evaluation draws 10,000 K-shot C-way tasks with C novel classes, K support images, and 15 query images per class.The main focus is one-shot and five-shot, five-way classification, with average accuracy and 95% confidence intervals reported.Source: §3.1 · p. 2

Methods

  • The study compares unnormalized, L2-normalized, and centered L2-normalized features followed by Euclidean nearest-neighbor classification.Five convolutional-network architectures are used as feature generators, and the networks are trained from scratch on base-class classification.Source: §3.1 · pp. 2–3

Feature transformations

  • L2-normalization consistently improves accuracy by at least 3% on miniImageNet, tieredImageNet, and CIFAR-100, while centering adds another 1−3%.These gains are measured relative to unnormalized nearest-neighbor features.Source: §3.2 · p. 3

Results

  • SimpleShot obtains the highest one-shot and five-shot accuracies for three of five network architectures on miniImageNet.The classifiers achieve accuracies comparable with or better than state-of-the-art few-shot learners.Source: §3.2 · p. 3

Additional analyses

  • Across training epochs, C2LN nearest-neighbor classifiers consistently outperform their UN and L2N counterparts.This result suggests the observed role of feature transformations does not depend on training duration, and CL2N also improves ProtoNet performance.Source: §3.2 · p. 3

4. Conclusion

The paper finds that simple feature transformations can substantially improve nearest-neighbor few-shot classifiers. The authors propose SimpleShot as a competitive baseline for future few-shot learning studies.

  • Centering and L2-normalization can improve representation quality enough for SimpleShot classifiers to outperform several state-of-the-art few-shot approaches.Source: §4 · p. 3
  • The paper analyzes simple feature transformations specifically in nearest-neighbor classifiers for few-shot learning.Source: §4 · p. 3
  • The authors hope SimpleShot will serve as a competitive baseline in future few-shot learning research.Source: §4 · p. 3

A. Meta-iNat Results

The meta-iNat experiments evaluate SimpleShot in 227-way multi-shot classification using a long-tailed dataset split into base and novel classes. L2-normalization improves accuracy, while additional centering does not improve it further.

Experimental setup

  • The meta-iNat benchmark uses 908 base classes and 227 novel classes for 227-way multi-shot evaluation, with the number of shots varying per class.Source: A. Meta-iNat Results · p. 6

Results

  • 62.13% per-class accuracy and 65.09% mean accuracy are reported as the highest SimpleShot accuracies on meta-iNat.Per-class accuracy averages accuracy across test classes, while mean accuracy averages across all test images.Source: A. Meta-iNat Results · p. 6, Table 5 · p. 6
  • L2-normalization improves SimpleShot accuracy on meta-iNat, but centering after L2-normalization does not improve it further.Source: A. Meta-iNat Results · p. 6
  • Figure 2 reports each classifier’s absolute per-class accuracy improvement relative to the unnormalized UN baseline.The compared classifiers are L2N and CL2N.Source: Figure 2 · p. 6, A. Meta-iNat Results · p. 6

Related papers

Loading 1911.04623v2…