Source-linked AI summary
SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learning
Yan Wang, Wei-Lun Chao, Kilian Q. Weinberger, Laurens van der Maaten
TL;DR
Few-shot learning typically combines convolutional features, meta-learning, and nearest-neighbor classification, but the accuracy of nearest-neighbor baselines without meta-learning remains a central question. The paper evaluates SimpleShot, which applies simple feature transformations before nearest-neighbor classification. Centering plus L2-normalization yields competitive results, including the highest one-shot and five-shot accuracies for three of five miniImageNet architectures.
Problem
The paper examines whether nearest-neighbor baselines without meta-learning can provide competitive few-shot learning accuracy.
Method
SimpleShot trains convolutional-network features on base classes and applies centering and L2-normalization before Euclidean nearest-neighbor or nearest-centroid classification.
Results
SimpleShot achieves the highest one-shot and five-shot accuracies for three of five network architectures on miniImageNet, while feature transformations consistently improve accuracy across evaluated datasets.
Takeaways & Limitations
SimpleShot reestablishes nearest-neighbor classification as a competitive baseline for few-shot learning.
Abstract
from arXiv · showhide
Few-shot learners aim to recognize new object classes based on a small number of labeled training examples. To prevent overfitting, state-of-the-art few-shot learners use meta-learning on convolutional-network features and perform classification using a nearest-neighbor classifier. This paper studies the accuracy of nearest-neighbor baselines without meta-learning. Surprisingly, we find simple feature transformations suffice to obtain competitive few-shot learning accuracies. For example, we find that a nearest-neighbor classifier used in combination with mean-subtraction and L2-normalization outperforms prior results in three out of five settings on the miniImageNet dataset.
1. Introduction
Few-shot learning targets recognition of novel visual classes from few examples, yet current systems commonly combine convolutional features, meta-learning, and nearest-neighbor classification. SimpleShot challenges this approach by showing that feature transformations can make nearest-neighbor baselines highly competitive without meta-learning.
- Motivation: Few-shot learning systems must reuse knowledge from base classes to recognize novel classes from only a few labeled examples.This setting is important for recognizing the many natural and man-made classes found in the world.
- Existing approaches: Many current few-shot learners combine convolutional-network features, meta-learning, and nearest-neighbor classification.Prior studies suggest meta-learning outperforms vanilla nearest-neighbor classification.
- SimpleShot: SimpleShot challenges the status quo by achieving state-of-the-art performance on popular few-shot benchmarks without meta-learning.The approach applies simple feature transformations before nearest-neighbor classification.
- SimpleShot: Mean subtraction and L2-normalization applied to DenseNet features produce very competitive nearest-neighbor results on miniImageNet and tieredImageNet.The observations generalize to other convolutional-network architectures.
2. Nearest Neighbors for Few-Shot Learning
SimpleShot trains a convolutional feature extractor on base classes, then classifies novel examples in feature space using Euclidean nearest neighbors or nearest centroids. It evaluates unnormalized, L2-normalized, and centered L2-normalized representations, with centering becoming effective alongside L2-normalization.
- Problem setting: Few-shot learning uses a base training set and a support set containing K examples for each of C novel classes.This is called the K-shot C-way setting.
- Feature extraction: SimpleShot extracts D-dimensional image features with a convolutional network trained using a linear classifier on the base classes.The network and classifier are trained jointly with cross-entropy loss and stochastic gradient descent.
- Nearest-neighbor rule: In one-shot classification, the nearest-neighbor rule assigns each test image the label of its most similar support example in feature space.The classifier uses a distance measure between extracted features.
- Nearest-centroid rule: In multi-shot settings, SimpleShot averages each class’s support features into a centroid and applies the nearest-neighbor rule to those centroids.Each centroid serves as a one-shot representative for its class.
- Feature transformations: SimpleShot uses Euclidean distance and compares unnormalized, centered, and L2-normalized feature representations.Centering subtracts the mean base-class feature vector, while L2-normalization scales each feature vector to unit norm; centering alone preserves Euclidean distances.
3. Experiments
The experiments evaluate SimpleShot across multiple datasets, architectures, feature transformations, and few-shot task protocols. Simple feature normalization consistently improves nearest-neighbor performance and yields results comparable with or better than state-of-the-art methods.
- Datasets: Experiments cover miniImageNet, tieredImageNet, and CIFAR-100, using predefined base, validation, and novel-class splits.The datasets differ in class counts and split construction, including WordNet-based semantic separation for tieredImageNet.
- Evaluation protocol: Evaluation draws 10,000 K-shot C-way tasks with C novel classes, K support images, and 15 query images per class.The main focus is one-shot and five-shot, five-way classification, with average accuracy and 95% confidence intervals reported.
- Methods: The study compares unnormalized, L2-normalized, and centered L2-normalized features followed by Euclidean nearest-neighbor classification.Five convolutional-network architectures are used as feature generators, and the networks are trained from scratch on base-class classification.
- Feature transformations: L2-normalization consistently improves accuracy by at least 3% on miniImageNet, tieredImageNet, and CIFAR-100, while centering adds another 1−3%.These gains are measured relative to unnormalized nearest-neighbor features.
- Results: SimpleShot obtains the highest one-shot and five-shot accuracies for three of five network architectures on miniImageNet.The classifiers achieve accuracies comparable with or better than state-of-the-art few-shot learners.
- Additional analyses: Across training epochs, C2LN nearest-neighbor classifiers consistently outperform their UN and L2N counterparts.This result suggests the observed role of feature transformations does not depend on training duration, and CL2N also improves ProtoNet performance.
4. Conclusion
The paper finds that simple feature transformations can substantially improve nearest-neighbor few-shot classifiers. The authors propose SimpleShot as a competitive baseline for future few-shot learning studies.
- Centering and L2-normalization can improve representation quality enough for SimpleShot classifiers to outperform several state-of-the-art few-shot approaches.
- The paper analyzes simple feature transformations specifically in nearest-neighbor classifiers for few-shot learning.
- The authors hope SimpleShot will serve as a competitive baseline in future few-shot learning research.
A. Meta-iNat Results
The meta-iNat experiments evaluate SimpleShot in 227-way multi-shot classification using a long-tailed dataset split into base and novel classes. L2-normalization improves accuracy, while additional centering does not improve it further.
- Experimental setup: The meta-iNat benchmark uses 908 base classes and 227 novel classes for 227-way multi-shot evaluation, with the number of shots varying per class.
- Results: 62.13% per-class accuracy and 65.09% mean accuracy are reported as the highest SimpleShot accuracies on meta-iNat.Per-class accuracy averages accuracy across test classes, while mean accuracy averages across all test images.
- Results: L2-normalization improves SimpleShot accuracy on meta-iNat, but centering after L2-normalization does not improve it further.
- Results: Figure 2 reports each classifier’s absolute per-class accuracy improvement relative to the unnormalized UN baseline.The compared classifiers are L2N and CL2N.