Source-linked AI summary
Prototypical Networks for Few-shot Learning
Jake Snell, Kevin Swersky, Richard S. Zemel
TL;DR
Few-shot classification requires adapting to unseen classes from only a few labeled examples, making overfitting a central challenge. Prototypical networks learn an embedding where each class is represented by its support-set mean, achieving state-of-the-art results on several few-shot benchmarks and CUB zero-shot classification while remaining simpler and more efficient than recent meta-learning approaches.
Problem
Few-shot classification asks models to recognize unseen classes from only a few examples, a difficult setting in which naïve retraining can severely overfit.
Method
Prototypical networks learn an embedding space, represent each class by the mean of its embedded support examples, and classify queries by nearest prototype.
Results
The approach achieves state-of-the-art performance on several benchmark tasks, including few-shot benchmarks and CUB zero-shot classification, with Euclidean distance outperforming cosine similarity.
Takeaways & Limitations
Simple distance and episodic-training choices can substantially improve few-shot performance, while prototypical networks offer a simpler, more efficient alternative to recent meta-learning approaches.
Takeaways & Limitations
The reported approach primarily focuses on squared Euclidean distance, whose effectiveness is attributed to non-linearity learned within the embedding function.
Abstract
from arXiv · showhide
We propose prototypical networks for the problem of few-shot classification, where a classifier must generalize to new classes not seen in the training set, given only a small number of examples of each new class. Prototypical networks learn a metric space in which classification can be performed by computing distances to prototype representations of each class. Compared to recent approaches for few-shot learning, they reflect a simpler inductive bias that is beneficial in this limited-data regime, and achieve excellent results. We provide an analysis showing that some simple design decisions can yield substantial improvements over recent approaches involving complicated architectural choices and meta-learning. We further extend prototypical networks to zero-shot learning and achieve state-of-the-art results on the CU-Birds dataset.
1 Introduction
Few-shot classification requires adapting to unseen classes from very limited data, making overfitting a central challenge. Prototypical networks address this by learning an embedding where each class is represented by a prototype and queries are classified by prototype distance.
- Problem: Few-shot classification adapts a classifier to unseen classes using only a few examples per class, while retraining on this limited data can severely overfit.The setting includes one-shot classification, where only one example per new class is available.
- Method: Prototypical networks impose a simple inductive bias by learning an embedding in which points cluster around one prototype representation per class.The approach learns a nonlinear mapping from inputs into an embedding space using a neural network.
- Method: Queries are classified by finding the nearest class prototype in the embedding space.The model can use a softmax over distances to class prototypes; in few-shot learning, prototypes are computed as means of embedded support examples.
- Contributions: The paper formulates prototypical networks for both few-shot and zero-shot learning and connects the method to matching networks in the one-shot setting.In zero-shot learning, prototypes are produced by embedding class metadata.
- Contributions: The paper relates class-mean prototypes to clustering under Bregman divergences, including squared Euclidean distance, and reports that distance choice is vital.These analyses are presented to justify using class means as prototypes.
2 Prototypical Networks
Prototypical networks classify queries by comparing embedded representations with class prototypes computed as means of labeled support examples. Their distance-based formulation supports episodic training, admits a mixture-density interpretation for Bregman divergences, and becomes a linear classifier under squared Euclidean distance.
- Prototype computation: Each class prototype is the mean of its embedded labeled support examples, computed by a learnable embedding function.The support set contains N labeled examples, and each prototype lies in an M-dimensional embedding space.
- Distance-based prediction: Queries receive class probabilities through a softmax over distances to the class prototypes in the embedding space.The distance function maps pairs of M-dimensional representations to nonnegative values.
- Episodic training: Training minimizes the negative log-probability of the true class with SGD over episodes containing randomly selected classes, support examples, and query examples.Within each episode, support and query examples are sampled from distinct subsets of each selected class.
- Probabilistic interpretation: For regular Bregman divergences, prototype computation yields optimal cluster representatives and prototypical networks are equivalent to exponential-family mixture density estimation.The distance choice therefore specifies assumptions about class-conditional distributions in the embedding space.
- Euclidean-distance analysis: Squared Euclidean distance makes prototypical networks equivalent to a linear model, while learned nonlinearity remains available through the embedding function.The paper primarily uses squared Euclidean distance, corresponding to spherical Gaussian densities, and reports it as effective despite this equivalence.
3 Experiments
The experiments evaluate prototypical networks on Omniglot and miniImageNet for few-shot classification, and on CUB-200-2011 for zero-shot learning. Results examine episode design and distance metrics, and report strong zero-shot performance against attribute-based methods.
- Experimental datasets: Experiments cover Omniglot and miniImageNet few-shot classification, plus CUB-200-2011 zero-shot classification.The few-shot datasets use established splits, while CUB experiments assess generalization to unseen bird classes.
- Omniglot: Prototypical networks use Euclidean distance in 1-shot and 5-shot Omniglot episodes, with 60 classes and 5 query points per class.Training-shot and test-shot are matched, and more classes per training episode are found advantageous.
- miniImageNet: miniImageNet experiments match train and test shots, using 30-way episodes for 1-shot and 20-way episodes for 5-shot classification.Each class contains 15 query points per episode, and comparisons use baselines reported by Ravi and Larochelle.
- miniImageNet: Further analysis compares cosine versus Euclidean distance and 5-way versus 20-way training episodes for matching and prototypical networks.The comparison covers 1-shot and 5-shot scenarios and uses a shared embedding architecture for method comparability.
- Zero-shot learning: On CUB-200-2011, the method achieves state-of-the-art zero-shot results by a large margin against approaches using attributes as class meta-data.Comparisons include ALE, SJE, DS-SJE/DA-SJE, and a clustering approach based on a fine-tuned AlexNet feature space.
4 Related Work
The section relates prototypical networks to metric-learning, nearest-class-mean, meta-learning, generative-modeling, and zero-shot learning approaches. It emphasizes differences in embedding, training procedure, prototype construction, and loss.
- Metric learning: Metric-learning methods such as NCA, neural-network extensions of NCA, and LMNN optimize neighbor-based classification in transformed spaces.NCA learns a Mahalanobis distance to maximize KNN leave-one-out accuracy, while LMNN uses a hinge-loss objective.
- Nearest class mean: Nearest class mean methods rapidly incorporate new classes without retraining but assume a linear embedding and many examples per novel class.Prototypical networks instead use nonlinear neural embeddings and episodic training for few-shot classification.
- Meta-learning: Ravi and Larochelle’s meta-learning method trains an LSTM to learn model-training dynamics for episode-level generalization, while matching and prototypical networks produce simple classifiers.The method treats LSTM dynamics and gradient descent as effectively equivalent.
- Generative modeling: Prototypical networks relate to the neural statistician because both summarize sets through encoded points, sample means, and post-processing networks.The neural statistician extends variational autoencoders to model datasets rather than individual points.
- Zero-shot learning: For zero-shot learning, prototypical networks resemble prior methods that predict linear-classifier weights from embedded metadata, but uniquely use episodic training rather than empirical risk loss.DS-SJE and DA-SJE also learn deep multimodal embeddings for images and class metadata, but neither cited approach uses episodic training.
5 Conclusion
The paper proposes prototypical networks, a simple episodically trained method that represents classes by means in a learned neural representation space and achieves state-of-the-art results without sophisticated extensions.
- Method: Prototypical networks represent each class by the mean of its examples in a representation space learned by a neural network.This class representation is the method’s central design idea.
- Training: The networks are trained with episodic training to perform specifically well in the few-shot setting.Training is aligned directly with the few-shot classification regime.
- Comparison: The approach is far simpler and more efficient than recent meta-learning approaches.The conclusion contrasts the method’s simplicity and efficiency with recent meta-learning methods.
- Results: The method produces state-of-the-art results even without sophisticated extensions.Its reported performance does not depend on the sophisticated extensions developed for more complex approaches.
A Additional Omniglot Results
Additional Omniglot experiments evaluate Euclidean prototypical networks across different episode configurations and visualize their learned embeddings. The embeddings cluster minor character variations closely around class prototypes.
- A Additional Omniglot Results: Test accuracy was evaluated for Euclidean prototypical networks trained with 5, 20, and 60 classes per episode.Accuracy was averaged over 1,000 randomly generated test episodes, with episodes specified by way, shot, and query.
- A Additional Omniglot Results: A t-SNE visualization shows minor variations of hand-drawn characters clustering closely around their class prototypes.The visualization uses a subset of test characters from the same alphabet, although actual test episodes may draw classes from different alphabets.
B Additional miniImageNet Results
The section reports additional miniImageNet experiments on training-episode configuration and distance-based comparisons. Varying the number of classes per episode shows that episode construction is important for few-shot classification.
- Training-episode configuration: Training episodes were varied from 5 to 30 classes while keeping 15 query points per class fixed.These experiments used Euclidean-distance prototypical networks.
- Training-episode configuration: The results indicate that training-episode construction is an important consideration for achieving good few-shot classification results.The comparison is presented in Figure 4, with full results in Table 6.
- Distance and episode comparisons: Table 5 compares matching and prototypical networks on miniImageNet across cosine versus Euclidean distance, 5-way versus 20-way, and 1-shot versus 5-shot settings.All experiments use a shared encoder with embedding dimension 1,600.
- Evaluation protocol: Classification accuracy is averaged over 600 randomly generated test episodes, with 95% confidence intervals reported.This reporting format is specified for the comparisons in Tables 5 and 6 and Figure 4.