Source-linked AI summary
Infinite Mixture Prototypes for Few-Shot Learning
Kelsey R. Allen, Evan Shelhamer, Hanul Shin, Joshua B. Tenenbaum
TL;DR
Few-shot classification must recognize new classes from only a few examples while avoiding underfitting and overfitting. IMP represents each class with data-determined clusters, learning an embedding and adapting capacity while incorporating unlabeled data. It improves complex alphabet recognition by 25% and matches or exceeds neighbors and unimodal prototypes across experiments, with semi-supervised and unsupervised capabilities.
Problem
Few-shot classification must generalize from only a few examples, while current fixed-capacity models can underfit or overfit distributions of differing complexity.
Method
IMP jointly learns an embedding and represents each class with clusters whose number is inferred from data, incorporating labeled and unlabeled points through multimodal clustering.
Results
25% improvement in accuracy for alphabet recognition was achieved while preserving character-recognition accuracy on Omniglot, with equal or better accuracy than neighbors and unimodal prototypes in all experiments.
Takeaways & Limitations
Adaptive-capacity multimodal prototypes support simple and complex distributions and enable semi-supervised learning and fully unsupervised clustering inference.
Takeaways & Limitations
Benchmark results are reported for models trained and tested with n-way episodes to equalize comparison across methods.
Abstract
from arXiv · showhide
We propose infinite mixture prototypes to adaptively represent both simple and complex data distributions for few-shot learning. Our infinite mixture prototypes represent each class by a set of clusters, unlike existing prototypical methods that represent each class by a single cluster. By inferring the number of clusters, infinite mixture prototypes interpolate between nearest neighbor and prototypical representations, which improves accuracy and robustness in the few-shot regime. We show the importance of adaptive capacity for capturing complex data distributions such as alphabets, with 25% absolute accuracy improvements over prototypical networks, while still maintaining or improving accuracy on the standard Omniglot and mini-ImageNet benchmarks. In clustering labeled and unlabeled data by the same clustering rule, infinite mixture prototypes achieves state-of-the-art semi-supervised accuracy. As a further capability, we show that infinite mixture prototypes can perform purely unsupervised clustering, unlike existing prototypical methods.
1. Introduction
Few-shot learning must balance underfitting and overfitting when only a few examples define each class. IMP addresses differing distribution complexity by inferring class-specific cluster counts, improving complex-class recognition while retaining performance on simpler tasks.
- Few-shot classification recognizes new classes from only a few examples, making generalization vulnerable to overfitting and underfitting.
- Model capacity matters because nearest neighbors capture complex distributions, whereas prototypes robustly fit simple distributions but require choosing between them.
- Infinite mixture modeling infers the number of components from data, spanning representations from nearest neighbors to prototypical models.
- Characters can be represented by one prototype, but alphabets may violate the unimodal assumption and require multimodal modeling.
- IMP represents each class with data-determined clusters, learns an embedding, adapts capacity, and incorporates unlabeled data for semi-supervised or unsupervised inference.
- 25% improvement in accuracy for alphabet recognition was achieved while preserving accuracy on character recognition on Omniglot.
- IMP adapts to simple or complex distributions with equal or better accuracy than neighbors and unimodal prototypes in all experiments.
2. Background
Few-shot methods use episodic support-query tasks and nonparametric representations to classify with sparse data. The background contrasts high-capacity neighbors and fixed unimodal prototypes with adaptive infinite-mixture clustering implemented by DP-means.
- Few-shot Classification: Few-shot episodes sample classes and partition examples into support sets for learning and query sets for inference, with way and shot describing episode size.
- Neighbors: Nearest-neighbor classification assigns each query the label of its closest support, fitting complex distributions at the cost of storing and searching the training set.
- Prototypes: Prototypical networks represent each class by the mean of its embedded support points and classify queries using a softmax over distances to prototypes.
- Prototypes: With Euclidean distance, prototypes are equivalent to a Gaussian mixture model in the embedding with identity covariance and learn unimodal class distributions.
- Infinite Mixture Modeling: Infinite mixture models infer the number of components from data, adapting capacity between overfitting with high capacity and underfitting with low capacity.
- Infinite Mixture Modeling: DP-means creates a new cluster when a point’s minimum distance to existing means exceeds threshold λ, optimizing reconstruction error plus a cluster penalty.
3. Infinite Mixture Prototypes (IMP)
IMP extends prototypical representations from one cluster per class to adaptive multi-modal clustering, combining learned variance with data-driven cluster creation for few-shot inference.
- Multi-modal representation: IMP represents each class with multiple clusters and infers their number directly from the data.This extends prototypical networks beyond uni-modal class representations.
- Cluster inference: DP-means creates a new cluster when a point’s minimum compatible distance exceeds threshold λ, with separate handling for labeled and unlabeled clusters.IMP extends DP-means with softened assignments, label-aware distances, and end-to-end optimization.
- Query inference: Queries are classified using the closest cluster in each class, allowing IMP to interpolate between nearest-neighbor and prototypical representations.Cluster means are computed as weighted means of their assigned support points.
- Adaptive variance: Learned cluster variance scales point-to-cluster assignments and is jointly optimized with the embedding parameters.Small variance emphasizes nearby points, while large variance includes farther points.
- Training objective: Masking the loss to the closest cluster avoids overpenalizing multi-modality and improves few-shot accuracy over losses incorporating all clusters.The multi-modal methods use the same closest-cluster principle during inference and training.
- Ablations and alternatives: Learning and inference with IMP is more robust to λ choices than applying DP-means only at inference, and one clustering iteration suffices.Even 100 inference iterations had no effect on accuracy, indicating stable clustering.
- Ablations and alternatives: Alternative Dirichlet-process methods are 5–10% less accurate in experiments, mainly because the CRP prior favors clusters with more assignments.This effect is especially problematic early in training when unlabeled points are often incorrectly assigned.
4. Experiments
Experiments evaluate IMP on Omniglot and mini-ImageNet across supervised, semi-supervised, and unsupervised settings. IMP improves handling of complex, multi-modal classes while retaining performance on simpler classes and standard benchmarks.
- IMP substantially outperforms prototypical networks on Omniglot alphabet and character recognition after alphabet training.
- 95.4% accuracy on 20-way 1-shot character recognition exceeds prototypical networks trained directly on characters at 94.9%.
- 61.9% accuracy from MAML on 10-way 10-shot alphabet recognition indicates that this parametric alternative was insufficient for the task.
- IMP loses less than 5% relative accuracy on held-out character modes, compared with 16% for prototypical networks on testing modes.
- IMP supports fully unsupervised clustering and maintains strong performance across many unlabeled clusters without knowing class counts or seeing test-class examples during training.
- In fully supervised benchmarks, IMP matches or outperforms prototypical networks, while semi-supervised evaluation compares IMP with methods trained using labeled and unlabeled examples.
5. Related Work
Related work spans prototype, neighbor, metric-learning, and cognitive categorization approaches. IMP extends prototype methods with adaptive multi-modal clustering over labeled and unlabeled data.
- Prototypical networks use one mean-based cluster per class, while semi-supervised prototypes refine the same uni-modal representation with soft k-means.
- Prior multi-prototype approaches fixed and hand-tuned the number of prototypes and did not incorporate unlabeled data.
- IMP generalizes prototype methods to one or many clusters per class and applies the same multi-modal clustering framework to labeled and unlabeled data.
- Metric-learning methods address recognition by retrieval through embeddings that bring similar pairs together and separate dissimilar pairs.
- Cognitive categorization work distinguishes exemplar and prototype representations, while intermediate cluster models lacked cluster-selection rules or representation learning.
6. Conclusion
The paper argues that few-shot methods should adapt capacity to data complexity. IMP combines infinite mixture modeling with deep metric learning and supports supervised, semi-supervised, and unsupervised inference.
- IMP adapts model capacity to the given data by combining infinite mixture modeling with deep metric learning.
- The multi-modal extension supports fully unsupervised inference and naturally incorporates semi-supervised data during learning.
A.1. Implementation Details
Implementation uses a shared convolutional embedding architecture with dataset-specific embedding sizes, optimization schedules, and variance settings.
- The embedding network uses four convolutional blocks with 64-filter 3 × 3 convolutions, batch normalization, ReLU, and 2 × 2 max pooling.
- The resulting embedding dimensions are 64 for Omniglot and 1600 for mini-ImageNet.
- Omniglot training uses an initial learning rate of 1e-3, scheduled halving, 160,000 iterations, and gradient accumulation over eight episodes for 5-way training.
- For Omniglot, σ_l is learned jointly while σ_u is fixed because learning σ_u was unstable.
- mini-ImageNet training stops at 100,000 iterations, learns both variances jointly, and uses α = 10^-5.
A.2. Controlling for the Number of Gradients Taken During Optimization
The number of gradients accumulated per update affects few-shot optimization accuracy, so episode design should account for gradient count rather than only episode structure.
- 20-way episodes accumulate 16 times as many gradients as 5-way episodes.
- Accumulating gradients from 16 5-way episodes produces significantly better results than a single 5-way episode.
- Gradient accumulation can match the results obtained with 20-way episodes.
A.3. Alternative Infinite Mixture Model Algorithms
The paper compares Gibbs-based MAP inference and soft-assignment EM for infinite mixture models, adapting both to episodic few-shot inference. The alternatives expose practical trade-offs: Gibbs convergence is impractical, hard assignments are unstable, and removing the CRP prior improves—but does not match—IMP performance.
- Alternative inference algorithms: Gibbs sampling estimates the MAP of a Chinese restaurant process, while expectation maximization maintains soft cluster assignments.
- Alternative inference algorithms: The CRP assigns examples to existing clusters according to cluster counts or to new clusters through concentration parameter α.
- MAP-DP inference: A single-pass approximation replaces Gibbs convergence by selecting the most probable assignment and updating cluster parameters during each episode.
- Semi-supervised inference: Unlabeled points are often incorrectly assigned to labeled clusters, reducing cluster variance and increasing its likelihood through the prior.
- EM inference: Soft-assignment EM initializes with labeled points, creates clusters when new-cluster probability exceeds ϵ, and uses softmax assignments.
- Ablation results: 48.6% accuracy on mini-ImageNet and 98.0% accuracy on Omniglot result when the CRP prior is ablated, still below IMP performance.