Source-linked AI summary
Learning from Few Examples: A Summary of Approaches to Few-Shot Learning
Archit Parnami, Minwoo Lee
TL;DR
Few-Shot Learning addresses learning and generalization with only a few training examples, where data scarcity, privacy, and preparation costs limit conventional deep learning. The survey synthesizes representative approaches across meta-learning, transfer learning, and hybrid formulations, reporting substantial benchmark progress but identifying deployment boundaries across task formats and data domains.
Problem
Few-Shot Learning addresses the difficulty of learning patterns and generalizing when only a few labeled examples are available, while conventional deep learning relies on large amounts of labeled data.
Method
The survey organizes representative FSL algorithms by meta-learning, transfer learning, and hybrid approaches, including metric-, optimization-, and model-based methods.
Results
Accuracy on 5-way 1-shot miniImageNet classification increased from 43% with Matching Networks to 80% by January 2020, with no clear consensus on a single best approach.
Takeaways & Limitations
FSL can reduce the need for extensive labeled-data collection and training resources in settings where data is costly, private, or difficult to acquire.
Takeaways & Limitations
FSL remains challenging beyond images because audio and wireless-signal domains often lack the large, uniformly curated datasets required by meta-learning.
Abstract
from arXiv · showhide
Few-Shot Learning refers to the problem of learning the underlying pattern in the data just from a few training samples. Requiring a large number of data samples, many deep learning solutions suffer from data hunger and extensively high computation time and resources. Furthermore, data is often not available due to not only the nature of the problem or privacy concerns but also the cost of data preparation. Data collection, preprocessing, and labeling are strenuous human tasks. Therefore, few-shot learning that could drastically reduce the turnaround time of building machine learning applications emerges as a low-cost solution. This survey paper comprises a representative list of recently proposed few-shot learning algorithms. Given the learning dynamics and characteristics, the approaches to few-shot learning problems are discussed in the perspectives of meta-learning, transfer learning, and hybrid approaches (i.e., different variations of the few-shot learning problem).
1 Introduction
Few-Shot Learning addresses the difficulty of training high-capacity models when labeled data is scarce, motivated by data, privacy, and preparation constraints. The survey organizes recent approaches and discusses their structure and progress.
- Deep learning optimization can break down in the small-data regime because high-capacity models require many updates across labeled examples.
- Few-Shot Learning enables models to generalize from few training examples, including one-shot learning when only one example is available.
- FSL can reduce labeled-data collection, computation time, and training costs while addressing settings where data is difficult or impossible to acquire.
- The survey categorizes FSL approaches and covers meta-learning, non-meta-learning, hybrid approaches, datasets, performance, and open challenges.
2 Background
The survey introduces meta-learning as learning priors across tasks to support efficient learning on new tasks, then formalizes task distributions, train/test splits, and model optimization. It also distinguishes meta-learning from transfer, multi-task, and ensemble learning.
- Meta-learning learns priors from multiple tasks so a meta-learner can support solving a similar new task.
- Transfer learning retrains a model from a data-rich source task on a target task, whereas meta-learning gathers experience across multiple similar tasks.
- The survey groups FSL into meta-learning-based and non-meta-learning-based approaches, with metric-, optimization-, and model-based meta-learning as main categories.
- For each task, parameters are optimized on Dtrain and evaluated on Dtest using a loss that measures prediction error.
- Meta-learning partitions tasks into meta-training and meta-testing distributions, with corresponding datasets used for learning and evaluating generalization.
3 Few-Shot Learning
The survey frames recent FSL progress primarily through meta-learning while also covering non-meta-learning approaches and applications beyond image classification.
- Recent FSL progress has largely come through meta-learning, motivating a division between meta-learning-based and non-meta-learning-based approaches.
- The approaches discussed were mainly developed for few-shot image classification but remain applicable to regression, detection, segmentation, recommendation, and reinforcement learning.
3.1 The Few-Shot Classification Problem
Few-shot classification seeks good generalization from very few labeled examples in an M-way-K-shot task. The survey defines its train/test formulation, meta-learning setup, loss, and commonly used datasets.
- Few-shot classification becomes difficult when the small training set prevents a model from approximating a function that generalizes well to Dtest.
- Meta-training uses labeled test-set samples to optimize parameters, while meta-testing predicts labels for a new task with unseen categories.
- In an M-way-K-shot task, M denotes classes, K denotes examples per class, and |Dtrain| = M × K.
- Few-shot classification experiments commonly sample M-way-K-shot tasks from larger datasets containing more classes than each task.
3.2 Meta-Learning-based Few-Shot Learning
Meta-learning-based few-shot learning learns priors across few-shot tasks so models can generalize to new tasks with limited examples. The survey organizes these methods into metric-based, optimization-based, model-based, and hybrid approaches, detailing how they represent tasks, compare examples, or adapt parameters.
- Meta-learning objective: Meta-learning learns a prior across M-way-K-shot tasks so a model can solve a new task with few examples.Training samples tasks from a task distribution, then evaluates performance on newly sampled few-shot tasks.
- Main approaches: Meta-learning-based FSL comprises metric-based, optimization-based, and model-based approaches, plus hybrids for variants such as cross-domain and generalized FSL.The survey discusses the three main approaches and then considers hybrid approaches.
- Metric-based meta-learning: Metric-based methods learn embeddings and either use a predefined distance function or jointly learn a distance function to compare support and query examples.The metric-learning pipeline transforms high-dimensional inputs into lower-dimensional embeddings before computing distances or similarity scores.
- Training paradigm: Episodic training samples M-way-K-shot support and query sets, updates model parameters using average query-set error, and evaluates on test episodes.This setup mimics the testing scenario by repeatedly training and evaluating on task episodes.
- Metric-based methods: Metric-based methods include Siamese, Matching, Prototypical, Relation, and task-adaptive networks with different embeddings, similarity functions, prediction rules, and losses.Siamese networks compare shared-weight CNN representations; Matching Networks use cosine-similarity attention; Prototypical Networks classify by distances to class prototypes; Relation Networks learn a similarity score with another CNN.
- Metric-based methods: Task-adaptive metric methods adjust distance scaling and feature extraction to the task rather than relying entirely on task-independent embeddings and fixed metric choices.TADAM introduces a learnable temperature for metric scaling and a dynamic feature extractor conditioned on task information.
end
The survey reviews meta-learning methods that adapt models efficiently to few-shot tasks, including optimization-, transfer-, latent-space, memory-, and sequence-based approaches. It also highlights methods designed for task heterogeneity and cross-domain generalization.
- Optimization-based Meta-Learning: TAML meta-trains an unbiased initial model by preventing over-performance on some tasks or minimizing performance inequality across tasks.The goal is improved generalization to unseen tasks.
- Optimization-based Meta-Learning: MAML++ addresses MAML’s training instability, restricted generalization, reduced flexibility, computational overhead, and costly hyperparameter tuning.It aims to retain MAML’s flexibility while improving stability, efficiency, and generalization.
- Optimization-based Meta-Learning: HSML learns task representations and assigns different task clusters their own initial parameters to address task uncertainty and heterogeneity.This differs from globally sharing knowledge or learning one initialization for all tasks.
- Optimization-based Meta-Learning: CAVIA adapts only task-specific context parameters at test time while meta-training shared parameters, producing a low-dimensional task representation.This parameter partition is intended to make the extension more interpretable and less prone to overfitting.
- Hybrid Approaches: MTL uses a pretrained deep network as a feature extractor and meta-learns classifier parameters because adapting deep networks can overfit.Its architecture shifts most representation learning into pretraining.
- Hybrid Approaches: LEO optimizes model parameters in a low-dimensional latent embedding conditioned on task data, providing task-specific starting points and more effective adaptation.The approach is motivated by the difficulty of optimizing high-dimensional parameters in extreme low-data regimes.
- Model-based Meta-Learning: Model-based methods tailor architectures for fast learning, including external-memory systems such as MANN and NTM.NTM controllers read and write memory rows through soft attention, while LRUA writes to least-used or most-recently-used locations.
- Model-based Meta-Learning: SNAIL formulates meta-learning as sequence-to-sequence prediction over labeled examples followed by an unlabeled query, enabling reference to prior examples.Each episode predicts the query from the preceding example-label sequence.
3.3 Non-Meta-Learning based Few-Shot Learning
Non-meta-learning approaches primarily transfer representations from pretrained networks or exploit query-set structure to classify with limited labeled data. The survey also includes generative methods that expand few-shot training sets with synthesized examples.
- Transfer Learning: Transfer learning pretrains a deep network on base classes and fine-tunes it on a new few-shot classification task.This transfers knowledge when the available data is insufficient to train a deep network from scratch.
- Embedding-Based Methods: SimpleShot uses pretrained-network embeddings, centers and L2-normalizes them, and classifies with Euclidean nearest-neighbor distance.The method replaces meta-learned feature extraction with a pretrained deep network.
- Embedding-Based Methods: Another approach reuses pretrained representations and trains a new classifier because limited samples produce poor representations when training from scratch.The cited method also applies L2 normalization before classifier training.
- Transductive Inference: Transductive fine-tuning uses labeled support examples and unlabeled query samples, with an entropy-minimizing regularizer on query predictions.The objective combines support-data fitting with query-set regularization.
- Transductive Inference: Laplacian-regularized inference combines nearest-prototype assignment with a pairwise term encouraging nearby points to share latent labels.The unary term uses support-derived prototypes, while the Laplacian term uses point similarity.
- Transfer Learning: Transfer-learning methods often achieve better or equivalent performance on few-shot tasks compared with complex meta-learning methods.The passage states this as a broad performance comparison without identifying one universally best method.
- Generative Few-Shot Learning: Generative few-shot learning uses a hallucinator to map real examples to synthesized examples and expand the training set for a learner.The few-shot training set is fed to the hallucinator before learner training.
4 Progress in Few-Shot Learning
Few-shot learning accuracy improved substantially across several method families during the survey’s reported period. However, the survey finds no clear consensus that one approach is best.
- Applications: Few-shot learning research expanded beyond image classification to object detection, segmentation, fault diagnosis, and text classification.Early work concentrated mainly on computer vision and image classification.
- Accuracy Progress: 43% to 80%: accuracy on miniImageNet 5-way 1-shot classification rose from Matching Networks to methods reported by January 2020.The progress includes optimization-, metric-, model-, hybrid-, and non-meta-learning approaches.
- Accuracy Progress: No single approach is clearly established as best, while metric-, optimization-, hybrid-, and non-meta-learning methods all appear competitive.The survey notes comparatively less progress for model-based approaches.
5 Challenges and Open Problems
The survey identifies scope and deployment challenges arising from episodic assumptions, task-distribution mismatch, retention of seen classes, and limited non-image datasets. These boundaries motivate broader few-shot learning settings.
- Training Assumptions: M-way K-shot episodic training makes classifiers rigid when deployment has unknown class counts or support-set sizes, and fewer than K examples can degrade performance.The training and testing setup assumes exactly M classes with exactly K examples each.
- Task Distribution: Sampling training and testing tasks from one distribution constrains learning to a single domain, motivating cross-domain few-shot learning.Examples include transfer from Omniglot to MNIST and from miniImageNet to fine-grained Birds or Cars.
- Seen and Unseen Classes: Conventional few-shot classifiers discard training classes after training and classify queries only among support-set classes, creating generalized few-shot learning as a challenge.The desired setting jointly classifies seen and unseen classes.
- Non-Image Domains: Few-shot methods developed for images are difficult to deploy on audio and wireless signals because these domains lack similarly large, uniformly curated datasets.Dataset uniformity is identified as essential to meta-learning in the cited discussion.