Source-linked AI summary
Meta-learning approaches for few-shot learning: A survey of recent advances
Hassan Gharoun, Fereshteh Momenifar, Fang Chen, Amir H. Gandomi
TL;DR
Deep learning often struggles with distribution shifts and generalization from few samples, motivating methods that can adapt to unseen tasks. This survey reviews meta-learning approaches across metric-based, memory-based, and learning-based categories, then synthesizes benchmark comparisons and current challenges. The reviewed literature includes strong benchmark results, while identified limitations include hyperparameter-tuning costs and dataset dependence in specific methods.
Problem
Deep learning models often assume matching training and testing distributions and require many samples, limiting generalization to unseen tasks and few-shot settings.
Method
The survey organizes and reviews meta-learning methods across metric-based, memory-based, and learning-based approaches, including learned initializations, parameters, and optimizers.
Results
The survey presents benchmark comparisons and reports that reviewed methods achieve strong performance across established few-shot learning benchmarks.
Takeaways & Limitations
Meta-learning provides a framework for transferring prior task knowledge so models can adapt to new tasks quickly from few data points.
Takeaways & Limitations
MAML can require expensive tuning of its α and β hyperparameters, while the Neural Statistician’s performance depends on a large dataset.
Abstract
from arXiv · showhide
Despite its astounding success in learning deeper multi-dimensional data, the performance of deep learning declines on new unseen tasks mainly due to its focus on same-distribution prediction. Moreover, deep learning is notorious for poor generalization from few samples. Meta-learning is a promising approach that addresses these issues by adapting to new tasks with few-shot datasets. This survey first briefly introduces meta-learning and then investigates state-of-the-art meta-learning methods and recent advances in: (I) metric-based, (II) memory-based, (III), and learning-based methods. Finally, current challenges and insights for future researches are discussed.
1 Introduction
Traditional machine and deep learning models often require many samples and assume matching training and testing distributions, limiting adaptation to unseen tasks. Meta-learning addresses this by transferring knowledge across tasks to adapt quickly from few examples, and this survey reviews methods, benchmarks, and challenges.
- Traditional ML and DL train task-specific models from scratch and often require many samples to generalize.Collecting additional data can be costly, time-consuming, or infeasible in real-world settings.
- Assuming identical training and testing distributions causes performance to suffer under distribution shifts.Training on diverse data alone does not guarantee domain adaptation.
- Transfer learning reuses knowledge from related source domains, but pretrained networks can deteriorate when fine-tuned with few or less-similar samples.Transferred knowledge may include model parameters, feature representations, and instances.
- Meta-learning transfers generic accumulated knowledge from prior tasks so models adapt quickly to new tasks with few data points without training from scratch.Its learning level is elevated from individual data examples to families of tasks.
- The survey organizes state-of-the-art methods by algorithm mechanics, compares reviewed methods on benchmarks, and discusses challenges and future directions.It covers preliminary concepts, method categories, benchmark comparisons, and limitations.
2 Meta-Learning Preliminary Terminology and Training Structure
Meta-learning separates data into meta-training and meta-testing sets, each containing support and query sets for episodic learning. Across episodes, models learn from small support datasets and rapidly adapt to new tasks before evaluation on query samples.
- Meta-learning divides data into non-overlapping meta-training and meta-testing sets, each subdivided into support and query sets.The support set is the inner training set, while the query set is the inner test set.
- Each episode samples an N-way k-shot support set and q query samples for every support-set category.Episodes repeatedly construct these support and query sets for meta-learning.
- During meta-learning, the model predicts on query data and updates from prediction error across episodes.Repeated episodes train the model to learn from small datasets.
- During adaptation, the model uses meta-test support data to build a classifier and learns task-specific parameters quickly.Performance is then evaluated using the corresponding meta-test query set.
- The survey compares methods using Omniglot, miniImageNet, and other datasets, with the three principal benchmarks used for performance comparison.Oxford-102, CIFAR-10, and MNIST are identified as less-used alternatives.
3 Meta-Learning
The survey organizes meta-learning research into metric-based, memory-based, and learning-based methods. Learning-based methods are further divided by whether they learn an initialization, parameters, or an optimizer.
- The survey reviews cutting-edge meta-learning studies in three main categories.These categories are illustrated in the paper’s method taxonomy.
- Metric-based methods are one principal category of meta-learning approaches.The taxonomy places metric-based methods alongside memory-based and learning-based methods.
- Memory-based methods form a second principal category in the survey’s organization.They are listed separately from metric-based and learning-based approaches.
- Learning-based methods are subdivided into learning the initialization, parameters, and optimizer.This subdivision groups approaches by the component whose adaptation is learned.
3.1 Metric-based methods
Metric-based meta-learning methods predict query labels by comparing learned representations with support-set examples or class prototypes. Recent advances extend these methods to model uncertainty, domain shifts, intra-class variation, multi-label and unseen classes, richer local features, and harder training episodes.
- Siamese Network: Siamese networks learn matching through shared embedding functions and an energy function that produces similarity scores for input pairs.They frame one-shot recognition as a matching task rather than direct classification.
- Prototypical Networks: Gaussian prototypical networks incorporate embedding covariance to form confidence regions and variance-weighted class prototypes.Query-to-prototype distances use covariance information in addition to embedding vectors.
- Prototypical Networks: Transferable prototypical networks reduce source–target embedding shifts by combining prototype-based classification with domain-level discrepancy minimization.They use labeled source data and unlabeled target data to construct domain-invariant representations.
- Empirical Advances: Several reviewed approaches report improved performance over generic baselines, including domain-aligned prototypes and improved prototypical networks.The supplied passages explicitly report that the domain-alignment model outperforms generic prototypical networks and that IPN does likewise.
- Prototypical Networks: Prototypical-network variants address shortcomings of single or equally weighted prototypes through weighted prototypes, multi-label representations, dummy prototypes, and multiple local-descriptor prototypes.These variants target intra-class distributions, multi-label ambiguity, unseen classes, and spatially structured features.
- Matching and Relation Networks: Other metric-based advances improve representation comparison by using spatial feature matching, memory-augmented information, and difficulty-aware sampling.These methods respectively emphasize feature positions, nearest-neighbor information, and challenging label sets during episodic training.
3.2 Memory-based Methods
Memory-based meta-learning adapts to tasks by retrieving information from prior inputs through internal or external memory. The surveyed methods use differentiable memory operations, specialized addressing, and attention or weighted prototypes to improve task-specific classification.
- Memory-based Methods: Memory-based meta-learners use internal or external memory to retrieve information from previous inputs and adapt dynamically to presented tasks.Neural Turing Machines use a controller and memory bank connected through selective differentiable read and write operations.
- Memory-based Methods: Neural Turing Machines use content-based or location-based addressing, with location-based access iterating across memory locations to support generalization.Content-based addressing matches current and controller values, whereas location-based addressing accesses locations rather than content.
- Memory-based Methods: MANN combines content-based reading with least recently used access writing to store coupled representation–class-label information for later classification.LRUA writes to the most recently used memory location, enabling retrieval when a sample from a seen class is later presented.
- Memory-based Methods: SNAIL and CNPs avoid external memory by combining temporal convolution with soft attention or by separating meta-learning and task-learning components.SNAIL provides high-bandwidth temporal access and targeted soft-attention access, while CNPs use a meta-learner and task-learner.
- Memory-based Methods: Memory-augmented matching networks use distance-weighted class prototypes when skewed support distributions make mean-based prototypes biased.Each sample’s weight is based on the inverse of its total distance to other samples in the same class.
3.3 Learning-based Methods
Learning-based methods train meta-learners to improve how base learners adapt to unseen tasks, by learning initializations, parameters, or optimizers. The survey covers methods ranging from MAML and LEO to parameter-generating models and differentiable optimization-based learners.
- Taxonomy: Learning-based methods comprise learning the initialization, learning the parameters, and learning the optimizer.These groups were previously described as model-based or optimization-based methods, but the survey distinguishes them mechanically.
- Learning the initialization: MAML learns a shared global initializer so new tasks can adapt with fewer data points and optimization steps.Its inner loop obtains task-specific parameters, while the outer loop updates the shared initialization across tasks.
- Learning the initialization: PLATIPUS extends probabilistic MAML with variational inference to capture uncertainty across multimodal task distributions.It interprets MAML as approximate inference over support-set parameter posteriors.
- Learning the initialization: LEO reduces the dimensionality of the embedding space while retaining inner-loop task adaptation and outer-loop global updates.The method addresses the difficulty of learning in high-dimensional parameter spaces.
- Limitations and advances: MAML requires tuning α and β, while BOIL changes the inner-loop update emphasis from representation reuse toward representation change.Hypergradient descent is described as a way to adapt MAML’s hyperparameters through backpropagation.
- Learning the parameters: Parameter-learning methods generate task-specific base-learner parameters with a meta-learner rather than initializing them randomly.Neural Statistician estimates task-parameter posteriors from dataset statistics, whereas MetaNet processes meta-information to generate weights quickly.
- Learning the optimizer: Optimizer-learning methods can use a machine-learning classifier as the base learner; MetaOptNet combines an SVM with a differentiable quadratic-programming solver.The reported method learns embedding-function parameters and outperformed R2-D2 and LR-D2.
4 Overview, Challenges and Future Directions
The survey reviews state-of-the-art meta-learning methods and reports benchmark accuracy results, then identifies unresolved challenges involving performance uncertainty, stability, domain transfer, representation learning, and computational cost.
- Overview: Meta-learning remains an active area despite promising achievements, with benchmark methods organized into metric-based, memory-based, and learning-based categories.The benchmark table distinguishes metric-based methods, memory-based methods, and several learning-based categories; the survey describes meta-learning as still in its infancy.
- Overview: Benchmark results cover Omniglot, MiniImageNet, and CUB-200-2011, with reported accuracies and 95% confidence intervals where available.Unreported values are marked with “-”, and the table notes that MANN performance was extracted from another source.
- Challenges and Future Directions: The survey identifies uncertainty in novel-class performance because unrelated or weakly related episodic-training tasks can produce divergent optimization directions and disparate performance.It suggests using more diversified tasks and grouping more similar tasks within episodes.
- Challenges and Future Directions: Episodic training can cause catastrophic forgetting and underfitting on base classes, motivating combinations of memory-based and metric-based methods.The stated goal is to improve performance on both base and unseen classes.
- Challenges and Future Directions: Metric-based methods are sensitive to dataset complexity: GCR performed best on Omniglot, whereas metric-based models did not compete with non-metric-based methods on MiniImageNet.The survey also notes that MAML is not robust to adversarial samples and points to further adversarial-sample research.
- Challenges and Future Directions: As the difference between base and novel domains increases, meta-learning performance generally declines, motivating multi-domain and cross-domain benchmarks and methods.The survey notes that current methods are trained on a single base domain and discusses memory-based methods or related-task grouping as possible directions.
- Challenges and Future Directions: The effects of embedding-function architecture and backbone choice remain insufficiently explored, including their relationship to performance and computational cost.Most methods use four-layer convolutional backbones, while relatively few studies use larger or multi-scale networks.
- Challenges and Future Directions: Reducing computational cost remains an important research direction because learning-based methods can be computationally expensive despite using few samples.The survey presents computational efficiency as an open issue for future work.