A Survey on Machine Learning from Few Samples

1 INTRODUCTION

• Scarce labeled data in fields such as medicine, military, and finance motivates FSL, while vanilla learning on few samples risks serious overfitting.The survey also notes FSL’s potential for lower-cost data preparation and quicker deployment on emerging tasks.Source: §1 · pp. 1–2
• The survey reviews 300+ FSL papers from the 2000s to 2019, from the early Congealing model through recent meta-learning approaches.The authors present this breadth as a way to capture FSL’s development and build a knowledge hierarchy.Source: §1 · p. 2
• Its hierarchical taxonomy groups FSL approaches by modeling principle into generative-model-based and discriminative-model-based approaches.Within each group, the survey further categorizes approaches by generalizable properties.Source: §1 · p. 2
• The survey emphasizes meta-learning approaches and divides them into five classes: Learn-to-Measure, Learn-to-Finetune, Learn-to-Parameterize, Learn-to-Adjust, and Learn-to-Remember.It also describes development relationships among these approaches.Source: §1 · p. 2
• It reviews emerging extensions beyond vanilla FSL, including semi-supervised, unsupervised, cross-domain, generalized, and multimodal FSL.The survey notes that previous reviews rarely covered these topics.Source: §1 · p. 2
• The survey summarizes FSL applications across computer vision, natural language processing, audio and speech, reinforcement learning and robotics, and data analysis, alongside benchmark performance.The authors aim to provide a reference for follow-up research in areas not studied by previous reviews.Source: §1 · p. 2

2 OVERVIEW

• The overview lays out FSL’s evolution, notation and definitions, and taxonomy of existing approaches.Source: §2 · p. 2

2.1 Evolution History

• FSL research moves from early generative approaches to deep learning after 2015, when discriminative methods and meta learning become prominent.Early milestones include Congealing, the Variational Bayesian framework, and Bayesian Program Learning; the 2015 Siamese CNN marks the deep period.Source: §2.1 · pp. 2–3

2.2 Notations and Definitions

• FSL uses a task-specific dataset with few supervised examples and may use an auxiliary dataset whose classes are irrelevant to the task.When the auxiliary data cover the task classes, the problem becomes traditional large-sample learning.Source: §2.2 · p. 4

2.3 Taxonomy

• The taxonomy separates generative methods, which model p(x|y), from discriminative methods, which directly model p(y|x).Discriminative approaches include augmentation, metric learning, and meta learning; the latter is divided into Learn-to-Measure, Learn-to-Finetune, Learn-to-Parameterize, Learn-to-Adjust, and Learn-to-Remember.Source: §2.3 · p. 4

• Generative FSL methods introduce latent variables to bridge inputs and targets and compute p(x|y), often relying on assumptions about the latent-variable distribution.The latent variable’s form varies across methods.Source: §3 · p. 4
• Bayesian Program Learning models character generation as a probabilistic program, while compositional patch models reconstruct test images from components and an AND-OR graph.BPL’s program represents primitives through images and supports one-shot classification and generation; compositional patch models assume same-class images share patch-based structure.Source: §3 · p. 5
• Chopping uses random auxiliary-data splits, Neural Statistician uses latent Gaussian statistics, and VBF and hierarchical Bayes use constellation and superclass structures, respectively.Source: §3 · p. 5
• The surveyed generative approaches construct latent variables in different ways, but most early methods are tailored to particular tasks or data and lack scalability to general cases.Early evaluations also used differing datasets and settings, without comparable benchmarks.Source: §3 · pp. 5–6, Table 1 · p. 5

• Augmentation methods learn a general function from auxiliary data to augment task-specific samples or their features.Source: §4 · p. 6
• Metric-learning methods learn pairwise similarity on auxiliary data and use nearest-neighbor classification for prediction.Source: §4 · p. 6
• Meta-learning methods create auxiliary-data tasks resembling the target task and train across tasks to distill transferable models, algorithms, or parameters.The survey also discusses other discriminative approaches.Source: §4 · p. 6

4.1.1 Supervised Augmentation

• Supervised augmentation maps between feature and side-information spaces, and its labeling costs make it better suited to tasks or datasets with side information.The mapping direction and module vary across approaches.Source: §4.1.1 · pp. 6, 8
• FTT uses consecutive scene attributes to synthesize features by varying attribute strength, while AGA transfers features between explicit attribute strengths.FTT maps side information to features; AGA maps features to features, using an encoder-decoder module.Source: §4.1.1 · pp. 6–7
• Dual TriNet, decodes sampled semantic vectors, AT generates features from class attributes, and ABS-Net samples pseudo-features from an attribute-feature repository.Dual TriNet uses a semantic-vector bottleneck; AT models image-attribute relationships with a topic model.Source: §4.1.1 · p. 7

4.1.2 Unsupervised Augmentation

• Unsupervised augmentation mines intra-class variation without side information; SH uses quadruplet analogies and an MLP to synthesize features, even from one task-class example.The analogies are mined from an auxiliary set; ∆-encoder instead uses an encoder-decoder bottleneck to capture transferable variation.Source: §4.1.2 · p. 8
• Hallucinator generates augmented features from an original feature and noise, as a plug-and-play module for meta-learning methods.DAGAN and IDeMe-Net also coordinate with ready-made meta-learning approaches; CP-ANN uses GAN-based set-to-set translation.Source: §4.1.2 · p. 8

4.1.3 Discussion

• Augmentation approaches complement metric- and meta-learning approaches: augment task training data first, then apply another FSL method.The survey describes augmentation strategies as usable plug-and-play modules rather than competing approaches.Source: §4.1.3 · p. 8

Metric-learning framework

• Metric-learning FSL learns a pairwise similarity metric on auxiliary data and applies it to novel classes, where nearest-neighbor classification can make predictions.The learned metric may be a distance, network, or other module that estimates similarity between samples or features.Source: §4.2 · p. 9

Distance and kernel methods

• CRM learns a Mahalanobis distance for nearest-neighbor classification, while KernelBoost combines weak GMM-based kernels into a pairwise distance.CRM learns its positive-semidefinite distance matrix from auxiliary data; KernelBoost optimizes its combined kernel with an exponential loss.Source: §4.2 · p. 9

Neural metric methods

• Siamese Nets learn a shared-weight CNN similarity from sample pairs, while Triplet Ranking Nets [98] and SRPN [99] extend or modify pairwise metric learning.SRPN adds a skip-residual network and GAN-based regularization.Source: §4.2 · p. 9, Figure 10 · p. 10

Memory and distribution-based methods

• MM stores class-labeled keys and uses a test feature as a query, whereas AdaptHistLoss learns from positive- and negative-pair similarity distributions.Histogram loss measures the reverse probability that a random negative pair is more similar than a random positive pair.Source: §4.2 · pp. 9–10

4.3 Meta Learning

• Meta-learning-based few-shot methods comprise five categories, distinguishing what the strategy learns: similarity, fine-tuning, task-specific parameters, sample-specific adjustments, or memory.The categories are Learn-to-Measure (L2M), Learn-to-Finetune (L2F), Learn-to-Parameterize (L2P), Learn-to-Adjust (L2A), and Learn-to-Remember (L2R).Source: §4.3 · p. 10

4.3.1 Learn-to-Measure

• L2M approaches use meta-learning to learn a similarity metric intended to transfer across tasks; Matching Nets, Prototypical Nets, and Relation Net are milestone members of this category.Matching Nets measure cosine similarity between query and support embeddings, while Relation Net uses a learnable CNN to measure pairwise similarity.Source: §4.3.1 · pp. 10, 12

4.3.2 Learn-to-Finetune

• L2F adapts a base learner to each task using few support samples, while a meta-learner is trained across tasks to enable rapid learning and generalization.MAML meta-learns an initialization for rapid adaptation; Meta-Learner LSTM instead uses an LSTM to update base-learner parameters from support-sample losses and gradients.Source: §4.3.2 · pp. 12–13

4.3.3 Learn-to-Parameterize

• L2P uses a task-specific parameter generator, trained alongside the base learner, to parameterize the whole learner or selected parts from a task’s few support samples.For example, LGM-Net generates base-learner convolutional weights from the average support-sample embedding, while other methods parameterize task-specific predictors or feature layers.Source: §4.3.3 · p. 14

4.3.4 Learn-to-Adjust

• L2A makes lighter, sample-specific adjustments to a base learner’s computation flow or nodes, unlike L2P’s task-specific parameterization.Methods adjust different components: MetaNet uses fast-weight layers, CSNs shifts neuron pre-activations, and MetaHebb [163] adds an auxiliary fast-weight matrix.Source: §4.3.4 · p. 15
• FEAT adapts support-sample embeddings to be more discriminative for each task, investigating BiLSTM, DeepSets, GCN, and Transformer set-to-set functions.Source: §4.3.4 · p. 15

4.3.5 Learn-to-Remember

• L2R models support samples as a sequence for queries to match prior information, using temporal networks such as MANN, ARCs, SNAIL, and APL.Their designs include memory, attention-based RNN, temporal convolution with soft attention, and a surprise-based memory network, respectively.Source: §4.3.5 · p. 15

4.3.6 Discussion

• L2M is way- and shot-agnostic, while L2F requires support-sample fine-tuning on each new task that can prolong adaptation.Source: §4.3.6 · p. 15
• L2P and L2A use separate meta learners to generate parameters or adjustments, increasing model size and training difficulty; L2A also struggles with more support samples due to sequence-learning dependence limits.Source: §4.3.6 · pp. 15–16

Multi-task learning

• Multi-task FSL shares upstream representations across tasks while keeping downstream modules task-specific, with variants using parameter regularization, task-conditioning GANs, or attribute prediction.MetaGAN uses generated fake samples as an auxiliary task; attribute prediction is jointly optimized with the main task loss.Source: §4.4 · p. 16

Self-supervised learning

• Self-supervised FSL adds auxiliary tasks such as rotation prediction and jigsaw puzzles to learn representations, and these approaches remain within multi-task learning.Some methods use self-supervision with labeled support samples; others combine it with the supervised Prototypical Nets loss or semi-supervised tasks.Source: §4.4 · p. 16, Figure 23 · p. 16

Other perspectives

• Transductive Propagation Network propagates labels across all query samples when they arrive together, while other approaches use machine-labeled web images or inter-class concept information.The survey also lists feature replacement, model adaptation, knowledge distillation, and saliency-guided data hallucination among approaches from other perspectives.Source: §4.4 · p. 16

Analysis of existing methods

• Analyses of existing FSL methods examine base-network depth and report that simple feature preprocessing, including mean subtraction and L2 normalization, can improve metric-learning performance.The depth comparisons cover Prototypical Nets, Matching Nets, Relation Net, and MAML.Source: §4.4 · p. 16

5.1 Semi-supervised Few Sample Learning

• Semi-supervised FSL adds unlabeled samples that may or may not belong to the task classes alongside the labeled support set.Methods include semi-supervised clustering, auxiliary self-supervised losses, pseudo-label self-training, and task-oriented models for question answering or disease prediction.Source: §5.1 · pp. 16–17, Figure 24 · p. 17

5.2 Unsupervised Few Sample Learning

• Unsupervised FSL uses a fully unlabeled auxiliary set, with approaches either pretraining representations or synthesizing pseudo-labeled meta-training tasks for existing FSL models.CACTUs uses unsupervised representation learning and clustering; UMTRA and AAL pair augmented samples as congeners before applying MAML.Source: §5.2 · p. 17

5.3 Cross-domain Few Sample Learning

• Cross-domain FSL uses auxiliary data from a different domain when relevant same-domain samples are unavailable, and the domain shift makes the task more challenging.Unlike few-sample domain adaptation, C-FSL has disjoint label spaces between auxiliary and task-specific datasets; surveyed approaches include adversarial training, feature transformation, alignment, and finetuning.Source: §5.3 · p. 17, Table 6 · p. 17

5.4 Generalized Few Sample Learning

• Generalized FSL jointly handles auxiliary-set and task-specific classes, addressing the risk that vanilla FSL forgets earlier classes as new tasks arrive.Approaches include augmenting sparse novel-class samples, modeling relationships among classes, and incrementally generating weights for novel classes.Source: §5.4 · pp. 17–18

5.5 Multimodal Few Sample Learning

• Multimodal FSL either maps between modalities, such as image and sentence, or fuses additional-modal information to help learning in the task modality.Fusion sources include word vectors, captions, attributes, class hierarchies, and cross-style datasets; these modalities provide additional prior knowledge.Source: §5.5 · p. 18, Figure 26 · p. 18

Computer Vision

• More than 20%: 5-way 1/5-shot accuracy on miniImageNet increased by more than 20% from 2016 to 2019, while Omniglot results approached saturation.The survey compiles reported miniImageNet accuracy results from existing FSL approaches.Source: Table 8 · p. 19, §6 · p. 20

Vision and Natural Language Processing

• FSL applications span image classification, segmentation, detection, and video tasks; NLP applications include text classification, word representation, relation learning, and knowledge graphs.The survey identifies computer vision as the main FSL testbed and NLP as its second-largest application field.Source: §6 · pp. 18, 20

Audio, Speech, Reinforcement Learning, and Robotics

• FSL addresses audio classification, keyword recognition, text-to-speech, and speech generation, where large-scale data collection and annotation are more difficult than for images or text.Robotics and reinforcement-learning applications include imitation learning, visual navigation, and policy learning.Source: §6 · p. 20

Data Analysis and Other Applications

• Data-analysis applications include regression and anomaly detection, while other professional areas include medicine, chemometrics, agriculture, sensors, and internet security.Source: §6 · p. 20

Cross-Field Applications

• Few-shot image-captioning and visual-question-answering tasks use a small number of image-text training pairs to generate descriptions or answer textual questions about images.Source: §6 · p. 20

Open Competitions

• A cross-domain few-shot challenge trains on ImageNet and evaluates on four other domains, with tracks that do or do not use unlabeled target-domain images.The named evaluation domains include plant disease, satellite, dermoscopic skin-lesion, and X-ray images.Source: §6 · p. 20, Figure 27 · p. 20

Robustness

• Robust FSL should address noisy or mislabeled samples and domain shifts between auxiliary and task-specific data.Source: §7 · pp. 20–21

Universality

• FSL needs model-level scalability beyond benchmark-specific tasks and application flexibility beyond small class sets with large labeled auxiliary datasets.Source: §7 · p. 21

Interpretability

• FSL interpretability may be enhanced by combining external prior knowledge with internal data knowledge, including compositional relationships and structural correspondences.Source: §7 · p. 21

Theoretical System

• A systematic theory could frame FSL methods as regularizing the function space: augmentation adds samples, while meta-learning regularizes across tasks.Source: §7 · p. 21

• The survey reviews FSL's evolution and advances, groups approaches in a taxonomy, and analyzes relationships among mainstream meta-learning methods.Source: §8 · p. 21
• It also summarizes emerging FSL topics, applications, benchmark datasets and performance, and potential research directions.Source: §8 · p. 21
• The authors expect the survey to promote understanding of FSL knowledge and collaborative development of the research area.Source: §8 · p. 21