PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalizing from a Few Examples: A Survey on Few-Shot Learning

Yaqing Wang, Quanming Yao, James Kwok, Lionel M. Ni

arXiv:1904.05046v3cs.LGcs.AI

TL;DR

Few-Shot Learning addresses the difficulty of learning new tasks from limited supervised data, where empirical risk minimization can become unreliable. The survey formalizes FSL, distinguishes it from related problems, and organizes methods by data, model, and algorithm perspectives. It synthesizes the field’s methods, trade-offs, and future directions while identifying boundaries involving task specificity and changing task distributions.

  • Problem

    Few-Shot Learning is needed because conventional machine-learning applications often require many supervised examples, while limited training data makes the empirical risk minimizer unreliable and prone to overfitting.

  • Method

    The paper conducts a comprehensive survey that formally defines FSL, compares related learning problems, and categorizes methods by data augmentation, hypothesis-space reduction, and algorithmic search.

  • Results

    The survey provides a systematic taxonomy of FSL methods, analyzes the core unreliable empirical risk minimizer, and discusses the pros and cons of each category.

  • Takeaways & Limitations

    FSL serves as a test-bed for AI, supports learning for rare cases, and can reduce the burden of collecting large-scale supervised data.

  • Takeaways & Limitations

    Task-invariant embeddings may be unsuitable when task specialty causes the target task to have few examples, and fixed task distributions do not capture streaming tasks.

Abstract

from arXiv · show

Machine learning has been highly successful in data-intensive applications but is often hampered when the data set is small. Recently, Few-Shot Learning (FSL) is proposed to tackle this problem. Using prior knowledge, FSL can rapidly generalize to new tasks containing only a few samples with supervised information. In this paper, we conduct a thorough survey to fully understand FSL. Starting from a formal definition of FSL, we distinguish FSL from several relevant machine learning problems. We then point out that the core issue in FSL is that the empirical risk minimized is unreliable. Based on how prior knowledge can be used to handle this core issue, we categorize FSL methods from three perspectives: (i) data, which uses prior knowledge to augment the supervised experience; (ii) model, which uses prior knowledge to reduce the size of the hypothesis space; and (iii) algorithm, which uses prior knowledge to alter the search for the best hypothesis in the given hypothesis space. With this taxonomy, we review and discuss the pros and cons of each category. Promising directions, in the aspects of the FSL problem setups, techniques, applications and theories, are also proposed to provide insights for future research.

1 INTRODUCTION

Few-Shot Learning addresses the difficulty of generalizing from limited supervised examples by incorporating prior knowledge. This survey formalizes the problem, identifies its unreliable empirical risk minimizer as the core issue, and organizes methods by data, model, and algorithm.

  • Motivation: Few-Shot Learning targets rapid generalization to new tasks from only a few supervised examples by using prior knowledge.The paper presents FSL as a way to learn new tasks with limited supervised information, motivated by human-like learning and difficult data-collection settings.
  • Motivation: FSL can reduce supervised data-gathering effort in applications such as image classification, retrieval, tracking, recognition, captioning, and visual question answering.It is especially useful when labeled examples are laborious, privacy-sensitive, unsafe, or ethically difficult to obtain.
  • Survey Scope: The survey fills a gap by providing an organized taxonomy that connects existing FSL methods and explains why they work.It also distinguishes FSL from related learning problems and extends beyond a prior survey focused on concept learning and experience learning for small samples.
  • Survey Scope: The survey identifies the unreliable empirical risk minimizer as the core issue in FSL supervised learning.Its contributions include formally defining FSL, comparing related problems, and analyzing this issue using machine-learning error decomposition.
  • Taxonomy: FSL methods are organized by how prior knowledge is used: augmenting data, reducing the hypothesis space, or altering the algorithm’s search strategy.The survey reviews each category, discusses its advantages and disadvantages, and proposes future directions across problem setups, techniques, applications, and theories.

2 OVERVIEW

FSL addresses learning tasks with limited supervised information by combining available examples with prior knowledge. The survey formalizes FSL, distinguishes it from related problems, identifies unreliable empirical risk minimization as its core difficulty, and organizes solutions by data, model, and algorithm.

  • Problem Definition: The survey formalizes FSL and compares it with weakly supervised, imbalanced, transfer, and other related learning problems.It explains that FSL may use pre-trained models, supervised data from other domains or modalities, and can include reinforcement learning, unlike narrower related settings.
  • Problem Definition: FSL targets good learning performance when the training set contains only a limited number of supervised input-output examples.The paper defines FSL as a special case of machine learning focused on limited supervised information.
  • Unreliable Empirical Risk Minimizer: In supervised FSL, few examples make empirical risk a poor proxy for expected risk, so the empirical risk minimizer may overfit and become unreliable.This unreliable empirical risk minimizer is identified as the core issue that makes FSL harder.

3 DATA

Data-perspective FSL methods use prior knowledge to enrich the few-shot training set, making the supervised experience more sufficient for learning. Strategies include transforming existing samples, selecting weakly labeled or unlabeled samples, and aggregating samples from similar larger datasets, but augmentation policies remain domain-dependent.

  • Data augmentation: Prior knowledge augments Dtrain so the enriched sample set supports a more reliable hypothesis.The survey presents this approach as solving FSL by increasing the supervised information available for learning.
  • Data augmentation: Hand-crafted transformations introduce invariances such as translation, flipping, scaling, cropping, and rotation.These rules are typically applied during preprocessing and rely heavily on domain knowledge.
  • Transforming samples from weakly labeled or unlabeled data: Weakly labeled or unlabeled datasets can supply target-label samples, reducing annotation effort while introducing sample-selection challenges.Examples include exemplar SVMs, label propagation, and progressive selection with pseudo-labels.
  • Aggregating samples from similar datasets: Samples from similar larger datasets can be aggregated and adapted, but unsuitable source samples may mislead augmentation.The aggregation weight is commonly based on sample similarity, with generative models used to address mismatched samples.
  • Choosing an augmentation strategy: Augmentation strategy depends on whether weakly supervised, unlabeled, transformed, or similar-domain data are available for the target task.The survey relates this choice to annotation and computational costs.
  • Discussion and summary: Data augmentation is straightforward, but policies are often dataset-specific and transfer poorly across domains, especially beyond images.Text and audio are harder to generate because they involve syntax and structure.

4 MODEL

Model-perspective methods address FSL by constraining the hypothesis space using prior knowledge. A small hypothesis space reduces sample demands but may lack the capacity for complicated real-world problems, motivating structured model restrictions.

  • Model perspective: FSL model methods constrain H to a smaller hypothesis space so learning remains feasible with few samples.The survey organizes this perspective around reducing the size of the hypothesis space.
  • Model perspective: Choosing a small H with simple models can be inadequate for complicated real-world problems and produce a large approximation error.The survey therefore favors a hypothesis space that is restricted but still expressive enough for the task.
  • Model perspective: Model-perspective methods are further classified into four types according to the prior knowledge they use.The detailed characteristics are summarized in Table 4.

4.1 Multitask Learning

Multitask learning uses related source and target tasks jointly, allowing few-shot tasks to benefit from information in tasks with more data. The survey distinguishes parameter sharing from parameter tying as two ways to constrain task models.

  • Multitask Learning: Multitask learning jointly exploits task-generic and task-specific information across related tasks, including few-shot target tasks and larger source tasks.The target tasks have few samples, while source tasks provide larger datasets.
  • Multitask Learning: Joint learning constrains each task’s parameters through the other tasks, and methods are divided into parameter sharing and parameter tying.This classification is based on how task parameters are constrained.
  • Parameter sharing: Parameter sharing directly reuses some parameters across task networks while retaining task-specific components for different outputs.Examples share early layers or a single embedding function, with later processing adapted to task differences.
  • Parameter tying: Parameter tying encourages task parameters to be similar, commonly through regularization penalties on differences between them.Examples penalize pairwise parameter differences or align corresponding CNN layers.

4.2 Embedding Learning

Embedding-learning methods map samples into a lower-dimensional space where similarity-based prediction can use a smaller hypothesis space. The survey contrasts task-specific, task-invariant, and hybrid embeddings, balancing adaptation to the current task against computational cost.

  • Embedding Learning: Embedding learning maps samples into a lower-dimensional space where similar samples are closer and fewer training samples may suffice.The embedding is learned mainly from prior knowledge and can also use task-specific training data.
  • Embedding Learning: Prediction compares an embedded test sample with embedded training samples using a similarity function and assigns the most similar sample’s class.The framework uses separate functions for test and training samples, although a shared function is possible.
  • Embedding Learning: Embedding methods are classified as task-specific, task-invariant, or hybrid according to whether embedding parameters vary across tasks.Hybrid models encode both task-specific and task-invariant information.
  • Task-specific embedding model: Task-specific methods learn embeddings from the current task, including by enumerating pairwise sample relations to increase training examples.This can enable embedding learning using only few-shot task information.
  • Task-invariant embedding model: Task-invariant methods learn a general embedding from large datasets and apply it to new few-shot tasks without retraining.Some methods simulate few-shot episodes during embedding-model training by withholding classes for evaluation.
  • Embedding methods: Prototype-based methods compare test embeddings with class prototypes, which can stabilize results and reduce computation.A prototype is formed by averaging the embeddings of the K training samples in a class.
  • Other embedding methods: Other embedding approaches use attention, recurrent comparison, relation networks, graph neural networks, or temporal processing for specialized applications.Temporal convolution and attention are highlighted for few-shot reinforcement learning tasks where time information matters.
  • Hybrid embedding model: Task-invariant embeddings reduce computation but do not use current-task knowledge, so hybrid models adapt the generic embedding with information from Dtrain.Hybrid approaches derive an embedding that acts as parameters for the task model.

4.3 Learning with External Memory

External-memory methods store knowledge extracted from training data and use memory contents to represent new samples, thereby restricting the hypothesis space. Their effectiveness depends on memory quality and size, with rare samples vulnerable to replacement.

  • Memory-based representation: External-memory methods extract knowledge from Dtrain and store it in memory, representing each new sample with a weighted average of retrieved contents.This restricts representations to memory contents and reduces the size of H.
  • Memory-based representation: A key-value memory embeds xtest to query similar keys, combines their values, and feeds the resulting representation to a simple classifier.The memory is usually small because manipulating it is expensive.
  • Memory design: Memory-Augmented Neural Networks meta-learn embeddings that map same-class samples to shared values, refining class representations into prototype-like memories.The resulting class representation can be viewed as a refined class prototype in ProtoNet.
  • Memory design: Memory-update strategies include surprise-based updates, abstract memories, and lifelong memories designed to preserve useful or rare information.Lifelong memory replaces the oldest value, while other methods selectively update or refine stored information.
  • Limitations: Rare samples can still be forgotten because they seldom update their slots, giving them a higher chance of being erased when memory is full.This remains a limitation even with lifelong memory.

4.4 Generative Modeling

Generative modeling uses prior distributions learned from other data to constrain few-shot inference and can support generation, recognition, reconstruction, and image flipping. Its methods differ according to the latent variable, inference strategy, and generative architecture.

  • Core approach: Generative modeling estimates p(x) from few observed samples using prior knowledge, commonly through p(x|y) and p(y).The approach supports generation, recognition, reconstruction, and image flipping.
  • Core approach: A learned latent prior p(z;γ) combines with Dtrain to constrain the posterior and reduce the effective hypothesis space from H to ˜H.The latent variable follows z ∼ p(z;γ), while observations are modeled through p(x|z;θ).
  • Method taxonomy: Generative-modeling methods are grouped into three types according to what the latent variable z represents.The survey uses this latent-variable role to organize the methods.
  • Prior knowledge: Shared components and priors can transfer knowledge to few-shot tasks, such as reconstructing faces from eyes, noses, and mouths or estimating a rare class prior from related classes.These examples motivate decomposable representations and groupwise shared priors.
  • Inference: Because the posterior denominator is intractable, methods often approximate p(z|x;θ,γ) with a learned variational distribution or inference network.Amortized inference makes the learned inference network directly reusable on a new task.
  • Architectures: VAE, autoregressive, GAN, and hybrid VAE-GAN architectures are used across surveyed generative FSL methods.The cited methods apply these architectures to different few-shot tasks.

4.5 Discussion and Summary

The survey summarizes when major FSL strategies are useful and identifies practical trade-offs. Their applicability depends on related data, task similarity, memory resources, or the desired task type.

  • Multitask learning: Multitask learning can constrain H when similar or auxiliary tasks exist, but requires retraining the joint model whenever a new few-shot task arrives.The few-shot task can also be overwhelmed when other tasks contain many more samples.
  • Embedding learning: Embedding learning benefits from large datasets with many classes, but may perform poorly when the few-shot task is not closely related to the other tasks.The survey also highlights the need to better mix invariant and task-specific information.
  • External memory: External-memory methods add space and computation costs that increase with memory size, so current memories remain limited in size.Selective update rules can protect memory slots but do not remove this resource constraint.
  • Generative modeling: Generative models are suitable when generation or reconstruction is required and can generate samples for augmentation, but they have high inference cost and are harder to derive than deterministic models.They use learned priors to reduce H to a smaller ˜H.

5 ALGORITHM

Algorithm-focused FSL methods use prior knowledge to improve how parameters are searched, either by initializing parameters or by learning search updates. The survey covers parameter refinement, model aggregation, added task-specific parameters, and meta-learned optimization, while noting trade-offs and open transfer issues.

  • Problem and taxonomy: Few-shot Dtrain makes the empirical risk minimizer unreliable, motivating prior knowledge to influence parameter initialization or search steps.The survey identifies this unreliability as the central algorithmic challenge.
  • Problem and taxonomy: The algorithm taxonomy includes refining existing parameters, refining meta-learned parameters, and learning an optimizer that outputs search steps.These strategies respectively use pretrained initialization, task-distribution meta-learning, or learned search directions and stepsizes.
  • Refining existing parameters: Pretrained parameters can be fine-tuned with regularization, early stopping, selective updates, grouped updates, or model regression to limit overfitting.These methods adapt θ0 while constraining how much or which parts change.
  • Parameter aggregation: When no single θ0 is suitable, parameters from related models can be aggregated or adapted using unlabeled, similar, or auxiliary datasets.Examples include combining models for facial components and reusing classifiers from similar classes.
  • Task-specific parameters: Adding task-specific parameters δ expands θ to {θ0,δ}, allowing pretrained representations to be retained while learning new-task information from Dtrain.One example uses pretrained CNN lower layers with a learned linear classifier.
  • Meta-learning: Meta-learning refines θ0 across tasks so a learner can quickly adapt it to a task-specific parameter ϕs using a few gradient steps.MAML updates θ0 using feedback from multiple meta-training tasks and few-shot training losses.
  • Discussion and limitations: Refining existing parameters usually reduces computation but may sacrifice precision because θ0 comes from different tasks, while meta-learning still faces cross-granularity and negative-transfer concerns.The survey identifies these as trade-offs and open issues for algorithmic FSL.

6 FUTURE WORKS

The survey identifies future directions for FSL across problem setups, techniques, applications, and theories, emphasizing multimodal knowledge, adaptive task distributions, automation, broader applications, and stronger theoretical analysis.

  • Problem setups: Future FSL methods could exploit abundant supervision from another modality when the target modality has few labeled examples.The survey cites extinct animals as an example where textual information may complement scarce visual examples.
  • Problem setups: Cross-modal FSL approaches include fine-tuning zero-shot models, shared embedding spaces, and graph-based alignment of structured label relationships.Fine-tuning on few-shot data may overfit, motivating alternative alignment strategies.
  • Techniques: Meta-learning can adapt rapidly with low inference cost, but unknown task relatedness can cause negative transfer when all tasks are used directly.The survey also highlights dynamic task distributions and catastrophic forgetting in streaming settings.
  • Techniques: Because FSL methods have different strengths and weaknesses, AutoML could learn task-aware augmentation, features, models, architectures, and search strategies.The survey presents automation as a route toward economical, efficient, and effective algorithm designs.
  • Applications: The survey frames FSL as useful for rare cases, reduced data-gathering effort, and applications spanning computer vision, robotics, language, and acoustic processing.These motivations support continued investigation across multiple application domains.
  • Applications: FSL applications extend beyond dominant computer-vision benchmarks to video, robotics, language, speech, drug discovery, architecture search, curve fitting, and logic reasoning.Computer vision remains the main testbed, with character recognition and image classification especially common; standard benchmarks already report very high accuracies.
  • Theories: FSL uses prior knowledge to increase supervised experience, reduce hypothesis-space complexity, or improve search, suggesting a route to lower sample complexity.The survey connects these mechanisms to theoretical sample-complexity analysis.
  • Theories: Theoretical understanding remains limited: few methods have domain-adaptation analyses, and convergence is not fully understood for meta-learning algorithms.The survey calls for broader risk-bound and convergence analyses, including meta-learning over task distributions.

7 CONCLUSION

FSL addresses rapid learning from few supervised examples by incorporating prior knowledge. The survey formalizes the problem, distinguishes related paradigms, organizes methods by how they use prior knowledge, and identifies future research directions.

  • Conclusion: FSL incorporates prior knowledge to learn new tasks from only a few supervised examples, addressing a gap between current AI and human learning.The survey also presents FSL as useful for rare cases and reducing large-scale data-collection burdens.
  • Conclusion: The survey formally defines FSL and distinguishes it from weakly supervised, imbalanced, transfer, and meta-learning problems.These comparisons clarify FSL’s position among related learning problems.
  • Conclusion: The survey identifies unreliable empirical risk minimization as FSL’s core issue and classifies solutions into data, model, and algorithm perspectives.Data augments supervised experience, model constrains the hypothesis space, and algorithm alters the search strategy.
  • Conclusion: Each FSL category is reviewed with attention to its advantages and disadvantages, alongside proposed directions for future research.The proposed directions cover problem setups, techniques, applications, and theories.

A APPENDIX: META-LEARNING

The appendix describes meta-learning as training across source tasks and testing on disjoint target tasks drawn from a task distribution. A meta-learner is optimized using learners’ errors across tasks.

  • Meta-training: Meta-learning trains on a set of source tasks sampled from a task distribution p(T), with each task containing training and test data.Each learner uses its task training set and is evaluated on the corresponding test set.
  • Meta-training: The meta-learner optimizes parameter θ0 to minimize error aggregated across the individual learners.The appendix presents this as the meta-training objective.
  • Meta-testing: Meta-testing evaluates generalization on a disjoint set of target tasks sampled from the same task distribution.The target tasks may contain a different number of classes from the source tasks.
  • Meta-testing: The reported meta-learning testing error is the loss averaged across the target tasks.This aggregation measures performance over the meta-testing task set.
Loading 1904.05046v3…