A Survey of Deep Meta-Learning

• Deep learning’s data and training demands constrain neural networks’ ability to learn new concepts quickly, motivating meta-learning as a strategy to overcome this limitation.Meta-learning agents improve their learning ability over time through inner-level adaptation to new tasks and outer-level knowledge accumulated across earlier tasks.Source: §1 · pp. 1–2
• This survey focuses on Deep Meta-Learning: procedures that learn inductive biases for deep neural networks, rather than the broader meta-learning field.The broader notion also includes algorithm selection and hyperparameter optimization techniques.Source: §1 · pp. 2–3
• The survey fills a gap by providing a detailed, unified overview of rapidly developing Deep Meta-Learning techniques, including their relationships, strengths, weaknesses, challenges, and future directions.It covers supervised and reinforcement learning methods, gives extra attention to MAML (Finn et al., 2017) and related techniques, and complements broader or less technical prior surveys by Vanschoren (2018) and Hospedales et al. (2020).Source: §1 · p. 3
• Larger network backbones tend to achieve better 1-shot few-shot classification performance, indicating that network size should be considered when comparing algorithms.The survey reports a strong relationship between network complexity and classification performance, contrary to the expectation that small networks should prevent overfitting in few-shot settings.Source: Figure 1 · p. 2, §1 · p. 3
• The taxonomy organizes Deep Meta-Learning approaches into metric-based, model-based, and optimization-based techniques.The taxonomy follows Vinyals (2017).Source: §1 · p. 3

2.1.1 Regular Supervised Learning

• Regular supervised learning minimizes empirical loss on one data set D, whereas meta-learning optimizes knowledge that supports rapid learning across many data sets.Evaluation on unseen examples measures generalization beyond D, while practical optimization often approximates globally optimal parameters.Source: §2.1.1 · pp. 4–5, §2.1.2 · p. 5

2.1.2 Supervised Meta-Learning

• Meta-learning learns meta-knowledge ω across a distribution of tasks, enabling a base learner to acquire new tasks quickly instead of relying on pre-defined learning guidance.Supervised meta-learning uses inner-level task-specific learning and outer-level optimization across multiple tasks.Source: §2.1.2 · p. 5

2.1.3 Regular Reinforcement Learning

• In reinforcement learning, an agent learns a policy πθ through MDP interaction to maximize expected reward, while meta-reinforcement learning maximizes expected reward across multiple MDPs sampled from p(T).The discount factor γ controls the relative emphasis on short- versus long-term rewards.Source: §2.1.3 · p. 6, §2.1.4 · pp. 6–7

2.1.5 Contrast with other Fields

• Meta-learning differs from transfer learning through its explicit task-distribution meta-objective and from multi-task learning through its focus on quickly learning previously unseen tasks.Transfer learning may instead pre-train on one large data set and fine-tune on a smaller one, whereas multi-task learning jointly trains on fixed tasks.Source: §2.1.5 · p. 7

2.2 The Meta-Setup

• The general meta-setup separates disjoint meta-train, meta-validation, and meta-test task sets for learning, hyperparameter feedback and meta-generalization assessment, and final performance estimation.Meta-validation evaluates unseen tasks, while meta-test tasks provide the final performance estimate.Source: §2.2 · p. 8

2.2.1 N-way, k-shot Learning

• N-way, k-shot episodes sample exactly N classes and k examples per class, using support sets to guide query predictions while disjoint stage labels test learning new concepts.During evaluation, meta-information ω is fixed but task-specific parameter updates remain allowed; small k yields few-shot learning.Source: §2.2.1 · pp. 8–9

2.2.2 Common Benchmarks

• Common evaluation resources include image benchmarks such as Omniglot, miniImageNet, tieredImageNet, CIFAR-10, CIFAR-100 and variants, CUB-200-2011, and MNIST.Source: §2.2.2 · pp. 9–10

2.2.3 Some Applications of Meta-Learning

• Deep meta-learning reduces dependence on data and computation by enabling rapid concept acquisition, supporting applications including few-shot classification, robotic control, hyperparameter optimization, learned rules, and abstract reasoning.The field comprises algorithm selection and hyperparameter optimization, alongside searches for inductive bias in self-improving deep networks; reviewed methods are grouped as metric-, model-, and optimization-based.Source: §2.2.3 · p. 10, §2.3 · p. 11, §2.4 · p. 12

3 Metric-based Meta-Learning

• Metric-based methods meta-learn an informative feature space or similarity kernel, then classify new inputs by comparing them with labeled examples according to similarity.They perform non-parametric inner-task learning without changing network parameters, while the embedding or kernel is learned across tasks.Source: §3 · p. 14, §3.1 · p. 16

3.1 Example

• The new input is classified as class 4 because it is most similar to the support input [6, 0], whose label is 4.Without normalization, the score vector is [−0.12, −0.58, −0.37, 0.99], with the largest entry corresponding to class 4.Source: §3.1 · p. 15

3.2 Siamese Neural Networks

• Siamese networks share weights across twin encoders, learn discriminative features through image verification, and assign a new input the label of its most similar support example.They provide a simple few-shot approach but are not readily applicable outside supervised learning.Source: §3.2 · pp. 16–17

3.3 Matching Networks

• Matching networks predict labels as a similarity-weighted combination of all support examples using embedding functions and attention over cosine similarity.Full Context Embeddings improved miniImageNet accuracy by roughly 2% over regular matching networks, while label-distribution bias can degrade performance.Source: §3.3 · pp. 17–18

3.4 Prototypical Networks

• Prototypical networks compare queries with class centroids rather than individual support examples, reducing pairwise comparisons and computational cost.They outperformed matching networks in 5-way, k-shot learning for k = 1, 5 on Omniglot and miniImageNet, despite using no complex task-specific embeddings.Source: §3.4 · pp. 18–20

3.5 Relation Networks

• Relation networks learn a trainable similarity function over concatenated query-support embeddings, yielding better performance than methods using fixed similarity metrics.They select the class with the highest relation score and optimize relation scores using mean-squared error.Source: §3.5 · pp. 20–21

3.6 Graph Neural Networks

• Graph neural networks represent each task as a fully connected graph and propagate labels from labeled support nodes to unlabeled query nodes.They learn both input representations and label-propagation mechanisms, achieve good few-shot performance, and apply to semi-supervised and active learning.Source: §3.6 · pp. 21–22

3.7 Attentive Recurrent Comparators (ARCs)

• Attentive recurrent comparators interleave two inputs and use recurrent attention to fuse information from corresponding image parts before classification.Although biologically inspired, they have higher computational cost and often do not outperform graph neural networks; overall, metric methods predict classes from learned similarity spaces.Source: §3.7 · pp. 22–23, §3.8 · p. 23

3.8 Metric-based Techniques, in conclusion

• Metric-based techniques predict classes by comparing support and query inputs, with later methods using graph-based information flow, class prototypes, learned similarity functions, or interleaved glimpses.Siamese networks introduced the comparison-based approach; matching networks train directly on few-shot tasks with cosine similarity, prototypical networks compare queries with class prototypes, relation networks learn domain-specific similarity, and GNNs make support-query information flow parametric.Source: §3.8 · p. 23
• Their main advantages are conceptual simplicity and fast test-time inference on small tasks because they avoid task-specific weight adjustments.Source: §3.8 · pp. 23–24
• Performance may degrade when meta-test tasks become distant from meta-training tasks because metric-based methods cannot absorb new task information into network weights.Source: §3.8 · p. 24
• Pairwise comparisons can become prohibitively expensive as tasks grow larger, while reliance on labeled examples limits these methods largely to supervised settings.Source: §3.8 · p. 24

4 Model-based Meta-Learning

• Model-based techniques sequentially update an internal task representation from support-set inputs, using memory to retain information for predictions on new inputs.Unlike metric-based methods limited to pairwise comparisons, they can learn internal dynamics for processing and predicting task data.Source: §4 · pp. 24–25

4.1 Example

• An external memory matrix can store the support examples as an internal task representation and predict class 4 for [4, 0.5] using the largest dot product.The example illustrates that storing the entire task in memory makes the technique model-based, even when its prediction uses a similarity function.Source: §4.1 · p. 25

4.2 Recurrent Meta-Learners

• Recurrent meta-learners use LSTMs or GRUs as dynamic task-embedding storage, adapting quickly through hidden states while training accumulates knowledge across episodic tasks.They perform almost as well as asymptotically optimal algorithms on simple reinforcement-learning tasks, but degrade when temporal dependencies span longer horizons (Wang et al., 2016; Duan et al., 2016).Source: §4.2 · pp. 25–26

4.3 Memory-Augmented Neural Networks (MANNs)

• Memory-augmented neural networks combine a recurrent controller with external memory for rapid task-specific adaptation and apply to both classification and regression, but their architecture is complex (Santoro et al., 2016).The controller provides long-term memory parameters, while external memory stores and retrieves prior representations; the approach uses LRUA to write to least- or most-recently-used locations.Source: §4.3 · pp. 27–29

4.4 Meta Networks

• Meta networks separate base- and meta-learners, using support-set information and external memory to generate task- or input-specific fast weights for prediction (Munkhdalai and Yu, 2017).Their flexibility across supervised and reinforcement learning is offset by many components, high memory and computation costs, and time-consuming architecture selection.Source: §4.4 · pp. 29–32

4.5 Simple Neural Attentive Meta-Learner (SNAIL)

• SNAIL replaces external memory with temporal convolutions and soft attention, providing high-bandwidth access and selective retrieval of prior experiences for supervised and reinforcement learning (Mishra et al., 2018).Its DenseBlock, TCBlock, and AttentionBlock learn internal dynamics, while selecting the correct convolutional architecture can be time-consuming.Source: §4.5 · pp. 32–33

4.6 Conditional Neural Processes (CNPs)

• Conditional neural processes avoid external memory by embedding support examples and aggregating their representations into a single latent representation for the task (Garnelo et al., 2018).The described workflow begins by observing the labeled support set, embedding each example with hθ, and aggregating the resulting representations with operator a.Source: §4.6 · pp. 33–34
• Conditional neural processes encode previously seen inputs into compact representations that support classifying new observations, but are often outperformed in few-shot settings by matching networks (Vinyals et al., 2016).They train by sampling tasks and conditioning-set sizes, then maximizing likelihood over the support set.Source: §4.6 · p. 34

4.7 Neural Statistician

• The neural statistician learns unsupervised dataset-level meta-features and predicts N-way few-shot classes by assigning the label whose context posterior is closest to the new input’s posterior (Edwards and Storkey, 2017).It supports supervised and unsupervised settings but requires many datasets for good performance (Edwards and Storkey, 2017).Source: §4.7 · pp. 34–35

4.8 Model-based Techniques, in conclusion

• Model-based approaches internalize tasks in the system state, enabling flexible internal dynamics and broader applicability but often weaker supervised performance and distant-task generalization.They may also struggle with larger datasets.Source: §4.8 · pp. 35–36

5 Optimization-based Meta-Learning

• Optimization-based techniques formulate meta-learning as bi-level optimization: task-specific inner updates are optimized across tasks at the outer level.The inner learner updates its parameters using an optimization strategy such as gradient descent, while the outer objective evaluates performance across tasks.Source: §5 · p. 36

5.2 LSTM Optimizer

• LSTM optimizers replace hand-crafted gradient updates with learned updates, tailoring the optimization policy to a task family (Andrychowicz et al., 2016).Their differentiable training objective can weight performance across update steps, while shared parameters across coordinates prevent parameter explosion.Source: §5.2 · p. 38
• LSTM optimizers can learn faster than hand-crafted optimizers, including on data sets different from those used for training (Andrychowicz et al., 2016).However, the method was not evaluated across tasks or in few-shot learning, leaving its few-shot performance and scalability to larger base learners unclear.Source: §5.2 · pp. 38–39

5.3 LSTM Meta-Learner

• LSTM meta-learners embed base-learner weights in the LSTM cell state, so cell-state updates directly become parameter updates.The cell state starts at c0 = θ0, is adjusted toward a task-shared initialization, and remains in one-to-one correspondence with the base-learner weights over time.Source: §5.3 · p. 39
• The LSTM meta-learner computes parameter-wise learning rates from gradients, losses, prior parameters, and prior learning rates, while forget gates control retained state.Weight sharing limits meta-learner size, second-order derivatives are ignored, and batch normalization stabilizes and accelerates training.Source: §5.3 · p. 40
• MAML (Finn et al., 2017) is a simpler approach that outperforms LSTM meta-learners while exploiting task commonalities for quicker optimization.LSTM meta-learners maintain a one-to-one correspondence between base-learner weights and LSTM cell states, enabling learned optimization across tasks.Source: §5.3 · p. 40

5.4 Reinforcement Learning Optimizer

• Li and Malik (2018) formulate optimization as a reinforcement-learning problem in which an optimizer learns weight updates toward solutions while penalizing undesirable trajectories.The objective can capture slow convergence, oscillations, and other unwanted properties.Source: §5.4 · p. 40
• Their optimizer treats optimization as a POMDP, using a recurrent neural network policy trained with guided policy search from weight, gradient, and objective observations.The state is the current weights and the action is the proposed update Δθt.Source: §5.4 · pp. 40–41
• Li and Malik (2018) showed that reinforcement-learning optimizers can generalize across network architectures and data sets, although their small base learner leaves scalability to larger architectures unresolved.This was an initial step toward general optimization through reinforcement learning.Source: §5.4 · p. 41

5.5 MAML

• MAML (Finn et al., 2017) learns an initialization θ from which simple gradient-based updates, such as stochastic gradient descent, rapidly adapt to new tasks.The learned initialization acts as an inductive bias about the overall task structure and is illustrated in Figure 24.Source: §5.5 · p. 41, Figure 24 · p. 41
• MAML explicitly optimizes low loss after a specified number of gradient steps, with the one-step case defining a task-averaged meta-loss over sampled tasks.The number of adaptation steps s is fixed in advance, and multiple-step variants extend the same objective.Source: §5.5 · pp. 41–42
• Full MAML requires second-order gradients through task-specific adaptation trajectories, making optimization expensive in time and memory, especially with more inner updates.FOMAML omits these second-order terms and performed reasonably similarly to MAML in Finn et al. (2017).Source: §5.5 · p. 42
• MAML trains by sampling task batches, adapting the shared initialization on each task, and using the resulting test losses for the outer update before applying the same adaptation procedure at test time.FOMAML follows the same procedure but changes the initialization update rule.Source: §5.5 · pp. 42–43
• MAML’s simplicity, general applicability, and strong performance have driven attention, while Antoniou et al. (2019) proposed stability, speed, flexibility, and generalization improvements.Despite these strengths, optimizing a base learner per task and computing higher-order derivatives can make MAML costly in runtime and memory.Source: §5.5 · p. 43

5.6 iMAML

• By approximating higher-order derivatives rather than ignoring them as FOMAML does, iMAML addresses the potential performance loss of first-order approximation with lower memory consumption.The method approximates the required derivatives without storing Hessian matrices.Source: §5.6 · pp. 43–44
• Conjugate-gradient optimization computes the approximate meta-gradient without storing Hessian matrices, enabling more inner gradient updates but requiring explicit regularization to avoid overfitting.Conventional MAML uses few inner steps, effectively applying early stopping, whereas iMAML can accommodate longer optimization paths.Source: §5.6 · p. 44
• iMAML substantially reduces memory costs, supports larger optimization paths, and permits flexible choices of inner optimizer while keeping computational costs roughly comparable to MAML.It avoids differentiating through the optimization process and can use non-differentiable inner optimizers.Source: §5.6 · pp. 44–45

Meta-SGD

• Meta-SGD optimizes support-to-query generalization after only one update step using a fully differentiable loss trainable with standard SGD.The objective includes query-set loss observed during meta-training.Source: §5.7 · pp. 45–46
• Meta-SGD jointly learns an initialization θ and per-parameter learning rates α, making it more expressive than MAML but increasing the number of hyperparameters.Unlike scalar learning rates, α is a vector whose elements adapt updates for individual parameters.Source: §5.7 · pp. 45–46

Method

• Reptile repeatedly trains on sampled tasks and moves the initialization toward the resulting task-adapted weights.This process seeks an initialization providing a good inductive bias for tasks from the same family.Source: §5.8 · p. 46
• Reptile can use task-update differences as gradients, apply a meta-optimizer, or average update directions across batches of n tasks.The update is represented as θ := θ + ϵ¯θ after averaging task updates.Source: §5.8 · p. 46

Interpretation

• Reptile can be interpreted as gradient descent minimizing the distance between initialization parameters and task-optimal weights.Its trajectory oscillates between the optimal weights of interleaved tasks, as illustrated in Figure 26.Source: §5.8 · p. 47, Figure 26 · p. 47

Comparison with MAML

• Reptile avoids differentiating through the optimization trajectory like MAML (Finn et al., 2017), reducing time and memory costs but weakening its theoretical foundation.Unlike MAML, it does not directly optimize for fast learning.Source: §5.8 · p. 47
• Reptile performance may be worse than MAML (Finn et al., 2017) in some settings.Source: §5.8 · p. 47

5.9 Latent embedding optimization (LEO)

• LEO targets poor generalization from estimating high-dimensional parameters with gradients from only a few examples in N-way, k-shot settings.Source: §5.9 · pp. 47–48
• LEO learns task-conditioned, lower-dimensional latent embeddings that indirectly produce good initial parameters θ, but experiments generate parameters only for final layers.Although LEO could initialize an entire base learner, high-dimensional decoding is problematic, so it uses pre-trained models and limits expressivity.Source: §5.9 · pp. 48–49
• LEO encodes support examples, forms all pairwise hidden-code combinations, and uses a relation network to parameterize class-specific latent-code distributions.The decoder samples latent codes and generates task-specific initializations, with losses propagated backward to optimize the embedding space.Source: §5.9 · p. 48
• LEO aids generalization through lower-dimensional optimization but is more complex than MAML (Finn et al., 2017) and applies only to few-shot learning.Source: §5.9 · p. 49

5.10 Online MAML (FTML)

• Online MAML (Finn et al., 2019) adapts MAML (Finn et al., 2017) to sequences of tasks by selecting parameters that perform well across their loss functions.Its online objective is evaluated using regret over the potentially infinite task sequence, comparing chosen updates with the best fixed parameters in hindsight.Source: §5.10 · p. 49
• FTML (Finn et al., 2019) minimizes online regret by setting each next-step initialization to the best parameters in hindsight and applying meta-updates over buffered tasks.MetaUpdate samples tasks and train/test mini-batches, then updates θ using the meta-learning rate β.Source: §5.10 · pp. 49–50
• Online MAML is robust for online learning (Finn et al., 2019), but storing all encountered task data causes computational costs and memory usage to grow over time.Future work includes reducing these costs and testing more than one inner gradient update per task.Source: §5.10 · p. 50

5.11 LLAMA

• LLAMA (Grant et al. (2018)) learns a probability distribution over task-specific parameters instead of a single parameter vector, enabling multiple potential solutions per task.The method is developed only for supervised learning settings.Source: §5.11 · pp. 50–51
• LLAMA maximizes query-set predictive log-likelihood while ML-LAPLACE computes adapted task parameters and updates the initialization through gradient descent.The task-specific curvature matrix is approximated with K-FAC (Martens and Grosse, 2015).Source: §5.11 · p. 51
• Task-specific solutions are modeled as a diagonal Gaussian, from which possible parameter vectors can be sampled for a new task.This probabilistic representation replaces selecting only one adapted solution.Source: §5.11 · p. 51
• LLAMA increases computational costs, while its Laplace approximation can be quite inaccurate (Grant et al. (2018)).These limitations qualify the computational and approximation benefits of producing multiple task solutions.Source: §5.11 · pp. 51–52

PLATIPUS

• PLATIPUS (Finn et al., 2018) learns probability distributions over initializations θ instead of task-specific parameters, then samples θ and adapts it into fast weights θ′ with gradient descent.This extends the probabilistic interpretation of LLAMA (Grant et al., 2018).Source: §5.12 · p. 52
• Unlike MAML, PLATIPUS learns query- and support-conditioned Gaussian priors whose means are updated by gradient steps and aligned with a Kullback–Leibler divergence loss.The additional parameter vectors represent distribution means, covariances, and learning rates for distribution updates.Source: §5.12 · pp. 52–53
• At test time, PLATIPUS samples multiple initializations from the prior conditioned on training data and performs gradient updates using the available support set, without access to test data.The test-time procedure supports multiple potential initializations for a task.Source: §5.12 · p. 53
• PLATIPUS is uncertainty-aware, enabling active learning by requesting labels for inputs about which it is unsure, particularly in critical domains such as medical diagnosis.This uncertainty substantially increases the applicability of deep meta-learning in such domains.Source: §5.12 · p. 53
• The approach has higher computational costs and is not applicable to reinforcement learning.These are identified as its principal downsides.Source: §5.12 · p. 53

• BMAML maintains M parameter particles and jointly optimizes them to represent multiple potential solutions rather than learning a single solution distribution.The particles approximate the probability distribution of test labels after training on each task’s support set.Source: §5.13 · pp. 53–54
• SVGD updates each particle using other particles’ gradients and a similarity kernel, while a repulsive force prevents particle collapse.BMAML uses an RBF kernel, although other kernels are theoretically possible.Source: §5.13 · p. 53
• The Chaser Loss trains BMAML by bringing the support-set particle distribution closer to a leader distribution obtained with additional SVGD steps on support and query data.The leader is treated as a closer approximation to the unknown true distribution as the number of updates increases.Source: §5.13 · p. 54
• BMAML is a robust optimization-based technique that can propose M potential task solutions and can extend to reinforcement learning through Stein Variational Policy Gradient.Source: §5.13 · p. 54
• Keeping M parameter sets in memory limits scalability, while performance is sensitive to the pre-defined kernel function.Reducing memory costs and learning the kernel function are identified as directions for future work.Source: §5.13 · pp. 54–55

Differentiable base-solvers

• R2-D2 and LR-D2 use simple base-learners with closed-form or differentiable solutions to improve learning efficiency (Bertinetto et al., 2019).R2-D2 uses ridge regression, whereas LR-D2 uses logistic regression with Newton’s method.Source: §5.14 · pp. 55–56

R2-D2

• Woodbury’s identity changes the ridge-regression computation from quadratic in embedding size to quadratic in the number of few-shot examples, making it more manageable.The original term is based on X^T X, while the transformed computation uses XX^T.Source: §5.14 · p. 55

LR-D2

• LR-D2 similarly applies Woodbury’s identity to logistic regression, but its Newton solver must run for multiple iterations to obtain W.The transformed computation is quadratic in input size, which is acceptable when the number of examples n is small.Source: §5.14 · p. 56

MetaOptNet

• MetaOptNet uses linear Support Vector Machines (SVMs) as its base-learner and achieved state-of-the-art performance on few-shot image classification (Lee et al., 2019).Source: §5.14 · p. 56

5.15 Optimization-based Techniques, in conclusion

• Optimization-based meta-learners learn optimizers and/or use multiple tasks, distinguishing them from base-level optimization while targeting rapid adaptation to new tasks.Figure 28 maps relationships among the covered optimization-based techniques.Source: §5.15 · p. 56, Figure 28 · p. 57
• The LSTM optimizer (Andrychowicz et al., 2016) initiated this line, while the LSTM meta-learner (Ravi and Larochelle, 2017) added learned initial weights for few-shot transfer across tasks.Li and Malik (2018) instead learned optimization procedures with reinforcement learning; MAML (Finn et al., 2017) replaced the trainable LSTM optimizer with hand-crafted gradient descent and inspired many works.Source: §5.15 · pp. 56–57
• Meta-SGD (Li et al., 2017) extends MAML by learning learning rates, while LLAMA (Grant et al., 2018), PLATIPUS (Finn et al., 2018), and online MAML (Finn et al., 2019) target active or online learning.LLAMA and PLATIPUS provide probabilistic MAML interpretations with multiple solutions and uncertainty, whereas BMAML (Yoon et al., 2018) jointly optimizes M initializations.Source: §5.15 · p. 57
• iMAML (Rajeswaran et al., 2019) reduces MAML’s second-order derivative costs and permits non-differentiable inner-loop optimization, while Reptile (Nichol et al., 2018) removes higher-order derivatives.LEO (Rusu et al., 2018) improves robustness by optimizing in lower-dimensional parameter space through an encoder-decoder architecture.Source: §5.15 · pp. 57–58
• R2-D2, LR-D2 (Bertinetto et al., 2019), and Lee et al. (2019) combine learned feature extractors with ridge regression, logistic regression, and SVM classifiers, respectively.These methods use simple classical machine-learning classifiers on top of learned representations.Source: §5.15 · p. 58
• Optimization-based methods can perform better across wider task distributions than model-based approaches, but task-specific base-learning and optimizer learning are computationally expensive.Future work should reduce computational demands while improving solution quality and generalization, supported by benchmarking and reproducibility research.Source: §5.15 · p. 58

6.1 Overview

• Deep Meta-Learning comprises metric-, model-, and optimization-based methods relying on input similarity, task embeddings with states, and task-specific updates, respectively.Metric methods are simple and effective but mainly supervised; model methods offer flexible dynamics but weaker distant-task generalization; optimization methods generalize better but are computationally expensive.Source: §6.1 · p. 59
• MetaOptNet achieves the best reported miniImageNet results, with 64.09% 1-shot and 80.00% 5-shot accuracy, using a deeper backbone than most alternatives.Across techniques, 5-shot performance typically exceeds 1-shot performance, and deeper backbones tend to produce better classification performance.Source: §6.1 · p. 59, Table 5 · p. 60

6.2 Open Challenges and Future Work

• Different backbones make reported performances difficult to compare, motivating large-scale benchmarks that evaluate techniques with identical feature extractors.Performance differences may reflect backbone expressivity rather than methodological improvements, and clearer evidence is needed on how feature extraction affects results.Source: §6.2 · pp. 60–61
• Meta-learners can overfit seen tasks and lose adaptation ability on distant tasks, where pre-training followed by fine-tuning may outperform state-of-the-art meta-learning.Meta-Dataset offers a more challenging evaluation setting with diverse benchmarks, including Omniglot and ImageNet.Source: §6.2 · p. 61
• Reducing meta-training computation time and memory costs remains essential for deploying Deep Meta-Learning in real-world domains.Although these systems can learn new tasks quickly, meta-training can be computationally expensive.Source: §6.2 · p. 61
• Raghu et al. (2020) found that MAML largely relies on feature reuse, leaving open whether methods can depend more on genuinely rapid learning.Future directions also include online or active learning and compositional systems that decompose inputs into previously learned components for low-data regimes.Source: §6.2 · pp. 61–62