On First-Order Meta-Learning Algorithms

Introduction

• Human rapid learning is attributed to prior knowledge and recombination of existing skills, while developing computationally feasible Bayesian deep-learning algorithms is challenging.The introduction cites human recognition of new object categories from a few images and novice progress in Frostbite after 15 minutes versus over 1000 times more experience for double-dueling-DQN [19].Source: §1 · p. 1
• Meta-learning optimizes a fast-learning algorithm on training tasks so it performs well on test tasks sampled from the same task distribution.Because each task is itself a learning problem, good task performance corresponds to learning quickly.Source: §1 · p. 1
• Unlike gradient-free recurrent-network approaches [3], MAML fine-tunes at test time and can generalize through a gradient-based learner on out-of-sample data [5].MAML differentiates through fine-tuning, making it poorly suited to settings requiring many test-time gradient steps.Source: §1 · p. 2
• First-order MAML [4] ignores second-derivative terms, and was found to work nearly as well as MAML on Mini-ImageNet [18].The paper explores first-order methods because higher-order gradients can be cumbersome, especially when many test-time gradient steps are needed.Source: §1 · p. 2
• The authors show that first-order MAML [4] is simpler to implement than previously recognized.Their empirical evaluation on Mini-ImageNet [18] and Omniglot also offers implementation best-practice insights.Source: §1 · p. 2
• Reptile is a first-order algorithm related to FOMAML that moves an initialization toward task-trained weights and does not require per-task training-test splits.Its similarity to joint training makes its success as a meta-learning algorithm surprising; it is also related to fast weights / slow weights.Source: §1 · p. 2
• The theoretical analysis shows that first-order MAML and Reptile both optimize for within-task generalization.Source: §1 · p. 2

2 Meta-Learning an Initialization

• MAML seeks an initialization φ that achieves low loss after k updates on a randomly sampled task τ.In few-shot learning, the update operator can perform gradient descent or Adam on batches sampled from τ.Source: §2 · pp. 2–3
• MAML uses training samples A for inner-loop updates and separate test samples B to compute loss, thereby optimizing for generalization.The passage likens this setup to cross-validation.Source: §2 · p. 3
• FOMAML treats inner-loop gradients as constants, replacing the update Jacobian with the identity and using the loss gradient at the adapted parameters.Its implementation samples a task, applies the update operator, and computes the gradient at the resulting parameters.Source: §2 · p. 3

3 Reptile

• Reptile learns an initialization by sampling a task, taking k SGD or Adam steps, and moving the initialization toward the task-trained parameters.The serial update is φ ← φ + ϵ(eφ − φ).Source: §3 · p. 3
• For k = 1, Reptile corresponds to stochastic gradient descent on the expected task loss.The paper also describes a parallel version that evaluates on n tasks per iteration.Source: §3 · p. 4
• For k > 1, Reptile’s expected update includes second- and higher-order loss terms, so it does not take a gradient step on expected loss and converges to a different solution.The algorithm can treat φ − eφ as a gradient for an adaptive optimizer such as Adam; the paper notes the natural Reptile gradient is (φ − eφ)/α, where α is the inner stepsize.Source: §3 · pp. 3–4

Case Study: One-Dimensional Sine Wave Regression

• Tasks vary in amplitude and phase, and learners use sampled input-output points to predict the whole sine function.Amplitudes are sampled from [0.1, 5.0], phases from [0, 2π], and inputs from [−5, 5].Source: §4 · p. 4
• Because random phase makes the average sine function zero everywhere, minimizing expected task loss yields the useless zero-function predictor.Source: §4 · p. 4
• After Reptile training, the network quickly converges to a sampled sine wave and infers values away from sampled points; MAML and random initialization are shown for comparison.The demonstration trains on 10 sampled points and performs 32 gradient steps on an MLP with layers 1 →64 →64 →1.Source: §4 · pp. 4–5, Figure 1 · p. 5

5.1 Leading Order Expansion of the Update

• Taylor expansions show Reptile, FOMAML, and MAML share an expected-loss term and a higher-order term that raises same-task minibatch-gradient alignment, promoting within-task generalization.Source: §5.1 · pp. 5, 7
• For k ≥2, the AvgGradInner-to-AvgGrad coefficient ratio ranks MAML > FOMAML > Reptile, grows linearly with α and k, and the approximation requires small αk.Source: §5.1 · p. 7

5.2 Finding a Point Near All Solution Manifolds

• A second, explicitly informal account casts Reptile as SGD toward an initialization close in Euclidean distance to every task’s optimal-solution manifold, approximating exact minimizers with k task-loss gradient steps.The authors caution that this argument should be taken less seriously than the Taylor-series analysis.Source: §5.2 · p. 8

6.1 Few-Shot Classification

• MAML, FOMAML, and Reptile perform similarly; Reptile is slightly better on Mini-ImageNet, slightly worse on Omniglot, and transduction boosts performance in all cases.The reported results are in Tables 1 and 2.Source: Table 1 · p. 9, Table 2 · p. 9, §6.1 · p. 9

6.2 Comparing Different Inner-Loop Gradient Combinations

• Using more inner-loop mini-batches improves all methods, with a larger gain for summed gradients (Reptile) than for the final gradient alone (FOMAML).On 5-shot, 5-way Omniglot, using only the first gradient is ineffective, and two-step Reptile performs worse than two-step FOMAML; the authors suggest Reptile can benefit from many inner-loop steps.Source: §6.2 · p. 10, Figure 3 · p. 10

6.3 Overlap Between Inner-Loop Mini-Batches

• Separate-tail FOMAML significantly outperforms shared-tail FOMAML, while separate-tail FOMAML and Reptile remain robust to changes in inner-loop iterations and batch size.Separate-tail FOMAML uses a final mini-batch disjoint from earlier batches, as in Finn et al. [4].Source: §6.3 · p. 11
• Shared-tail FOMAML degrades when its final mini-batch overlaps earlier batches: cycling drops from five iterations, while random sampling degrades more gradually, including at batch sizes above 25.A step-size sweep also found shared-tail performance equally poor across choices; the authors hypothesize that gradients after several steps on a sample provide little useful information as optimization approaches a local optimum.Source: §6.3 · pp. 11–12, Figure 4 · p. 11

7 Discussion

• Reptile’s training differs only subtly from joint training and uses only first-order gradients, within a family of appealing test-time gradient-descent methods [5].The effectiveness of fine-tuning from ImageNet-trained models further supports the authors’ confidence in these approaches.Source: §7 · p. 12
• A Taylor-series analysis explains Reptile through an SGD-induced second-order term that encourages gradients from different minibatches of the same task to generalize to one another.The authors also offer an informal geometric explanation: Reptile seeks a point close in Euclidean distance to the optimal solution manifolds of the training tasks.Source: §7 · p. 12
• The Taylor-series analysis suggests SGD generally performs a MAML-like update that promotes generalization across minibatches, partly explaining fine-tuning from ImageNet to smaller datasets [20].The authors hypothesize that joint training followed by fine-tuning will remain a strong meta-learning baseline across machine-learning problems.Source: §7 · p. 12

• The authors propose studying how much SGD automatically optimizes for generalization and whether this effect can be amplified without meta-learning.Source: §8 · p. 12
• Reptile has so far produced negative results in reinforcement learning, where joint training is a strong baseline; modifications may be needed.Source: §8 · p. 12
• Another direction is to explore whether deeper classifier architectures improve Reptile’s few-shot learning performance.Source: §8 · p. 12
• Future work will test whether regularization improves few-shot performance, given the large gap between training and testing error.Source: §8 · p. 12
• The authors also propose evaluating Reptile on few-shot density modeling [14].Source: §8 · p. 12

A Hyper-parameters

• Outer step sizes were linearly annealed to zero, and confidence intervals used the standard deviation across three random-seed runs.These procedures were used for all experiments.Source: A Hyper-parameters · p. 14
• CMA-ES tuning showed that most Reptile hyper-parameters had little effect, so the settings were simplified and shared across experiments when appropriate.Source: A Hyper-parameters · p. 14
• Tables 3–6 report Reptile hyper-parameters for Omniglot and Mini-ImageNet comparisons and for Sections 6.2 and 6.3.Source: Table 3 · p. 14, Table 4 · p. 14, Table 5 · p. 15