Source-linked AI summary
ES-MAML: Simple Hessian-Free Meta Learning
Xingyou Song, Wenbo Gao, Yuxiang Yang, Krzysztof Choromanski, Aldo Pacchiano, Yunhao Tang
TL;DR
MAML’s policy-gradient training makes second-derivative estimation difficult, motivating an alternative for reinforcement-learning meta-learning. ES-MAML applies Evolution Strategies to the meta-learning objective, including with nonsmooth adaptation operators. It is competitive with PG-MAML in reported low-K benchmarks, with task-dependent comparisons and greater stability reported for ES-MAML in that setting.
Problem
Policy-gradient MAML requires estimating second derivatives, which is difficult with backpropagation on stochastic policies.
Source: §1 · p. 1Method
ES-MAML applies Evolution Strategies to the MAML objective and can use nonsmooth adaptation operators such as hill climbing.
Source: §1 · p. 2, §3.2 · p. 5, §3.3.2 · p. 6Results
On low-K benchmarks, ES-MAML is similar or better on Forward-Backward tasks, while PG-MAML generally outperforms it on Goal-Velocity tasks; ES-MAML is relatively stable.
Source: §4.4 · p. 10Takeaways & Limitations
The framework supports hill-climbing adaptation that the authors found improved exploratory behavior and sparse-reward performance, and works with compact deterministic policies.
Source: §5 · p. 10When the adaptation operator is estimated, the resulting estimator for ∇eJσ is generally biased because the true adapted trajectory distribution is unavailable.
Source: §3.2 · p. 6
Abstract
from arXiv · showhide
We introduce ES-MAML, a new framework for solving the model agnostic meta learning (MAML) problem based on Evolution Strategies (ES). Existing algorithms for MAML are based on policy gradients, and incur significant difficulties when attempting to estimate second derivatives using backpropagation on stochastic policies. We show how ES can be applied to MAML to obtain an algorithm which avoids the problem of estimating second derivatives, and is also conceptually simple and easy to implement. Moreover, ES-MAML can handle new types of nonsmooth adaptation operators, and other techniques for improving performance and estimation of ES methods become applicable. We show empirically that ES-MAML is competitive with existing methods and often yields better adaptation with fewer queries.
1 Introduction
The introduction frames RL meta-learning as a data-efficiency problem: MAML adapts to new tasks with a few gradient updates, but its policy-gradient implementation is difficult to estimate reliably. ES-MAML replaces policy gradients with Evolution Strategies, avoiding second-derivative estimation while supporting flexible adaptation and deterministic policies.
1 Introduction
- RL meta-learning targets rapid adaptation to new tasks, where collecting samples can require costly simulations or real-world interactions.MAML trains a meta-policy to adapt through one or a few gradient updates in a new environment.Source: §1 · p. 1
- MAML is challenging to implement because stochastic-policy backpropagation makes second-derivative estimation difficult, while policy-gradient estimates can also have high variance.The cited approaches use unbiased higher-order estimators, control variates, hyperparameter tuning, or learning-rate annealing to address these issues.Source: §1 · pp. 1–2
- ES-MAML replaces the policy-gradient basis of prior MAML algorithms with Evolution Strategies, and its zero-order formulation requires no second-derivative estimates.The introduction also describes ES as simpler to implement, runnable on CPUs without backpropagation, and compatible with different adaptation operators.Source: §1 · p. 2
- ES explores in parameter space and uses episode-level total reward, a setup the authors associate with deterministic policies and more stable training profiles.The paper contrasts this with policy-gradient exploration through randomized actions and use of individual state-action pairs.Source: §1 · p. 2
2 Model Agnostic Meta Learning in RL
The section defines MAML in reinforcement learning as finding a policy that earns reward on new tasks after a small number of policy-gradient adaptation steps. Its standard meta-gradient includes a second-order term, making estimation difficult and potentially high-variance; stochastic actions can also be risky in some adaptation settings.
2 Model Agnostic Meta Learning in RL
- MAML seeks a policy that can adapt quickly to an unknown task by taking one or a few policy-gradient steps.The paper adopts a single gradient step to simplify notation, while noting that multiple steps are readily accommodated.Source: §2 · p. 3
- The adapted policy is defined by applying a task-specific gradient update to the initial policy, and training seeks expected reward under the adapted policy across tasks.The formulation uses a task distribution and trajectories generated under the adapted policy.Source: §2 · pp. 2–3
- PG-MAML computes the meta-objective gradient through the adaptation update, whose gradient contains a second-order term in the policy parameters.The section calls the policy-gradient algorithms that use this expression PG-MAML.Source: §2 · p. 3
- Correctly estimating the second-order term is difficult and can have extremely high variance, while proposed estimators and variance-reduction methods retain drawbacks.The section notes that LVC introduces unknown estimator bias and that the cited solutions are complicated.Source: §2 · p. 3
- Stochastic policies can make randomized-action exploration risky during adaptation, particularly in robotics tasks with differing system dynamics.The paper presents these concerns, together with Hessian-estimation difficulty and policy-gradient variance, as motivations for ES-MAML.Source: §2 · p. 3
- Despite its name, Evolvability ES targets diversity of behavioral characteristics rather than the standard MAML problem addressed by ES-MAML.The passage distinguishes the objectives of the two methods.Source: §2 · p. 3
3 ES-MAML Algorithms
ES-MAML applies Evolution Strategies to MAML, smoothing task rewards and estimating the meta-gradient without second derivatives. The framework also supports flexible adaptation operators and ES-based improvements to meta-training and adaptation.
3.1 Evolution Strategies (ES)
- ES optimizes a Gaussian-smoothed black-box reward, with gradient estimators that can use forward finite differences or symmetric antithetic differences.The paper notes that these variants are usually preferred over the basic estimator.Source: §3.1 · p. 4
3.2 Meta-Training MAML with ES
- Zero-order ES-MAML smooths the MAML reward and estimates its gradient by sampling tasks and perturbations, evaluating the adapted policy’s reward, without second derivatives.The approach treats task rewards as black boxes and allows nonsmooth rewards or adaptation operators; Algorithm 3 estimates adaptation with an ES gradient in the inner loop.Source: §3.2 · pp. 4–5
- An alternative first-order formulation retains a Hessian term, whose Monte Carlo estimator has high variance and is often harmful in training.The paper contrasts this “MAML of the smoothing” with Algorithm 3’s “smoothing of the MAML.”Source: §3.2 · pp. 5–6
- When the adaptation operator is estimated, the resulting estimator for the smoothed meta-gradient is generally biased.The paper compares this bias to that in PG-MAML, where the true adapted trajectory distribution is unavailable.Source: §3.2 · p. 6
3.3 Improving the Adaptation Operator with ES
- The framework permits improved ES gradient estimators and other ES methods to improve both meta-training and adaptation.The authors emphasize that ES-MAML can directly use newly developed ES enhancements.Source: §3 · p. 4, §3.1 · p. 4, §3.3 · p. 6
3.3.1 Improved Exploration
- DPP sampling selects diverse perturbations for the adaptation gradient and can use a learned, data-dependent kernel informed by meta-training.The authors report that DPP-ES can improve adaptation in MAML.Source: §3.3.1 · p. 6
3.3.2 Hill Climbing and Population Search
- Nondifferentiable local search can choose the highest-reward policy within a neighborhood; hill climbing makes monotonic reward progress, unlike the stochastic-gradient operator.The paper notes that this property appears beneficial on some difficult problems.Source: §3.3.2 · p. 6
4 Experiments
The experiments assess data efficiency through adaptation query count and meta-training speed. They compare exploration, policy architectures, stability, and low-query performance across reinforcement-learning tasks.
4.1 Exploration: Target Environments
- At K=20, basic ES-MAML finds four-corners targets without exploration modifications; HC explores six-circle targets, and HC/DPP-ES improve exploration when K is lower.Figure 1 compares exploration behavior and methods under limited K or large wrong-goal penalties; Figure 2 shows the six-circle task at K=20.Source: §4.1 · pp. 7–8, Figure 1 · p. 8, Figure 2 · p. 8
4.2 Good Adaptation with Compact Architectures
- On Forward-Backward and Goal-Velocity, ES-MAML trains successful linear policies faster than deep networks; linear policies can outperform hidden-layer policies, and HC is strongest on Forward-Backward Ant.Figure 3 compares linear and one-hidden-layer policies across different K values.Source: §4.2 · pp. 8–9, Figure 3 · p. 9
4.3 Deterministic Policies
- ES-MAML maintains maximum reward (500) on Biased-Sensor CartPole; linear policies outperform one- and two-hidden-layer policies on Swimmer and Walker2d.Figure 4 compares ES and PG stability across these environments and policy architectures.Source: §4.3 · p. 9, Figure 4 · p. 9
4.4 Low-K Benchmarks
- At low K, standard ES-MAML remains competitive: it is similar or better on Forward-Backward, while PG-MAML generally wins Goal-Velocity and can be highly unstable.The comparison covers four environments and two architectures; Figure 5 presents the low-K comparisons.Source: Figure 5 · p. 10, §4.4 · p. 10
5 Conclusion
ES-MAML applies Evolution Strategies to MAML, avoiding Hessian-estimation problems while allowing flexible adaptation operators. The authors report benefits from nonsmooth hill climbing and compact deterministic policies in the settings they study.
5 Conclusion
- ES-MAML avoids Hessian-estimation problems that complicated PG-MAML and is straightforward to implement.Source: §5 · p. 10
- Nonsmooth hill climbing yielded better exploratory behavior and performance on sparse-reward environments.Source: §5 · p. 10
- ES-MAML performs well with linear or compact deterministic policies, which can help when adapting under possibly unstable state dynamics.Source: §5 · p. 10
A.1.1 Algorithm
This section describes first-order ES-MAML: Gaussian smoothing makes the MAML objective differentiable, while its gradient can be estimated using a reward-gradient estimate and a Hessian estimate. The resulting estimate is supplied to SGD.
A.1.1 Algorithm
- Gaussian smoothing before forming the MAML problem makes the objective differentiable, so first-order methods can be applied directly.Source: §A.1.1 · p. 14
- The classical gradient includes a Hessian term, whose accurate estimation is a central difficulty in this formulation.Source: §A.1.1 · p. 14
- Algorithm 5 combines an independent adapted reward-gradient estimate with a Hessian estimate to form a gradient estimate for SGD.Source: §A.1.1 · p. 14
A.1.2 Experiments with First-Order ES-MAML
The first-order ES-MAML experiments compare variants with and without an estimated Hessian, and compare FO-NoHessian with Algorithm 3 by rollout count. Outcomes vary by task: omitting the Hessian helps on Forward-Backward Ant but performs poorly at low query counts on HalfCheetah.
A.1.2 Experiments with First-Order ES-MAML
- The authors interpret the pattern as potentially reflecting high-variance Hessian estimates that slow easier walking tasks but may increase exploration on four corners.Source: §A.1.2 · p. 15
- Prior PG-MAML findings are not directly comparable: cited work used modified Hessian estimators, and the cited supervised MAML comparison found omission slightly worse.Source: §A.1.2 · p. 15
- On Forward-Backward Ant, FO-NoHessian outperformed FO-Hessian, while on four corners FO-Hessian slightly outperformed FO-NoHessian amid large error bars.Source: Figure A1 · p. 15, §A.1.2 · p. 15
- FO-NoHessian was slightly faster than Algorithm 3 on Ant by rollout count, but at K = 5 on HalfCheetah it performed poorly and did not reach similar scores despite many more rollouts.Source: §A.1.2 · p. 15, Figure A2 · p. 16
A.2 Handling Estimator Bias
Estimating MAML gradients can be biased when the adapted policy cannot be evaluated exactly. The section discusses ES reformulations and E-MAML's alternative, while noting that a stochastic-composition method does not directly extend to full MAML.
A.2 Handling Estimator Bias
- Because the adapted policy generally cannot be evaluated exactly, unbiased estimates of its task reward are difficult to obtain in both PG-MAML and ES-MAML.Source: §A.2 · p. 16
- For PG-MAML, the nonlinear dependence on the adaptation operator requires exact sampling from adapted trajectories, so sampling from an estimated adaptation does not yield an unbiased gradient estimate.Source: §A.2 · p. 16
- E-MAML offers an unbiased gradient estimator by moving the expectation outside, but cited work argues this can impair credit assignment to the pre-adaptation policy in RL.Source: §A.2 · p. 16
- ES can reformulate the problem to avoid evaluating the adaptation operator exactly and thereby circumvent the estimator-bias question.Source: §A.2 · p. 17
- ASC-PG does not directly extend to full MAML because the outer task expectation breaks the required stochastic-composition form, though per-task solutions can fit a consensus framework.Source: §A.2 · p. 17
A.3 Extensions of ES
The section describes ways to improve ES gradient estimation in both meta-training and adaptation. These include variance-reduction methods and estimators with special properties.
A.3 Extensions of ES
- Improved ES gradient estimators can replace the estimator used for the adaptation operator, as well as improve the outer-loop meta-training gradient.The adaptation estimator may also depend on data collected during meta-training.Source: §A.3 · p. 17
- Quasi Monte Carlo sampling is one technique for reducing estimator variance, alongside methods with special properties.Source: §A.3 · p. 17
A.3.1 Active subspaces
Active Subspaces seeks a low-dimensional subspace that captures the gradient's strongest contribution, enabling optimization through its projection. In MAML, this can guide ES exploration using information from meta-training.
A.3.1 Active subspaces
- Active Subspaces updates a low-rank subspace to maximize the gradient projection and uses that projection without explicitly computing the gradient.The lower-dimensional optimization may be more computationally efficient.Source: §A.3.1 · p. 17
- In MAML, data from meta-training can be used to learn a subspace for adaptation, following ASEBO's guided, anisotropic ES exploration approach.Source: §A.3.1 · p. 18
A.3.2 Regression-Based Optimization
The appendix describes alternative gradient estimators and experiments, then covers Navigation-2D, RL benchmarks, supervised regression, and implementation settings. Its comparisons include estimator trade-offs and a regression setting where gradients are available.
A.3.2 Regression-Based Optimization
- RBO estimates gradients by regressing multiple finite differences, with regularization that can recover gradients despite substantial reward corruption.It uses zero-order evaluations but not Gaussian smoothing as in ES.Source: §A.3.2 · p. 18
A.3.3 Experiments
- With 25% of adaptation queries corrupted by enormous additive noise, RBO does not appear more robust than standard ES, suggesting ES has inherent noise robustness.Source: §A.3.3 · p. 18
- ASEBO-MAML outperforms ES-MAML in iterations on low-K Goal-Velocity HalfCheetah, but ES-MAML is more effective in real time because ASEBO requires additional linear algebra.Source: §A.3.3 · pp. 18–19, Figure A3 · p. 18
A.4 Navigation-2D Exploration Task
- On dense-reward Navigation-2D, PG-MAML triangulates the target with small movements, while ES-MAML uses parameter perturbations that produce differently directed paths.Source: §A.4 · p. 19, Figure A4 · p. 19
A.5 PG-MAML RL Benchmarks
- Figure A5 compares ES-MAML and PG-MAML on Forward-Backward and Goal-Velocity across four environments using the original experiments' query counts.Source: §A.5 · p. 20, Figure A5 · p. 20
A.6 Regression and Supervised Learning
- In sine regression, ES-MAML approaches PG-MAML's reported loss of ≈0.5 after one step with K = 5, but requires more training iterations; the comparison's iteration count for PG-MAML is unspecified.Because gradients are available for regression, the authors note that using gradients is more efficient than approximating them.Source: §A.6 · pp. 20–21, Figure A6 · p. 21
A.7.1 Environments
- RL experiments generally use K = 20 and horizon = 200; the Ant environments omit auxiliary costs to focus evaluation on the main task rewards.Source: §A.7.1 · p. 22
A.7.2 ES-MAML Hyperparameters
- Tasks are sampled without replacement, important when fewer than five distinct tasks are possible, and ES-MAML and PG-MAML use three seeded runs.Source: §A.7.2 · p. 22