Improving Generalization in Meta Reinforcement Learning using Learned Objectives

• Artificial agents rely on human-engineered learning rules, limiting them to the inductive biases their designers can incorporate.Source: §1 · p. 1
• Earlier meta-RL work generalized to slightly different environments or goals, but transfer to entirely different environments remained open.Source: §1 · p. 1
• MetaGenRL distills experiences from many agents into a low-complexity neural objective that determines how future agents learn, using second-order gradients.The authors contrast this approach with EPG, which they say does not use second-order gradients.Source: §1 · p. 1
• MetaGenRL does not overfit like RL2 and trains randomly initialized agents in entirely different environments; it is more sample efficient than EPG and outperforms it under a fixed interaction budget.The evaluation compares MetaGenRL with RL2, EPG, and human-engineered algorithms on continuous-control tasks.Source: §1 · pp. 1–2

• The standard MDP is specified by states, actions, transition probabilities, initial-state distribution, reward, discount factor, and episode length; the policy objective is expected discounted return.A trajectory records states, actions, and rewards over an episode.Source: §2 · p. 2
• REINFORCE estimates policy gradients using the likelihood-ratio trick, while GAE combines with REINFORCE to form another policy-gradient estimator.TRPO and PPO are described as extensions that discourage bad updates, with PPO using first-order approximations.Source: §2 · p. 2
• Because policy-gradient estimators can be viewed as objectives differentiated with respect to policy parameters, a neural objective can be optimized to maximize return across different environments.Source: §2 · p. 2

3 META-LEARNING NEURAL OBJECTIVES

• MetaGenRL uses a shared neural objective, learned with value functions and second-order gradients, to generalize across vastly different environments.The schematic shows each agent’s policy updated through the shared objective while its critic uses TD-error updates.Source: §3 · p. 2, Figure 1 · p. 3

3.1 FROM DDPG TO GRADIENT-BASED META-LEARNING OF NEURAL OBJECTIVES

• The critic measures how changing the objective affects policy gradients, enabling a population of agents to improve one objective; at meta-test time, that fixed objective trains a randomly initialized policy.Meta-training alternates critic, objective, and policy updates, and can use replay-buffer trajectories.Source: §3 · p. 3, §3.1 · p. 4

3.2 PARAMETRIZING THE OBJECTIVE FUNCTION

• The LSTM objective processes trajectories in reverse, using rewards, actions, policy actions, time, and value estimates to produce per-step values that sum to a differentiable scalar.Its action embeddings accommodate different action dimensionalities; the authors note that it is difficult to assess whether the learned objective incorporates bias- or variance-reduction techniques.Source: §3.2 · p. 4

3.3 GENERALITY AND EFFICIENCY OF METAGENRL

• MetaGenRL shares one objective across agents in potentially different environments, combining their gradients; off-policy second-order training avoids evaluating fully trained random agents, which the authors say speeds credit assignment.The paper attributes generalization in part to separating the policy from the learning rule, which makes it difficult for the meta-learner to overfit training environments.Source: §3.3 · p. 5

4 RELATED WORK

• Early meta-learning included genetic-algorithm meta-hierarchies, supervised-learning update rules with free parameters learnable by gradients, and a differentiable self-referential RNN able to modify its own weights.Source: §4 · pp. 5–6
• RL2 uses recurrent networks for reinforcement-learning meta-learning [Wang et al., 2016; Duan et al., 2016], but the authors report weak generalization and overfitting to training environments.They attribute this to the lack of separation between policy and objective and an O(n^2)-to-O(n) parameter-to-activation imbalance.Source: §4 · p. 6
• Policy-initialization methods use second-order gradients on the policy parameters [Schulman et al., 2017], rather than a separate learned objective.The authors describe their update rules as complicated and generalization as difficult.Source: §4 · p. 6
• Earlier learned-objective approaches include Evolved Policy Gradients and learned losses for RL [Bechtle et al., 2019], which the authors distinguish by interaction cost or task-distribution scope.Other work learns optimizers that modulate gradients of a fixed objective [Li & Malik, 2016; 2017; Andrychowicz et al., 2016].Source: §4 · p. 6
• Meta-learning an objective can also include learning the gradient estimator itself, whereas intrinsic-reward methods normally keep that estimator fixed.The paper also contrasts this with prior transfer-learning approaches that rely on human-engineered learning algorithms [Kistler et al., 1997; Parisotto et al., 2015; Rusu et al., 2016; 2019; Nichol et al., 2018; Sung et al., 2017].Source: §4 · p. 6

5 EXPERIMENTS

• The experiments use continuous-control tasks whose system properties and dynamics differ, with meta-training and testing across environments used to assess out-of-distribution generalization.Unless otherwise stated, MetaGenRL uses 20 agents and 600K interactions per agent for meta-training, followed by 1M meta-test interactions.Source: §5 · pp. 6–7

5.1 COMPARISON TO PRIOR WORK

• Across previously seen environments, MetaGenRL generally beats PPO and REINFORCE, usually matches DDPG except on Cheetah, and consistently beats EPG and often RL2.On Lunar, MetaGenRL is similar when meta-trained on Lunar and Cheetah; EPG does not find a good objective even with one billion interactions.Source: Table 1 · p. 7, §5.1 · pp. 7–8
• On unseen Hopper, MetaGenRL outperforms PPO and REINFORCE, and its best configuration also outperforms DDPG; RL2 fails and EPG remains poor.The comparison evaluates randomly initialized agents trained with learned objectives on significantly different environments.Source: Figure 3 · p. 8, §5.1 · p. 8

5.2.1 META-TRAINING PROGRESSION OF OBJECTIVE FUNCTIONS

• After 28K meta-training steps, the objective yields only marginal test-time improvement on Hopper, while after 86K steps it makes consistent progress and later becomes faster, more stable, and lower-variance.The objective is evaluated at five stages after meta-training on Cheetah and Lunar.Source: §5.2.1 · pp. 8–9, Figure 4 · p. 9

5.2.2 ABLATION STUDY

• Without time input t, the objective still performs well at meta-test but converges more slowly on Cheetah during meta-training.Figure 5 compares alternative parametrizations trained on Lunar and Cheetah and tested on Cheetah.Source: Figure 5 · p. 9, §5.2.2 · p. 9

5.2.3 MULTIPLE GRADIENT STEPS

• Using three inner gradient steps slightly increases test-time variance on Hopper and Cheetah, while five steps harms performance.The comparison uses one, three, or five inner steps on LunarLander and HalfCheetah.Source: Figure 6 · p. 10, §5.2.3 · p. 10

5.2.2 ABLATION STUDY

• Removing value estimates Vt prevents meaningful test-time progress, whereas removing only Vt+1 can sometimes increase performance.The authors note that the no-value ablation also exhibits possibly large variance during meta-training.Source: §5.2.2 · pp. 9–10

6 CONCLUSION

• MetaGenRL learns general objective functions with off-policy gradients from a population of DDPG-like agents.The authors argue that generalization across different tasks relates to separating the policy from its learning rule, the objective’s functional form, and training across multiple agents and environments.Source: §6 · p. 10
• Second-order gradients increase MetaGenRL’s sample efficiency by several orders of magnitude compared with EPG.The comparison is attributed to Houthooft et al. (2018).Source: §6 · p. 10
• The current objective cannot observe environment observations or the hidden state of the recurrent policy.The authors identify these as areas for future extensions.Source: §6 · p. 10

Hopper

• On Hopper, MetaGenRL performs well in within- and out-of-distribution testing and significantly outperforms PPO, REINFORCE, RL2, and EPG.RL2 obtains reward only when Hopper is included in meta-training; EPG shows some progress on one split but none on another.Source: §A.1 · p. 15, Figure 7 · p. 15

Cheetah

• On Cheetah, MetaGenRL significantly outperforms PPO and REINFORCE, but DDPG notably outperforms MetaGenRL.RL2 and EPG have only within-distribution results because Cheetah has larger observation and/or action spaces than Hopper and Lunar.Source: §A.1 · p. 15, Figure 8 · p. 16

Lunar

• On Lunar, MetaGenRL is marginally better than PPO and REINFORCE within distribution, but worse out of distribution.Its out-of-distribution performance still exceeds RL2 and EPG; the authors report large variance from runs that become stuck early or recover only after delay.Source: §A.1 · pp. 15–16, Figure 9 · p. 16

Comparing final scores

• MetaGenRL outperforms PPO and off-/on-policy REINFORCE in most configurations, while DDPG with TD3 tricks is stronger on two of three environments.Table 2 reports final mean returns across seeds for previously seen and unseen test environments; DDPG is not currently representable by MetaGenRL.Source: §A.1 · p. 16

A.2 STABILITY OF LEARNED OBJECTIVE FUNCTIONS

• One Lunar run converged to a local optimum early and could not recover, while other runs remained stuck for a long time before rapid learning progress.The authors suggest the objective can sometimes struggle to provide meaningful policy-parameter updates during early training.Source: §A.2 · pp. 16–17
• Early Lunar meta-training convergence leaves later objective updates based mostly on converged policies, and those updates appear to hurt test-time performance.The authors hypothesize that the objective then “forgets” early training of randomly initialized agents; Figure 10 evaluates the objective at different training stages.Source: §A.2 · p. 17, Figure 10 · p. 17
• Excluding four of twelve random seeds significantly reduces variance and increases the mean of MetaGenRL’s results.Figure 11 shows the Lunar meta-test results with all seeds and with the four worst seeds removed.Source: §A.2 · p. 17

A.3 ABLATION OF AGENT POPULATION SIZE AND UNIQUE ENVIRONMENTS

• A population of 20 agents is used during meta-training to diversify the conditions under which the objective function must optimize.Population size is described as crucial for stable meta-optimization, which becomes increasingly unstable as the population shrinks.Source: §A.3 · p. 17
• Adding Cheetah, Lunar, Walker, and Ant as meta-training environments does not significantly improve Hopper results and sometimes decreases performance.The authors suggest the simple objective’s lack of access to environment observations or hyperparameters needing further tuning may explain the outcome; Figure 13 compares 20- and 40-agent setups.Source: §A.3 · pp. 17–18, Figure 13 · p. 18

B EXPERIMENT DETAILS

• The experiment-details section covers architectures, meta-training, hyperparameters, and baselines, and notes that reproduction code is available.The passage provides a code URL.Source: B EXPERIMENT DETAILS · p. 19

B.1 NEURAL OBJECTIVE FUNCTION ARCHITECTURE

• An LSTM processes each trajectory backward, receiving rewards, value estimates, timestep, and both replay-buffer and policy actions.The architecture uses one-dimensional convolutions over actions followed by a mean reduction, allowing different action sizes across environments.Source: §B.1 · p. 19
• The action inputs are transformed using a learnable matrix, with optional nonlinearities; ReLU and square activations were helpful but not required.The feed-forward critic and policy use ReLU and layer normalization, while the LSTM uses tanh state transformations and sigmoid gates.Source: §B.1 · p. 19
• Each LSTM step outputs a scalar bounded by −η and η; summing these values forms the objective, whose policy gradients improve πφ.Gradients flow backward through πφ(st) to φ; the implementation is closely related to REINFORCE with generalized advantage estimation.Source: §B.1 · p. 19
• Adding full observation sequences, bidirectional recurrence, or policy hidden states could expand the objective’s expressiveness, but these options were not explored.The authors suggest such inputs could support predictive representations and pressure policy actions to explore unknown dynamics.Source: §B.1 · p. 19

B.2 META-TRAINING

• Early in meta-training, the randomly initialized objective can irreversibly break policies; its gradients are linearly annealed with DDPG gradients for the first 10k timesteps, about 2% of training.An exponential schedule was preliminary and could still yield unsuccessful runs, so it was omitted.Source: §B.2 · p. 19
• For each environment timestep, one replay-buffer gradient update is applied; the critic is updated twice as often as the policy and objective.The objective gradient is combined with φ using a fixed learning rate, while other parameters use Adam.Source: §B.2 · p. 20
• The multi-step analysis applies objective-driven policy updates before critic-based second-order gradients, using disjoint batches; the policy update itself remains a single step.The setup can extend beyond two policy updates before computing the second-order gradients.Source: §B.2 · p. 20

B.3 BASELINES

• RL2 uses PPO rather than TRPO on MuJoCo; other baselines include official EPG, tuned PPO and on-policy REINFORCE, off-policy GAE, and DDPG.The off-policy GAE implementation uses replay-buffer experience; an importance-weighted unbiased estimator performed poorly.Source: §B.3 · p. 20
• Architecture and training hyperparameters are listed in Tables 3 and 4.The table captions identify their scopes as architecture and training hyperparameters, respectively.Source: Table 3 · p. 21, Table 4 · p. 21