Some Considerations on Learning to Explore via Meta-Reinforcement Learning

• Unlike supervised learning with fixed data, an RL policy can affect the distribution of states sampled during learning.The paper describes the policy as a sampling distribution over states.Source: §1 · p. 1
• Meta-RL aims to train agents to learn transferable knowledge that helps them generalize to new situations through adaptation.Source: §1 · p. 1
• The proposed sampling strategy seeks to inform adaptation to new environments, a scheme the authors describe as previously unexplored.Source: §1 · p. 2
• E-MAML optimizes per-task sampling during adaptation for expected future returns, while E-RL2 is a less principled extension of RL2.The paper presents both as methods for learning transferable structures for meta-adaptation.Source: §1 · p. 2
• The authors report that their algorithms outperform baselines on Krazy World and present maze results to check for environment-specific overfitting.Source: §1 · p. 2

• The task family is modeled as a distribution of MDPs, from which a task is sampled for adaptation.Source: The Meta Reinforcement Learning… · p. 2
• Meta-RL seeks a policy and update method that solve sampled tasks quickly, meaning more sample-efficiently than solving each task from scratch.The paper illustrates the target with tasks requiring over 100,000 samples for good returns, where an ideal method would need fewer than 10 trajectories.Source: The Meta Reinforcement Learning… · p. 2
• The meta-RL objective is similar to MAML's, where the update operator is stochastic gradient descent parameterized by the task.Source: The Meta Reinforcement Learning… · p. 2

3.1 Fixing the Sampling Problem with E-MAML

• Half-cheetah is characterized as a weak benchmark because it can be learned in a few gradient steps from a random prior or solved with a linear policy.The passage urges against using it as a benchmark.Source: §3.1 · p. 2
• E-MAML adds a sampling-distribution gradient term so initial trajectories can be reinforced according to their effect on post-adaptation returns.The authors describe its exploratory interpretation as an open question.Source: §3.1 · p. 3

3.2 Choices for the Update Operator U, and the Exploration Policy πθ

• With one inner policy-gradient iteration, the initial policy determines exploratory sampling; with more iterations, exploration also depends on the update operator and their interplay.The experiments assume stochastic gradient descent for U, though other choices are possible.Source: §3.2 · p. 3, §3.1 · p. 3

3.3 Understanding the E-MAML Update

• E-MAML treats the inner policy-gradient update as stochastic, whereas MAML treats it as deterministic and omits the update operator's causal entropy term.The stochasticity can be tuned with λ.Source: Figure 2 · p. 4, §3.3 · p. 4

3.4 E-RL2

• E-RL2 separates Explore-rollouts from Exploit-rollouts, zeroing Explore-rollout rewards in the return used for the policy gradient while retaining their information in the forward pass.The proposed approach aims to encourage sampling that supports later system identification and higher rewards in subsequent episodes.Source: §3.4 · p. 5

4.1 Krazy World Environment

• Krazy World tests whether meta-updates account for how per-task sampling affects post-adaptation reward, since agents must identify changing tile types and dynamics.Example dynamics include reward, energy replenishment, obstacles, hazards, keys, and sliding tiles; these must be re-identified in each sampled world.Source: §4.1 · pp. 5–6, Figure 3 · p. 5

4.2 Mazes

• The maze tasks place the agent at a random square in an unrendered 20 × 20 maze, requiring navigation to the goal using state information.Exploration should learn the layout while minimizing repeated mistakes.Source: §4.2 · p. 6

4.3 Results

• On Krazy World, E-RL2 achieves the best final performance, while E-MAML gains fastest early and MAML reaches comparable performance more slowly.E-RL2 passes E-MAML around 14 million training steps and converges by 25 million; RL2 performs comparatively poorly with high variance.Source: Figure 4 · p. 6, §4.3 · p. 6
• On mazes, RL2 and E-RL2 outperform MAML and E-MAML; E-RL2 solves many mazes, while RL2 and E-MAML do so sporadically.The authors suspect RNN memory matters more in mazes; MAML often learns only to avoid the wall penalty, which E-RL2 and RL2 discover more slowly.Source: Figure 5 · p. 6, §4.3 · pp. 6–7
• All algorithms improve after an update; the comparison measures initial versus updated performance, using one policy-gradient step for MAML and three exploratory episodes for RL2.The authors treat a larger gap as a sign of adaptation; the RL2 measurement allows its RNN hidden state time to update.Source: Figure 6 · p. 7, §4.3 · p. 7
• On Krazy World, the exploration heuristics track tile-type coverage, repeated deaths at death tiles, and goal visits; E-MAML checks tile types most diligently.RL2 tends to visit fewer goals, often finding one and exploiting it; the authors interpret improved heuristic scores as evidence of at least some system identification.Source: Figure 7 · p. 7

5 Related Work

• Deep RL advances support discrete control from raw images and high-dimensional continuous control from raw state information.Source: §5 · p. 7
• Exploration remains difficult in RL; high-dimensional observations, implementation challenges across domains, and weak results have limited practical use of many methods.Deep RL commonly uses epsilon-greedy or Boltzmann exploration.Source: §5 · p. 8
• Hierarchical RL emphasizes reusable primitives for state coverage, whereas meta-learning directly optimizes for such behaviors rather than specifying architectures intended to produce them.Source: §5 · p. 8
• Prior similar approaches emphasize curiosity or free-learning phases and intrinsic motivation signals, while this work uses existing rewards and connects those phases to meta-updates.Source: §5 · p. 8

6 Closing Remarks

• The two algorithms tend to learn faster and cover more of their environments’ states during learning than existing algorithms.Source: §6 · p. 8
• Future work may meta-learn curiosity signals robust across tasks or learn an explicit exploration policy.Source: §6 · p. 8
• Learning intrinsic rewards that communicate long-horizon goals is another proposed direction, potentially supporting agents that want to explore rather than being forced by their objectives.Source: §6 · p. 8

Appendix A: Krazy-World

• After 5 million Q-Learning steps per test environment, the mean score was 1.2 per environment, below the human baseline of 2.7.The paper describes the environment as challenging even for state-of-the-art RL algorithms.Source: Appendix A · p. 8
• Eight tile types create distinct mechanics, including goals, ice, death, walls, keys and locks, teleporters, energy, and ordinary movement.Goals give +1 reward and do not terminate an episode; death ends it, while running out of energy prevents further movement.Source: Appendix A · p. 8
• Random color permutations change which colors represent tile types, requiring the agent to identify the new system to score well.The experiments use basis-vector representations, which the paper says are more sample-efficient.Source: Appendix A · p. 9
• Dynamics can be randomized by permuting input-action mappings or allowing multi-step movement in arbitrary directions.Source: Appendix A · p. 9
• The experiments use local observations, in which the agent sees a fixed area around itself rather than the entire grid.Figure 8 contrasts a centered 3 × 3 local view with global access to the whole environment.Source: Figure 8 · p. 8, Appendix A · p. 9

Appendix B: Table of Hyperparameters

• Table 1 presents the hyperparameters for E-MAML.Source: Table 1 · p. 9

Appendix C: Experiment Details

• Each iteration initializes 32 training and 64 test environments, each with a different seed, and samples algorithm hyperparameters uniformly from predefined ranges.Data from the training environments informs a meta-update before the algorithm takes a test-time training step.Source: Appendix C · p. 9
• Test-time evaluation uses one VPG update for MAML and E-MAML, but three exploratory episodes followed by loss measurement on episodes four and five for RL2 and E-RL2.Meta-updates for both algorithm families are calculated with PPO [27].Source: Appendix C · p. 9
• The procedure repeats 64 times from the beginning, and the resulting test-environment results are averaged into the final learning curves.Source: Appendix C · p. 9