Source-linked AI summary
Meta-Reinforcement Learning of Structured Exploration Strategies
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, Sergey Levine
TL;DR
Reinforcement-learning exploration is often task agnostic despite agents facing multiple related tasks, motivating methods that use prior experience. MAESN combines gradient-based fast adaptation with a learned latent exploration space, and the paper reports stronger exploration than prior meta-RL, learning-from-scratch, and task-agnostic approaches across simulated tasks.
Problem
Existing exploration methods are largely task agnostic, while agents learning multiple related tasks could use prior experience to guide exploration in new tasks.
Method
MAESN combines policy-gradient meta-learning with a learned latent space that injects temporally correlated structured stochasticity and adapts its policy and latent distribution to new tasks.
Results
MAESN outperforms prior meta-learning methods and reinforcement-learning algorithms trained from scratch, including task-agnostic exploration methods, across manipulation and locomotion evaluations.
Takeaways & Limitations
Structured, task-aware stochasticity allows exploration to sample useful behaviors while omitting behaviors that are never useful.
Takeaways & Limitations
Meta-training remains challenging when tasks are too difficult to solve from scratch, so the experiments use reward shaping during meta-training.
Abstract
from arXiv · showhide
Exploration is a fundamental challenge in reinforcement learning (RL). Many of the current exploration methods for deep RL use task-agnostic objectives, such as information gain or bonuses based on state visitation. However, many practical applications of RL involve learning more than a single task, and prior tasks can be used to inform how exploration should be performed in new tasks. In this work, we explore how prior tasks can inform an agent about how to explore effectively in new situations. We introduce a novel gradient-based fast adaptation algorithm -- model agnostic exploration with structured noise (MAESN) -- to learn exploration strategies from prior experience. The prior experience is used both to initialize a policy and to acquire a latent exploration space that can inject structured stochasticity into a policy, producing exploration strategies that are informed by prior knowledge and are more effective than random action-space noise. We show that MAESN is more effective at learning exploration strategies when compared to prior meta-RL methods, RL without learned exploration strategies, and task-agnostic exploration methods. We evaluate our method on a variety of simulated tasks: locomotion with a wheeled robot, locomotion with a quadrupedal walker, and object manipulation.
1. Introduction
Exploration becomes difficult in complex, temporally extended tasks, while many existing methods are task agnostic. MAESN uses prior related tasks to learn structured stochastic exploration that adapts through policy gradients.
- Complex or temporally extended tasks make naive exploration strategies less effective, making effective exploration a critical reinforcement-learning challenge.
- Many existing exploration methods guide behavior using task-agnostic criteria such as intrinsic motivation, visitation counts, uncertainty, or parameter-space perturbations.
- Prior tasks can inform exploration in new tasks because related experience may let agents explore new skills without reacquiring already learned abilities.
- Existing meta-RL methods struggle because effective exploration requires stochasticity, whereas optimal policies are often deterministic, and recurrent learned algorithms may lack convergent iterative optimization.
- MAESN combines policy-gradient adaptation with learned latent structured stochasticity, using prior experience to initialize policies and generate temporally coherent exploration behaviors.
- MAESN’s experiments compare it with prior meta-RL, learning from scratch, and task-agnostic methods across manipulation and locomotion tasks.
2. Related Work
Prior exploration methods are generally task agnostic and do not exploit knowledge from related tasks. MAESN adds meta-learned structured stochasticity to latent-space exploration and reports advantages over prior approaches.
- Sophisticated exploration methods improve exploration but generally remain task agnostic, leaving prior knowledge from other tasks unused.
- Meta-learning methods aim to learn models, optimizers, update rules, or reinforcement-learning algorithms that adapt quickly to new tasks.
- MAESN introduces structured stochasticity through a learned latent space, unlike prior latent-space methods that do not explicitly train for fast adaptation.
- The paper contrasts MAESN with concurrent meta-exploration work that lacks structured stochasticity and reports no significant improvement over MAML and other methods.
3. Preliminaries
Meta-RL optimizes adaptation to new tasks drawn from a task distribution. The paper builds on MAML’s policy-gradient adaptation and preserves conventional policy-gradient behavior outside the training distribution.
- The paper introduces meta-learning and MAML as frameworks for learning parameters that improve performance after policy-gradient adaptation.
- Meta-RL models tasks as distinct MDPs sampled from p(τ), with task-specific transitions and rewards, and seeks efficient adaptation to novel tasks.
- MAML’s objective optimizes initial policy parameters so one policy-gradient step improves performance across meta-training tasks.
- Because MAML reverts to conventional policy gradient on out-of-distribution tasks, it provides a baseline that preserves at least the asymptotic performance of learning from scratch.
4. Model Agnostic Exploration with Structured Noise
MAESN combines gradient-based meta-learning with a learned latent space to produce temporally coherent exploration that adapts quickly to new tasks. It addresses limitations of action-space noise and recurrent meta-RL by optimizing both policy and latent distributions for post-update performance.
- Overview: MAESN combines MAML-style gradient adaptation with a learned latent space that injects structured, temporally correlated stochasticity into the policy.A latent variable is sampled per episode while actions remain sampled per time-step.
- Adaptation: At meta-test time, each trial samples a latent variable whose distribution is adapted to the new task using policy-gradient updates.The fixed per-episode latent sample supplies structured exploration while the distribution parameters are updated for the task.
- Motivation: Typical policies add independent action-space noise at each time-step, limiting their ability to represent coherent exploratory behavior.The policy can effectively change its exploration intention from one time-step to the next.
- Meta-learning objective: MAESN meta-trains shared policy parameters and per-task latent-distribution parameters to maximize reward after a policy-gradient update.The inner update uses REINFORCE, while the meta-optimization differentiates through that update.
- Meta-learning objective: The objective combines post-update expected reward with a KL-divergence term between each task’s variational distribution and the prior.The KL term regularizes the per-task latent distributions toward the prior.
- Adaptation: Meta-training commonly leaves task variational parameters near the prior at convergence, enabling new tasks to begin from the prior when no good initialization is available.This occurs because optimization targets rewards after updating the variational parameters, not only before the update.
5. Experiments
The experiments evaluate MAESN across diverse robotic manipulation and locomotion task distributions, comparing its adaptation and exploration against prior methods. Results show faster, stronger learning on sparse-reward novel tasks, coherent exploration, and structured latent adaptation.
- Experimental design: Experiments compare MAESN with MAML, RL2, latent-space learning, learning from scratch, and task-agnostic exploration across three task distributions.The distributions cover wheeled locomotion, quadrupedal locomotion, and robotic block manipulation.
- Task setup: The manipulation tasks require moving the task-relevant block among randomly positioned blocks to a goal location using a robotic hand.Different tasks vary which block matters, its position, and the goal location.
- Task setup: The locomotion tasks require wheeled and quadrupedal robots to discover randomly located goals, with quadrupedal exploration additionally requiring coordinated walking.At meta-test time, rewards are sparse and provided near the goal.
- Adaptation to new tasks: MAESN explores and adapts faster on sparse-reward validation tasks than MAML, RL2, latent-space learning, learning from scratch, and task-agnostic exploration.Rewards are averaged across 100 validation tasks, and the comparison covers all three task families.
- Adaptation to new tasks: Only MAESN adapts effectively on the challenging legged-locomotion task, while learning from scratch approaches MAESN asymptotically on manipulation but learns more slowly.Learning from scratch is competitive with prior meta-learning methods in asymptotic performance on many tasks.
- Exploration strategies: Visualizations show MAESN produces broad coherent exploration that better captures task distributions than random exploration and MAML.The trajectories represent manipulator position for block pushing and center-of-mass position for locomotion.
- Latent space structure: In the learned latent space, pre-update distributions approach N(0, I), while policy-gradient updates move post-update distributions to task-specific locations.This preserves effective prior initialization while enabling task adaptation.
- Latent space structure: Structured latent noise produces a wider range of ant trajectories than keeping the latent variable fixed, indicating that the learned noise is used for exploration.The comparison evaluates center-of-mass visitation with and without latent-space sampling.
6. Conclusion
MAESN combines gradient-based meta-learning with a learned latent exploration space to produce temporally coherent stochasticity for fast adaptation. Experiments show it outperforms prior meta-learning and task-agnostic alternatives, while remaining compatible with intrinsic-motivation methods.
- MAESN combines gradient-based meta-learning with a learned latent exploration space.
- The learned latent space injects temporally correlated, coherent stochasticity during meta-test exploration.
- MAESN samples useful behaviors while omitting behaviors that are never useful.
- MAESN outperforms prior meta-learning methods and from-scratch RL algorithms using task-agnostic exploration.
- The approach can potentially be combined with novelty-based methods such as VIME and pseudocount-based exploration.
A. Experimental Details
The implementation uses small feedforward policies and one-step meta-training. Meta-learning a separate stepsize for each parameter is crucial for MAESN to work well.
- The policies are two-layer feedforward networks with 100 units per layer and ReLU nonlinearities.
- Meta-training uses a single adaptation step, although longer adaptation could be used in principle.
- Per-parameter stepsizes are crucial for MAESN, rather than using a fixed stepsize.
B. Reward Functions
The experiments use dense rewards during meta-training and sparser rewards at test time. Rewards remain uninformative until the agent or object reaches a threshold distance from the goal, after which negative distance is provided.
- Dense reward functions are used during training for all tasks to enable meta-training.
- At test time, rewards are sparse and provided only within a region around the target position.
- The block-task reward uses negative object-goal distance when the distance is at most 0.2, otherwise assigning −cmax.
- The locomotion reward uses negative center-of-mass-to-goal distance when the distance exceeds 0.8, otherwise assigning −cmax.
- The reward is uninformative until the agent or object reaches a threshold distance around the goal, after which negative distance is provided.
C. Ablation Study
The ablation study isolates bias transformation and adaptive stepsizes within MAML variants. Bias transformation helps, but does not match MAESN’s performance.
- The ablation compares MAESN with MAML variants using bias transformation, adaptive stepsizes, or both.
- Adding a learned bias-transformation latent space is helpful but does not match MAESN’s performance.
- The variants include standard MAML, adaptation of all parameters, and adaptation of only bias parameters.