Source-linked AI summary
Meta Learning Shared Hierarchies
Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, John Schulman
TL;DR
Reinforcement learning typically lacks prior knowledge for rapidly solving new tasks, motivating a method that shares useful behavior across a task distribution. The paper learns hierarchically organized primitives and task-specific switching policies end-to-end, achieving speedups and diverse locomotion behaviors across complex environments.
Problem
Reinforcement learning often learns each task from scratch, creating a need for shared prior knowledge that supports rapid adaptation to unseen tasks.
Method
MLSH learns shared sub-policies that operate for extended periods and task-specific master policies that switch among them, training the hierarchy over sampled tasks.
Results
The method achieves significant learning speedups, discovers diverse primitives without hand engineering, and supports locomotion and sparse-reward transfer across physics environments.
Takeaways & Limitations
Shared hierarchies can produce meaningful task-transferable behaviors solely through interaction with task distributions, including four-legged directional movement and humanoid walking and crawling.
Takeaways & Limitations
The method relies on approximations, including optimizing sub-policies within a single episode rather than across the full T-step training horizon, and assumes warmup learns an optimal master policy.
Abstract
from arXiv · showhide
We develop a metalearning approach for learning hierarchically structured policies, improving sample efficiency on unseen tasks through the use of shared primitives---policies that are executed for large numbers of timesteps. Specifically, a set of primitives are shared within a distribution of tasks, and are switched between by task-specific policies. We provide a concrete metric for measuring the strength of such hierarchies, leading to an optimization problem for quickly reaching high reward on unseen tasks. We then present an algorithm to solve this problem end-to-end through the use of any off-the-shelf reinforcement learning method, by repeatedly sampling new tasks and resetting task-specific policies. We successfully discover meaningful motor primitives for the directional movement of four-legged robots, solely by interacting with distributions of mazes. We also demonstrate the transferability of primitives to solve long-timescale sparse-reward obstacle courses, and we enable 3D humanoid robots to robustly walk and crawl with the same policy.
1 INTRODUCTION
The paper addresses slow reinforcement learning on new tasks by sharing motor primitives across related task distributions while adapting task-specific master policies. It formulates and approximately optimizes hierarchies for rapid learning, demonstrating transfer across locomotion and sparse-reward environments.
- Reinforcement learning agents often solve tasks independently from scratch, requiring more experience than humans and facing speed limits without prior knowledge.
- The proposed architecture shares motor primitives across related tasks and uses task-specific master policies to switch between them.The method is related to the options framework and aims to learn new tasks by updating only the master policy.
- The paper formulates hierarchy quality as finding low-level primitives that enable the master policy to be learned quickly.
- The optimization algorithm repeatedly resets master policies so shared sub-policies adapt for fast learning.
- Experiments cover continuous movement, gridworlds, and 3D physics, including directional primitives for four-legged robots and shared humanoid walking and crawling.The method also transfers sub-policies to sparse-reward obstacle courses and outperforms previously proposed algorithms.
2 RELATED WORK
The paper connects its approach to two research traditions: hierarchical reinforcement learning, which recombines temporally extended primitives, and metalearning, which uses prior experience for rapid task adaptation.
- Hierarchical reinforcement learning: Hierarchical reinforcement learning accelerates learning by recombining temporally extended primitives, with Options as a prominent formulation.Earlier work assumed options were given, while later work learned them automatically or end-to-end.
- Metalearning: Metalearning uses information from past experiences to learn quickly on specific tasks through recurrent networks, temporal convolutions, or parameter-update mechanisms.
3 PROBLEM STATEMENT
The problem statement seeks shared parameters that support high return while agents adapt task-specific parameters on newly sampled MDPs. MLSH implements this with sub-policies selected by a slower-timescale master policy.
- The optimization problem considers a distribution over tasks and seeks parameters that enable agents to learn quickly on sampled tasks.
- An MDP is defined through state and action spaces and a transition function mapping the current state and action to the next state and reward.
- Shared parameters φ remain fixed across tasks, while randomly initialized per-task parameters θ are updated during interaction with each sampled MDP.The agent interacts for T timesteps over multiple episodes while iteratively updating θ.
- The meta-learning objective maximizes expected lifetime return across sampled tasks as agents adapt θ from scratch.The lifetime return is the sum of rewards over T timesteps.
- The architecture divides φ into sub-policy parameters φ1 through φK, while θ parameterizes a master policy that selects the active sub-policy.The master policy makes selections every N timesteps, whereas sub-policies operate at the faster action timescale.
- Because the master policy acts every N timesteps, its learning horizon is 1/N as long as the original horizon, enabling faster adaptation when strong sub-policies are available.
4 ALGORITHM
MLSH learns shared sub-policies that task-specific master policies can select, alternating master-policy warmup with joint updates of master and sub-policy parameters. Experience is interpreted at different timescales: master actions persist for N timesteps, while sub-policy updates target only the activated primitive.
- The method seeks sub-policies that let newly initialized agents maximize reward over T-step interactions across task distributions.
- MLSH begins with randomly initialized parameters and alternates master-policy warmup with joint optimization of master and sub-policy parameters.The algorithm is organized into a warmup period followed by a joint update period.
- After each task-specific training cycle, MLSH samples a new task, resets the master policy, and repeats while retaining the shared sub-policies.
- During warmup, the master policy selects sub-policies as single actions, with each selection covering the next N timesteps as one environment transition.The master policy is updated with an arbitrary reinforcement learning algorithm using the resulting experience.
- During joint updates, the master policy is optimized at the 1/N timescale while each N-timestep slice updates only the sub-policy activated by the master policy.The master decision is treated as part of the sub-policy observation, and actions at other timesteps are ignored for that sub-policy update.
5 RATIONALE
The rationale is to adapt sub-policies when the master policy is informative and nearly optimal, while limiting updates before retraining the master policy. This balances task-specific coordination with the changing behavior induced by sub-policy updates.
- MLSH optimizes sub-policies for reward within a single episode rather than over the entire inner training loop.The rationale assumes the warmup period will learn an optimal or near-optimal master policy.
- Warmup matters because optimal primitives can depend on the master policy’s task information: random routing favors midpoint-reaching behaviors, whereas informed routing favors separate destination-reaching behaviors.
- The method stops the joint update after U iterations because changing sub-policies makes the assumption of a stable master policy progressively less reliable.The master policy is then retrained once the threshold is reached.
6 EXPERIMENTS
The experiments evaluate MLSH across moving-bandit, gridworld, and physics tasks, comparing shared hierarchies with baselines and testing transfer to complex environments. MLSH learns meaningful movement sub-policies, outperforms baselines on Twowalk and Walk/Crawl, and supports exploration in sparse-reward obstacle courses.
- 2D Moving Bandits: In 2D moving bandits, MLSH learns separate sub-policies toward both potential goals, whereas a shared policy ignores one goal and cuts expected reward by half.The master policy therefore focuses on identifying the correct goal rather than learning primitive movement from scratch.
- 2D Moving Bandits: Removing warmup initially makes both sub-policies move toward the same goal, while removing temporal extension produces behavior similar to learning a shared policy.These ablations identify warmup and temporally extended sub-policies as important components of the method.
- Four Rooms: In Four Rooms, MLSH reaches high reward faster than the PPO baseline when fine-tuned on an unseen goal position.Option Critic performs similarly to its corresponding baseline in this comparison.
- Physics environments: In physics environments, MLSH significantly outperforms baselines on Twowalk and Walk/Crawl while learning directional Ant primitives and diverse Humanoid walking and crawling sub-policies.The learned Ant primitives are used to explore mazes, and the Walk/Crawl setting tests multiple locomotion styles and an unseen walk-then-crawl combination.
- Sparse-reward transfer: For the sparse-reward obstacle course, naive PPO cannot learn, whereas MLSH explores sequences of sub-policies that more easily lead to reward.The obstacle course gives reward only for reaching the goal, with walls and a chasing enemy along the route.
7 DISCUSSION
MLSH metalearns hierarchical policies end to end by sharing sub-policies across tasks while training task-specific master policies. Its framework is method-agnostic and can extend beyond shared motor primitives, though several approximations remain for future work.
- MLSH represents shared information as sub-policies and trains hierarchical models end to end over distributions of environments.The framework achieves significant learning speedups without optimizing the true objective and discovers diverse sub-policies without hand engineering.
- Hard one-hot communication removes gradient transfer between master and sub-policies, making MLSH independent of the reinforcement-learning method used.The experiments use policy gradients, but the authors state that evolution or Q-learning could train the master or sub-policies.
- The training framework jointly optimizes two parameter sets and can share observational networks across tasks with similar observations but different reward functions.In that example, the policies remain independent while the observational network is shared.
- Future work could explore alternative architectures and reduce MLSH approximations, including training φ over the full T-timestep horizon rather than one episode.The authors identify the intersection of hierarchical reinforcement learning and metalearning as an underexplored architecture space.