Source-linked AI summary
Discovery of Options via Meta-Learned Subgoals
Vivek Veeriah, Tom Zahavy, Matteo Hessel, Zhongwen Xu, Junhyuk Oh, Iurii Kemaev, Hado van Hasselt, David Silver, Satinder Singh
TL;DR
Discovering options from experience that remain useful across tasks is an open problem in reinforcement learning. MODAC uses meta-gradients to learn task-independent option subgoals within a hierarchical agent. In gridworld and DeepMind Lab analyses, its options are used during training and support transfer to new tasks.
Problem
Automated discovery of temporal abstractions from experience, without extensive domain-specific knowledge, remains an open problem for reducing sample complexity across task distributions.
Source: §1 · p. 1Method
MODAC trains a task-conditioned manager to select options or primitive actions and uses meta-gradients to learn task-independent option rewards and terminations.
Source: §1 · p. 1, §2 · p. 2, §2.1 · p. 2Results
In gridworld and DeepMind Lab analyses, MODAC discovers meaningful options that are used during training and supports faster learning on new tasks than hierarchical baselines.
Source: §1 · p. 2, §3.1 · p. 5, §5 · p. 8Takeaways & Limitations
The discovered options capture diverse properties of training-task behavior, and DeepMind Lab results indicate MODAC is scalable for option discovery in challenging RL domains.
Source: §5 · p. 8The algorithm is instantiated for an actor-critic architecture, while adaptation to other RL updates is left for future work.
Source: §2.2 · p. 3
Abstract
from arXiv · showhide
Temporal abstractions in the form of options have been shown to help reinforcement learning (RL) agents learn faster. However, despite prior work on this topic, the problem of discovering options through interaction with an environment remains a challenge. In this paper, we introduce a novel meta-gradient approach for discovering useful options in multi-task RL environments. Our approach is based on a manager-worker decomposition of the RL agent, in which a manager maximises rewards from the environment by learning a task-dependent policy over both a set of task-independent discovered-options and primitive actions. The option-reward and termination functions that define a subgoal for each option are parameterised as neural networks and trained via meta-gradients to maximise their usefulness. Empirical analysis on gridworld and DeepMind Lab tasks show that: (1) our approach can discover meaningful and diverse temporally-extended options in multi-task RL domains, (2) the discovered options are frequently used by the agent while learning to solve the training tasks, and (3) that the discovered options help a randomly initialised manager learn faster in completely new tasks.
1. Introduction
Options can support faster learning through temporally extended behavior and reuse across tasks, but designing abstractions that remain useful across tasks is difficult. The paper proposes meta-gradient discovery of task-independent options and evaluates their use and transfer in multi-task settings.
1. Introduction
- Options can support faster learning through longer value-function backups in model-based RL and intentional multi-step behavior that can improve exploration in model-free RL.These benefits make temporally extended actions useful across more than one learning setting.Source: §1 · p. 1
- Reusing temporal abstractions across a task distribution can reduce sample complexity, while abstractions unhelpful to downstream tasks can make exploration harder.This makes hand-designing abstractions that work broadly a challenge and motivates automated discovery from experience.Source: §1 · p. 1
- The paper proposes a hierarchical agent and meta-gradient algorithm that discovers options reusable across tasks by learning their subgoal rewards and terminations.The option-reward and termination parameters are evaluated by their utility to the hierarchical agent across training tasks.Source: §1 · pp. 1–2
- Gridworld and DeepMind Lab analyses examine option meaning, frequency of use during training, and transfer to test tasks against MLSH and Option-Critic.The reported analysis includes qualitative option visualizations, option-use measurements, and transfer comparisons.Source: §1 · p. 2
2. MODAC: Meta-gradients for Option Discovery using Actor-Critic
MODAC discovers task-independent options by evaluating how changes to their subgoals affect a hierarchical agent’s learning across tasks. Its manager can choose options or primitive actions, while meta-gradients update option rewards and terminations using task samples.
2. MODAC: Meta-gradients for Option Discovery using Actor-Critic
- MODAC uses meta-gradients to evaluate how option changes affect future performance, while conventional RL gradients adapt the option-based behavior.The paper names the method Meta-gradients for Option Discovery using Actor-Critic.Source: §2 · p. 2
2.1. Agent Architecture
- The task-conditioned manager selects among option-policies and primitive actions; each option has a policy, a reward defining its subgoal, and a termination function.Only the manager receives the task goal, while the option networks do not; the option-reward and termination networks define an option’s subgoal.Source: §2.1 · pp. 2–3
- Allowing the manager to choose primitive actions as well as options lets it complete behavior that options alone may not provide, such as moving from a doorway to a room’s center.The architecture also permits the manager to select primitive actions when needed to solve a task optimally.Source: §2.1 · p. 3
2.2. MODAC’s Discovery Algorithm
- The inner loop trains the manager and option-policies on task rewards and option-rewards; the outer loop evaluates updated policies on new transitions and meta-updates option rewards and terminations.The outer-loop update back-propagates through the inner-loop updates, using transitions from another sample of tasks.Source: §2.2 · pp. 3–4
- The manager’s update includes a switching cost on option termination, which encourages selecting temporally extended options.The algorithm is instantiated for an actor-critic architecture; adapting it to other RL updates is left to future work.Source: §2.2 · pp. 3–4
2.3. Differences Between MODAC and Prior Option-Discovery Methods
- Unlike prior multi-task approaches that train options on the current task reward, MODAC gives each option a task-independent objective intended to make options reusable across tasks.The paper characterizes the resulting options as general-purpose and disentangled, with potential to speed learning on new tasks.Source: §2.3 · p. 4
3. Empirical Results
Across gridworld and DeepMind Lab, MODAC discovered options that were used during learning and transferred to unseen tasks. The experiments also report meaningful, diverse behaviours, with transfer results varying by task and training setup.
3.1. An Illustrative Gridworld
- In gridworld transfer, options were selected 56.11% of the time but controlled behavior more than 85% of the time, with options lasting 5.46 steps on average.Source: §3.1 · p. 5
- Gridworld options (a–c) led to rooms containing training goals, while option (d) appeared redundant, plausibly reflecting the concentration of training goals in three rooms.Source: §3.1 · p. 5, Figure 3 · p. 6
- On held-out gridworld tasks, a randomly initialized manager learned substantially faster with MODAC options than with Flat, MLSH, or Option-Critic.Source: §3.1 · p. 5, Figure 4 · p. 6
3.2. Scaling Up to DeepMind Lab
- In DeepMind Lab mazes, options supported exploration and were chained more often in hard test tasks, while primitive actions handled navigation near the goal.Sampled option policies also showed diverse trajectories from the same initial state, including in test mazes.Source: §3.2 · pp. 6–7
- On DeepMind Lab, MODAC options outperformed MLSH and Option-Critic in all four domains, learned faster than Flat, and reached higher asymptotic performance in three.On the fourth task, learning was substantially faster.Source: Figure 7 · p. 7, §3.2 · p. 8
- In DeepMind Lab transfer, options were selected 63.76% of the time and, given their roughly 12-step duration, controlled behavior more than 95% of the time.Source: §3.2 · pp. 7–8
- With 5M and 10M discovery samples, MODAC learned faster than Flat on one DeepMind Lab task set.Source: Figure 8 · p. 8, §3.2 · p. 8
4. Related Work
Temporal abstractions can speed learning, but hand-design is challenging across task distributions, motivating option discovery from experience. MODAC differs from prior approaches by learning task-independent options through subgoals and learned temporal scales.
4. Related Work
- Prior temporal abstractions can speed learning in single-task and multi-task settings, but typically require hand-design, which is challenging across task distributions.The paper instead targets discovery from experience without human supervision or domain knowledge.Source: §4 · p. 8
- Prior abstraction discovery is mostly unsupervised or single-task, using methods such as bottleneck detection, state clustering, proto-value functions, and diversity maximisation.MODAC is motivated by multi-task RL and reusable, task-independent options.Source: §4 · p. 8
- MODAC discovers options through option-specific rewards and terminations and learns each option’s temporal scale, unlike approaches such as MLSH that fix it.MLSH directly optimises task rewards for its option policies; most prior approaches fix temporal scale.Source: §4 · p. 8
5. Conclusions
The paper introduces MODAC, a hierarchical agent and meta-gradient algorithm for discovering options in multi-task RL. Gridworld and DeepMind Lab visualisations show diverse behaviours, and DeepMind Lab results support scalability to challenging RL domains.
5. Conclusions
- MODAC combines a hierarchical agent with a meta-gradient algorithm to discover options in multi-task RL.The options capture diverse and important properties of behaviours required by the training-task distribution.Source: §5 · p. 8
- DeepMind Lab results show that MODAC scales to option discovery in challenging RL domains.The paper identifies applying meta-gradients to discover options for model-based planning as future work.Source: §5 · p. 8
- The paper defines unsupervised approaches as those that do not require extrinsic reward to drive discovery.Source: §5 · p. 8
A.1. Additional Qualitative Experiments on Gridworld
Additional gridworld visualisations vary the number of discovered options from the four used in the main results. With two options, behaviours lead to different rooms; with eight, several options target training-task rooms while others are similar or lack clear subgoals.
A.1. Additional Qualitative Experiments on Gridworld
- The additional visualisations show options learned with K = 2 and K = 8, alongside the K = 4 used for the main gridworld results.K denotes the number of options and is a hyperparameter.Source: §A.1 · p. 12
- With K = 2, option (a) leads to the upper-left room, while option (b) leads to either the upper-right or lower-right room depending on the start state.Source: Figure 9 · p. 12
- With K = 8, options (b, e, f, g, h) lead to rooms where training tasks are concentrated, while (a, c) move left and (d) lacks a clear subgoal.The authors hypothesise that the first group was picked more often during training, leading to better-defined subgoals.Source: Figure 10 · p. 12
A.2. Additional Qualitative Visualisations from DeepMind Lab
In DeepMind Lab, sampled option policies produce diverse, structured trajectories on both training and corresponding test tasks. A trained manager also combines discovered options with primitive actions to reach training-task goals.
A.2. Additional Qualitative Visualisations from DeepMind Lab
- Sampled option policies produce diverse, structured exploratory behaviours on both a training task and its corresponding test task.The goal is top-left and outside the agent’s initial line of sight in both visualisations.Source: §A.2 · p. 12, Figure 11 · p. 13, Figure 12 · p. 13
- A trained MODAC manager relies on options in both training and test tasks, supporting the authors’ claim that the options are reusable and transferable.Source: §A.2 · p. 13
- In five sampled training-task trajectories, the manager reaches the goal using a mixture of discovered options and primitive actions.The figure distinguishes options by arbitrary colours and primitive actions in blue.Source: Figure 13 · p. 13
A.3. Additional Experiments on Atari
The Atari experiments test whether options discovered from pixel-control tasks within each game transfer to the game-score task. MODAC’s options speed learning across all four games, while comparisons of termination handling vary by game.
Experimental Setup
- Atari games have mutually inconsistent dynamics, so the experiments generated multiple training tasks separately within each game rather than discovering options across games.Pixel-control task definitions ignored episode terminations to avoid leaking test-task information.Source: §A.3 · pp. 13–14
Qualitative Results
- The test-task trajectories show the manager reaching the goal using mixtures of discovered options and primitive actions.Figure 14 presents five distinct trajectory samples.Source: Figure 14 · p. 14
Quantitative Analysis
- MODAC with discovered options learned game-score rewards faster than a flat agent in all four games and outperformed MLSH and Option-Critic in transfer learning.Options were chosen 60.79% of the time; the options were discovered from pixel-control tasks on the corresponding game.Source: §A.3 · p. 14, Figure 15 · p. 14
Effect of Episode Terminations
- Ignoring episode terminations yielded slightly faster learning in MsPacman and Boxing, stable asymptotic performance in Hero, and no visible difference in Riverraid.The comparison is between MODAC options discovered with and without episode terminations.Source: §A.3 · p. 15, Figure 16 · p. 15
A.4. Objective and Update Equations for Discovering Option-Rewards and Terminations Using Meta-gradients
The objective is to discover option rewards and terminations that improve the hierarchical agent’s performance. Meta-gradients update those functions through the option policies they induce.
Objective
- MODAC optimizes option-reward and termination parameters for hierarchical-agent performance through the option policies those functions induce.The manager’s policy maximizes extrinsic rewards, while option policies maximize their corresponding option rewards with discounts based on termination.Source: §A.4 · p. 15
- The objective uses the manager’s n-step return, including discounted rewards, a switching cost, and a bootstrapped manager value.The performance expectation is over option-policy parameters, manager-policy parameters, and training tasks.Source: §A.4 · p. 15
Update Equations
- Score-function estimation and the chain rule yield stochastic updates computed from environment samples, with gradients passing through inner-loop option-policy parameters to option rewards and terminations.Multiple inner-loop updates are performed per outer-loop update.Source: §A.4 · p. 15
A.5. Neural Network Architecture
The MODAC agent’s manager, option-policy, option-reward, and option-termination networks use the same torso architecture. The appendix specifies domain-specific torso choices, activations, and task-information inputs.
Network Architecture
- The manager, option-policy, option-reward, and option-termination networks use an identical torso, and the hierarchical baselines use identical architecture choices.Source: §A.5 · p. 16
- Gridworld uses a two-layer CNN with 32 filters per layer and a 256-unit fully connected layer, while DeepMind Lab adds a 256-unit LSTM after a Deep ResNet and Atari uses a Deep ResNet.Source: §A.5 · p. 16
- Intermediate layers use ReLU, option-reward outputs use arctan, and option-termination outputs use sigmoid; task information enters through an image channel in gridworld and an embedding in DeepMind Lab and Atari.For DeepMind Lab and Atari, the 128-dimensional task embedding is concatenated with the manager’s feed-forward torso output.Source: §A.5 · p. 16
A.6. Preprocessing
The appendix records preprocessing, tuning, DeepMind Lab task setup, baselines, resource use, and computing infrastructure. These details specify the experimental conditions and implementation context.
A.6. Preprocessing
- DeepMind Lab and Atari inputs stack four frames, repeat each action for four time steps, grayscale and downsample to 84 × 84, max-pool the last two frames, and clip rewards to [-1, 1].Source: §A.6 · p. 16
A.7. Hyperparameters
- Hyperparameters were tuned separately per agent, then one set was used across DeepMind Lab and Atari; tuning covered entropy weights, learning rates, and switching costs, using RMSProp.For hierarchical agents, manager and option-policy entropy weights and learning rates were tied; MODAC used one learning rate for option-reward and termination parameters.Source: §A.7 · p. 16
A.8. Experimental Setup for DeepMind Lab
- DeepMind Lab evaluation used four navigation task sets, each with a simpler training task and a test task, with layouts and specified starts, goals, object counts, or locations procedurally generated.Source: §A.8 · pp. 16–17
A.9. Baselines
- MODAC was compared with two hierarchical option-learning baselines, MLSH and a multi-task Option-Critic extension, and a flat actor-critic agent.MLSH uses a fixed worker timescale; the Option-Critic baseline uses task-conditional manager policies and task-independent worker policies and terminations.Source: §A.9 · p. 17
A.10. Resource Usage
- Hierarchical agents take significantly longer than flat actor-critic during training, but their test-phase running times are similar because they reuse discovered options.Source: §A.10 · p. 17
A.11. Computing Infrastructure
- The distributed JAX infrastructure uses parallel actors and learners on machines with 56 CPU cores and 8 TPU cores, and reports a meta-gradient norm clip of 1.Table 1 is identified as the source of MODAC’s detailed hyperparameters.Source: §A.11 · pp. 17–19, Table 1 · p. 19