PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Hierarchical Planning from Pixels

Danijar Hafner, Kuang-Huei Lee, Ian Fischer, Pieter Abbeel

arXiv:2206.04114v1cs.AIcs.LGcs.ROstat.ML

TL;DR

Long-horizon reinforcement learning remains difficult, and existing hierarchical approaches often rely on manually specified subtasks or goal spaces. Director plans over discrete latent goals in a learned world model, and the paper reports success on sparse-reward benchmarks and across varied environments.

  • Problem

    Long tasks can require millions of decisions, but current agents are limited to shorter horizons and hierarchical methods often require manually specified subtasks or semantic goal spaces.

    Source: §1 · pp. 1–2

  • Method

    Director learns a world model, uses a manager to select discrete latent goals for task and exploration rewards, and trains a worker to reach those goals.

    Source: §1 · p. 2, §2 · p. 3, §2.2 · p. 4

  • Results

    Director solves two sparse-reward benchmark suites and learns across varied domains, including Atari, Control Suite, DMLab, and Crafter, without task reward for the worker.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Decoding latent goals into images makes Director’s decisions more interpretable and reveals diverse strategies for breaking tasks into subgoals.

    Source: §5 · p. 9

    The fixed goal-switch interval may limit adaptation on tasks requiring precise timing.

    Source: §5 · p. 9

Abstract

from arXiv · show

Intelligent agents need to select long sequences of actions to solve complex tasks. While humans easily break down tasks into subgoals and reach them through millions of muscle commands, current artificial intelligence is limited to tasks with horizons of a few hundred decisions, despite large compute budgets. Research on hierarchical reinforcement learning aims to overcome this limitation but has proven to be challenging, current methods rely on manually specified goal spaces or subtasks, and no general solution exists. We introduce Director, a practical method for learning hierarchical behaviors directly from pixels by planning inside the latent space of a learned world model. The high-level policy maximizes task and exploration rewards by selecting latent goals and the low-level policy learns to achieve the goals. Despite operating in latent space, the decisions are interpretable because the world model can decode goals into images for visualization. Director outperforms exploration methods on tasks with sparse rewards, including 3D maze traversal with a quadruped robot from an egocentric camera and proprioception, without access to the global position or top-down view that was used by prior work. Director also learns successful behaviors across a wide range of environments, including visual control, Atari games, and DMLab levels.

1 Introduction

Long-horizon tasks strain agents that reason at primitive-action timescales, while manually specified subtasks and reward terms are costly. Director learns a hierarchy in a world model’s latent space and is reported to solve sparse-reward benchmarks and learn across varied domains.

1 Introduction

  • Long tasks can require millions of time steps, while manually specifying subtasks or dense rewards is expensive and prone to errors.Current reinforcement learning methods often reason at the clock rate of primitive actions.Source: §1 · pp. 1–2
  • Most hierarchical reinforcement learning approaches depend on manually specified subtasks or semantic goal spaces, and learning hierarchies from sparse rewards has had limited success.Some sparse-reward approaches also give task reward to the low-level controller.Source: §1 · p. 2
  • Director learns hierarchical behavior by planning in a learned world model’s latent space, with compact discrete goals and temporally extended exploration.The high-level policy selects goals while the low-level policy learns to reach them.Source: §1 · p. 2, Figure 2 · p. 2
  • Director solves two sparse-reward benchmarks and learns across Atari, Control Suite, DMLab, and Crafter.Its decoded latent goals provide visualizations of the agent’s internal subgoals; in Ant Maze M, the sequence includes a wall, the reward object, and another wall.Source: Figure 1 · p. 1, §1 · p. 2

2 Director

Director combines a learned world model with a discrete-goal manager and a goal-conditioned worker. The world model supplies compact representations for planning; the manager selects goals and exploration rewards, while the worker learns to reach decoded goals.

2 Director

  • Director jointly optimizes a world model, goal autoencoder, manager, and worker, training representations from replay and policies from imagined rollouts.The manager selects goals to maximize task and exploration rewards; the worker chooses primitive actions to reach them.Source: §2 · p. 3

2.1 World Model

  • The world model integrates observations and actions into latent states, predicts future states, reconstructs observations, and predicts rewards.Its variational objective encourages informative, predictable latent states and accurate dynamics.Source: §2.1 · p. 3

2.2 Goal Autoencoder

  • To avoid giving the manager a 1024-dimensional continuous goal space, Director compresses model states into factorized discrete codes.The autoencoder learns informative reconstructible codes while regularizing their distribution toward a uniform prior.Source: §2.2 · pp. 3–4

2.3 Manager Policy

  • The manager selects discrete-code goals every K = 8 steps and maximizes discounted task and exploration rewards.High reconstruction error under the replay-trained goal autoencoder serves as a signal for novel states.Source: §2.3 · p. 4

2.4 Worker Policy

  • The worker receives decoded goals and earns goal-similarity rewards, with max-cosine similarity designed to match both goal angle and magnitude.It receives only goal rewards in the main design; an ablation adds task reward and further improves performance.Source: §2.4 · pp. 4–5
  • The Ant Maze setting uses egocentric camera input and sparse reward, unlike prior approaches using global coordinates or top-down views and dense rewards.The worker must reach goals selected by the manager in this more challenging setting.Source: Figure 3 · p. 5

3 Experiments

Experiments test Director on sparse visual tasks, standard RL domains, and decoded subgoals, using shared hyperparameters across tasks and domains.

3 Experiments

  • Evaluation spans Atari, Control Suite, Crafter, and DMLab, with fixed hyperparameters across tasks and domains.The standard worker learns from goal rewards; a second variant also uses task returns.Source: §3 · p. 5, §3.2 · p. 7

3.1 Sparse Reward Benchmarks

  • Director reliably reaches the goal zone in all four maze difficulties, while no baseline succeeds in the larger mazes.The larger mazes take longer to master.Source: Figure 4 · p. 6, §3.1 · p. 6
  • Director discovers the correct pad-activation sequence in all four environments, while Plan2Explore struggles as the number of pads increases.Plan2Explore earns some reward with four pads, struggles with five, and fails with six.Source: Figure 5 · p. 7, §3.1 · p. 7

3.2 Standard Benchmarks

  • Director learns across many standard environments, and giving task reward to the worker closes the gap to DreamerV2.The goal-only worker also succeeds across a wide range of tasks.Source: §3.2 · p. 7

3.3 Goal Interpretations

  • In Ant Maze, early training uses intermediate goals and later fewer; in DMLab, reward-triggered teleport goals need no fine-grained subgoals.The DMLab worker can reach the teleport goal without joint-level control.Source: §3.3 · p. 8
  • Crafter goals track resources and shelter, while Walker uses pose and floor cues and leaves leg movements to the worker.The same hyperparameters yield semantically meaningful goals across domains without domain-specific knowledge.Source: §3.3 · p. 9

4 Related Work

Prior hierarchical reinforcement learning has used several approaches to learning skills and goals, but few methods learn hierarchies directly from pixels without domain-specific knowledge. Director is positioned as addressing that gap through representation learning and temporally abstract exploration.

Approaches to Hierarchical Reinforcement Learning

  • Prior hierarchical methods learn low-level policies through pretraining, discover latent skills with mutual-information objectives, or train goal-conditioned policies.Examples include pretraining (Heess et al., 2016), skill discovery (Gregor et al., 2016), and goal-conditioned policies (Andrychowicz et al., 2017).Source: §4 · p. 9
  • Few methods learn hierarchical behavior directly from pixels without domain-specific knowledge such as global coordinates, pretraining tasks, or diverse experience datasets.HSD-3 (Gehring et al., 2021) showed transfer benefits in low-dimensional control; HAC (Levy et al., 2017) required semantic goal spaces.Source: §4 · p. 9
  • FuN (Vezhnevets et al., 2017) did not show clear benefits over an LSTM baseline; Director uses representation learning and temporally abstract exploration.The paper reports benefits over flat policies on sparse-reward tasks and learning without task reward for the worker across many domains.Source: §4 · p. 9

5 Discussion

Director learns hierarchical behavior from pixels by planning in a learned world model's latent space and compressing goals into discrete codes. The paper highlights interpretability alongside design choices that may restrict performance.

Discussion

  • Director plans in a learned world model's latent space and compresses goals into discrete codes to simplify the manager's control problem.Decoded latent goals also reveal diverse strategies for dividing tasks into subgoals.Source: §5 · p. 9

Limitations and future work

  • The manager treats goals as black-box actions, changes them at fixed intervals, and selects point goals; adaptive triggers and distributional goals are proposed alternatives.The authors suggest these changes could improve performance on tasks requiring precise timing or targets limited to currently relevant state parts.Source: §5 · p. 9
  • More than two hierarchy levels currently require exponentially longer batches; learning temporally abstract dynamics is proposed as a way to address this.Source: §5 · p. 9

Societal Impact

The paper notes that practical real-world hierarchical reinforcement learning still needs further research, while identifying possible longer-term benefits and unresolved safety concerns.

Societal Impact

  • Real-world applications of hierarchical reinforcement learning still require further research.The paper says the approach could eventually help automate more tasks by reducing the need to specify intermediate rewards.Source: Societal Impact · p. 13
  • Learning from sparse rewards may reduce human effort and the potential for reward hacking, but does not eliminate that possibility.The paper says reward hacking will need attention as systems become more powerful and are deployed around people.Source: Societal Impact · p. 13
  • Director does not directly address safety issues, though decoded goal images may support failure audits and, with a monitor, interventions.The paper frames both uses as possibilities, not guarantees.Source: Societal Impact · p. 13

Checklist

The checklist indicates that the authors addressed limitations and potential negative societal impacts, and report plans and references for experimental reproducibility.

Checklist

  • The checklist marks discussion of the work's limitations and potential negative societal impacts as included.It directs readers to Section 5 for limitations.Source: Checklist · p. 13
  • The authors state that they will provide code and training curves upon publication and direct readers to Table F.1 for training details.For error bars, they refer to Figures 4, 5, A.1, J.1, and K.1.Source: Checklist · p. 13
  • Figure A.1 reports successful learning across Atari, Crafter, Control Suite, and DMLab, with the worker receiving no task reward.When task reward is also provided to the worker, performance matches Dreamer.Source: Figure A.1 · p. 14

B Ablation: Goal Autoencoder

The ablation compares Director's Goal Autoencoder with direct selection of continuous world-model goals, which works in some environments but struggles with sparser rewards.

B Ablation: Goal Autoencoder

  • The ablation compares Director with a hierarchical agent whose manager directly selects 1024-dimensional continuous world-model goals.Director uses a Goal Autoencoder to provide a contrasting goal-selection approach.Source: Figure B.1 · p. 14
  • Direct continuous-goal selection works surprisingly well in some environments.The passage does not specify which environments succeed.Source: Figure B.1 · p. 14
  • The direct-goal approach fails in environments with sparser rewards, likely because the manager's control problem becomes too challenging.The explanation is presented as likely, not established as a cause.Source: Figure B.1 · p. 14

C Ablation: Goal Reward

Director is robust to several inner-product goal rewards, while the L2 reward performs substantially worse; the authors propose a state-dimension explanation.

C Ablation: Goal Reward

  • The unnormalized inner-product reward can vary in scale with goal magnitude and encourages the worker to overshoot goals in magnitude.The normalized inner-product version normalizes reward scale across goals but still encourages overshooting.Source: Figure C.1 · p. 15
  • Director is robust to the precise goal reward, with three inner-product-based rewards performing well across tasks.The comparison includes cosine-max, Inner, and InnerNormed rewards.Source: Figure C.1 · p. 15
  • The L2 reward performs substantially worse; the authors hypothesize inner products let goals ignore state dimensions by setting them to zero.For L2, zeroing dimensions still requires the worker to care about them, according to the proposed explanation.Source: Figure C.1 · p. 15

D Ablation: Exploration

Exploration ablations show that manager-level exploration is required for successful Ant Maze learning, while low-level exploration hurts; Director also represents states as discrete tokens for manager selection.

D Ablation: Exploration

  • Manager exploration is required for successful Ant Maze learning, but low-level exploration makes leg movements too chaotic; neither bonus is needed on short-horizon benchmarks.The manager bonus produces temporally abstract exploration.Source: Figure D.1 · p. 15

E Pseudocode

  • The pseudocode initializes a replay buffer and neural networks, then repeats its training loop until convergence.Source: E Pseudocode · p. 16

F Hyperparameters

  • Director uses the same hyperparameters across experiments, with world-model training and policy optimization settings unchanged from DreamerV2.Source: Table F.1 · p. 16

G Vector of Categoricals

  • Director converts state representations into manager-selectable discrete tokens using L categorical dimensions with C classes, sampled as one-hot codes and flattened to L active features.Gradients through sampling use the gradient with respect to the one-hot matrix as the gradient with respect to categorical probabilities.Source: Figure G.1 · p. 17

H Policy Optimization

Director optimizes its manager and worker using imagined world-model trajectories and λ-returns. Its task-agnostic worker solves tasks across visual control and Atari, with task rewards further improving results.

H Policy Optimization

  • Both policies use Dreamer actor-critic optimization on imagined latent trajectories, with λ-returns balancing farther-ahead rewards against the variance of long sampled returns.The actor uses policy gradients, a value baseline, and an entropy term; multiple reward signals use separate critics and normalized returns and baselines.Source: H Policy Optimization · p. 17

I Further Related Work

  • DSN (Tessler et al., 2017), MLSH (Frans et al., 2017), HeLMS (Rao et al., 2021), and MODAC (Veeriah et al., 2021) require manually specified diverse training tasks.The passage says it is unclear whether these approaches generalize beyond their training tasks; Director instead learns a goal-conditioned worker using dense feature-space similarity.Source: I Further Related Work · p. 18
  • Mutual-information skill-discovery approaches include future-state prediction and learned skill-state similarity, while open challenges include skill diversity without mode dropping and precision for solving tasks.Source: I Further Related Work · p. 18
  • Prior goal-reaching methods variously rely on hindsight, privileged state or top-down inputs, manually designed goal spaces, or user-specified goals; Director learns world-model goals end-to-end from pixels without domain-specific knowledge.Source: I Further Related Work · p. 18

J Full Visual Control Suite

  • Across 20 visual-control tasks, Director solves a wide range without task reward to the worker; adding it reaches Dreamer’s state of the art and exceeds it on some tasks.Experiments used no action repeat and trained every 16 environment steps, yielding faster wall-clock time but lower sample efficiency than results reported by Hafner et al. (2019).Source: Figure J.1 · p. 19

K Full Atari Suite

  • Across 55 Atari games, Director solves a wide range of tasks without task reward to the worker; adding that reward reaches Dreamer and exceeds its human-normalized median score.Experiments used no action repeat and trained every 16 environment steps, yielding faster wall-clock time but lower sample efficiency than results reported by Hafner et al. (2019).Source: Figure K.1 · p. 20

Related papers

Loading 2206.04114v1…