PaperlayerView exact revision on arXiv

Source-linked AI summary

Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?

Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, Sergey Levine

arXiv:1909.10618v2cs.LGcs.AIstat.ML

TL;DR

In Markovian settings, it is unclear why HRL should outperform shallow RL, despite its empirical successes and proposed benefits. This paper empirically evaluates those benefits across locomotion, navigation, and manipulation tasks, finding exploration explains most observed gains and that simpler hierarchy-inspired methods can match HRL performance.

  • Problem

    Despite HRL’s empirical success, it is unclear which proposed benefits explain its advantage over shallow RL in Markovian settings.

    Source: §1 · p. 1

  • Method

    The authors empirically isolate claimed hierarchy benefits across tasks involving locomotion, navigation, and manipulation, and develop simpler hierarchy-inspired exploration methods.

    Source: §1 · p. 2

  • Results

    Most observed benefits of hierarchy in the considered settings are attributable to improved exploration, and the proposed exploration methods achieve performance competitive with state-of-the-art HRL.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings suggest exploration research can draw on successful HRL techniques to develop temporally extended and semantically meaningful exploration strategies.

    Source: §1 · p. 2

    The conclusions are limited to the hierarchical designs and tasks evaluated; benefits in other settings, including multi-task transfer, were not assessed.

    Source: §6 · p. 9

Abstract

from arXiv · show

Hierarchical reinforcement learning has demonstrated significant success at solving difficult reinforcement learning (RL) tasks. Previous works have motivated the use of hierarchy by appealing to a number of intuitive benefits, including learning over temporally extended transitions, exploring over temporally extended periods, and training and exploring in a more semantically meaningful action space, among others. However, in fully observed, Markovian settings, it is not immediately clear why hierarchical RL should provide benefits over standard "shallow" RL architectures. In this work, we isolate and evaluate the claimed benefits of hierarchical RL on a suite of tasks encompassing locomotion, navigation, and manipulation. Surprisingly, we find that most of the observed benefits of hierarchy can be attributed to improved exploration, as opposed to easier policy learning or imposed hierarchical structures. Given this insight, we present exploration techniques inspired by hierarchy that achieve performance competitive with hierarchical RL while at the same time being much simpler to use and implement.

1 INTRODUCTION

Hierarchy has shown promise on difficult RL tasks, but its empirical advantage over shallow methods is not explained by intuitive arguments alone. The paper tests those claimed benefits and finds that improved exploration accounts for most of hierarchy’s observed benefit in the considered settings.

1 INTRODUCTION

  • Hierarchy is motivated by tasks whose natural structure can combine learned behaviors, such as walking and turning or grasping and handling.The introduction situates HRL as a way to address tasks built from sequences of behaviors.Source: §1 · p. 1
  • Prior work reports HRL solving quadrupedal manipulation tasks where non-hierarchical methods make negligible progress, and finding navigation policies with 3–5x fewer environment interactions.The examples cited are Nachum et al. (2018a;b; 2019) and Levy et al. (2017).Source: §1 · p. 1
  • Although HRL’s empirical success is clear, the reasons for it are difficult to explain, especially because Markovian systems need not use hierarchical policies to be optimal.The paper asks which proposed benefits explain HRL’s empirical advantage.Source: §1 · pp. 1–2
  • Most observed benefits of hierarchy in the considered tasks are attributable to improved exploration, and simpler hierarchy-inspired exploration methods achieve performance competitive with state-of-the-art HRL.The authors also argue that current methods realize only a subset of hierarchy’s claimed benefits.Source: §1 · p. 2

2 RELATED WORK

The paper focuses on options and goal-conditioned hierarchies, evaluating their practical benefits empirically. It positions this analysis against theory that depends on near-optimal options and prior empirical work that identified exploration as a benefit under restrictive conditions.

2 RELATED WORK

  • This study evaluates options and goal-conditioned hierarchies, selected for their recent success, while noting that other architectures remain for future investigation.The selected paradigms are the options framework and goal-conditioned hierarchies (Nachum et al., 2018b).Source: §2 · p. 2
  • Tabular sample-complexity bounds motivate hierarchy through smaller high-level action spaces or shorter effective horizons, but continuous-state analysis assumes near-optimal options often unavailable in practice.The authors say it is unclear whether theoretical benefits explain empirical successes in complex environments.Source: §2 · p. 2
  • Jong et al. (2008) also identified exploration as a benefit of options, but realizing it required specially designed options and substantial task knowledge.They attributed most remaining benefits to experience replay, which alone matched the performance on a non-hierarchical agent.Source: §2 · p. 2
  • The authors frame their work as a recalibration of current HRL: it tests which claimed benefits explain practical performance differences between hierarchical and non-hierarchical methods.Figure 1 compares three HRL algorithms with shallow agents, with and without n-step returns, over four tasks and 10M training steps averaged across five seeds.Source: Figure 1 · p. 3, §2 · p. 3

3 HIERARCHICAL REINFORCEMENT LEARNING

The evaluated HRL methods use a higher-level policy to direct lower-level behavior, with options selecting discrete policies and goal-conditioned hierarchies selecting goals. Training distinguishes single-step lower-level learning from temporally extended high-level transitions.

3 HIERARCHICAL REINFORCEMENT LEARNING

  • In the options framework, each discrete high-level action activates one of m lower-level policies for c steps; goal-conditioned hierarchies instead issue a continuous-valued goal to one lower-level policy.The high-level policy chooses new actions every c timesteps.Source: §3 · p. 3
  • Options train separate lower-level Q-functions from single-step transitions, while goal-conditioned methods train a goal-conditioned lower-level Q-function and policy using single-step transitions.For goal-conditioned learning, the intrinsic reward is the negative L2 distance to the current goal.Source: §3 · p. 3
  • Higher-level policies are trained on c-step transitions using the summed environment rewards, with discrete options chosen by maximizing Q and continuous goals selected to maximize Q.The analysis restricts temporal abstractions to a fixed c; evaluating variable-length abstractions is left for future work.Source: §3 · pp. 3–4
  • Multi-step rewards alone can achieve almost all benefits associated with hierarchical training when exploration benefits are controlled, despite using atomic actions.The paper distinguishes this from HRL, which temporally extends both rewards and actions in its Q-value regression.Source: §3 · p. 4
  • The study evaluates four simulated-robot tasks spanning locomotion, navigation, and manipulation, comparing HRL methods with non-hierarchical methods that struggle to progress.For controlled analysis, goal coordinates are fixed as relative x-y positions, although goal-conditioned HRL can perform well without that prior knowledge.Source: §3 · p. 4

4 HYPOTHESES OF THE BENEFITS OF HIERARCHY

The paper organizes hierarchy’s proposed benefits by whether they arise from temporally extended or semantically meaningful actions, and whether they affect training or exploration. These yield four hypotheses about learning and exploration.

4 HYPOTHESES OF THE BENEFITS OF HIERARCHY

  • The four hypotheses divide into hierarchical training (H1, H3) and hierarchical exploration (H2, H4), based respectively on temporal extension and semantic meaning.The paper notes that a further modularity hypothesis appears in the Appendix.Source: §4 · pp. 4–5
  • H1 proposes that temporally extended high-level actions shorten effective episodes, propagate rewards faster, and improve learning.High-level actions span multiple environment steps.Source: §4 · p. 4
  • H2 proposes that temporally extended actions create correlated exploration across environment steps, allowing an HRL agent to explore more efficiently.The motivating example is a random walk whose distribution widens when noise is temporally correlated.Source: §4 · p. 4
  • H3 proposes that training on semantically meaningful abstract actions makes future values easier to learn than training on atomic actions.The example contrasts navigation-coordinate changes with robot-joint torques.Source: §4 · p. 4
  • H4 proposes that applying exploration strategies to semantic actions makes exploration more meaningful than applying them to atomic actions.The navigation example contrasts x-y coordinates with joint torques.Source: §4 · p. 4

5 EXPERIMENTS

The experiments isolate temporal abstraction, hierarchical training, and exploration across difficult tasks. They find that non-hierarchical agents can approach HRL performance with multi-step rewards and temporally extended exploration, while explicit hierarchies are not necessary in most tested cases.

5.2 EVALUATING THE BENEFITS OF HIERARCHICAL TRAINING (H1 AND H3)

  • A non-hierarchical shadow agent learning from HRL-collected experience achieves competitive performance in all environments except AntMaze when given an appropriate multi-step reward horizon.This suggests hierarchical action representations are not a core training advantage in these domains; the observed training benefit can be reproduced with multi-step rewards.Source: Figure 3 · p. 6, §5.2 · p. 7

5.1 EVALUATING THE BENEFITS OF TEMPORAL ABSTRACTION (H1 AND H2)

  • For ctrain, AntMaze and AntPush require ctrain > 1 for adequate performance, but larger horizons show no consistent advantage.For cexpl, higher values correlate with better performance in AntPush, AntBlock, and AntBlockMaze, but not AntMaze; cexpl = 1 can still make non-negligible progress over shallow policies.Source: Figure 2 · p. 5, §5.1 · p. 6

5.3 EVALUATING THE BENEFITS OF HIERARCHICAL EXPLORATION

  • Explore & Exploit and Switching Ensemble match HRL on most tasks; the exceptions are Explore & Exploit on AntBlockMaze and Switching Ensemble on AntPush.Using cswitch > 1 is often significantly better than switching every step.Source: Figure 4 · p. 7, §5.3 · p. 8
  • Explore & Exploit uses goal-reaching alongside reward maximization, while Switching Ensemble randomly selects among five reward-maximizing agents at intervals of cswitch.Both methods use multi-step environment rewards with crew = 3.Source: §5.3 · p. 8
  • The results suggest HRL’s success on these tasks is largely due to better exploration, and non-hierarchical agents can achieve similar performance with multi-step rewards and temporally extended exploration.The exploration can be based on goal-reaching or randomized value functions.Source: §5.3 · p. 8
  • Separate networks for policy levels benefit HRL, and separate networks are crucial to the effectiveness of both proposed exploration methods.These modularity results are presented in the Appendix.Source: §5.3 · p. 8

6 DISCUSSION AND CONCLUSION

The empirical analysis attributes hierarchy’s main benefit to exploration, while finding limited benefits from abstract actions and temporally extended training. The authors also identify boundaries on how broadly these findings apply.

  • Semantically meaningful abstract actions have negligible effect on HRL success, while temporally extended training matters only insofar as it enables multi-step rewards.Source: §6 · p. 8
  • Temporally extended goal-reaching and agent-switching let nonhierarchical agents solve tasks that only hierarchical agents solved in these experiments, pointing to exploration as hierarchy’s main benefit.The authors conclude that hierarchical agents’ empirical effectiveness reflects the improved exploration they can attain.Source: §6 · pp. 8–9
  • The authors call for further work on semantic and temporally extended training, and on additional nonhierarchical exploration strategies inspired by hierarchy.Source: §6 · p. 9
  • The conclusions are limited to a specific set of hierarchical designs, tasks, and environments; other settings, including multi-task transfer, were not evaluated.The authors also identify more complex or sparse-reward settings as candidates for complementary exploration mechanisms.Source: §6 · p. 9

A TRAINING DETAILS

The diagram depicts a two-level HRL agent: a higher-level policy periodically selects an action, and a lower-level policy generates atomic actions until control returns.

  • Every c steps, the higher-level policy πhi selects a high-level action gt.Source: Figure 6 · p. 12
  • In the options framework, gt identifies one of m options; in goal-conditioned HRL, gt is a goal-state.Source: Figure 6 · p. 12
  • The lower-level policy πlo produces atomic actions for c steps, after which control returns to the higher-level policy.Source: Figure 6 · p. 12

B EVALUATING THE BENEFITS OF MODULARITY

The modularity evaluation compares separate policy networks with combined multi-head networks and finds that separation consistently performs better.

C EXPERIMENT DETAILS

The experiments use HIRO-based implementations and describe adjustments for low-level training, baseline methods, and exploration alternatives.

  • HIRO is implemented from its open-source code with default hyperparameters, and nonhierarchical agents use TD3 with matching network and training hyperparameters unless stated otherwise.Source: C EXPERIMENT DETAILS · p. 13
  • To control for goal-change frequency in low-level training ablations, the next goal is replaced with the current goal; this change has negligible effect on default HRL performance.Source: C EXPERIMENT DETAILS · p. 13
  • HAC adds hindsight experience replay to HIRO’s low-level policy, while Option-Critic uses five reward-trained lower-level policies and a discrete double-DQN high-level agent.The Option-Critic policies use n-step returns with n = 3, and the high-level agent uses ϵ-greedy exploration with ϵ = 0.5.Source: C EXPERIMENT DETAILS · p. 13
  • Explore & Exploit and Switching Ensemble use three-step environment rewards and collect two environment steps per training step; each variant receives five million training steps.That budget corresponds to 10M environment steps, matching HRL.Source: C EXPERIMENT DETAILS · p. 13
Loading 1909.10618v2…