PaperlayerView exact revision on arXiv

Source-linked AI summary

Data-Efficient Hierarchical Reinforcement Learning

Ofir Nachum, Shixiang Gu, Honglak Lee, Sergey Levine

arXiv:1805.08296v4cs.LGcs.AIstat.ML

TL;DR

Existing hierarchical reinforcement learning methods often require task-specific design and on-policy training, limiting their generality and sample efficiency. HIRO uses learned state goals, off-policy training, and an off-policy correction, and outperforms prior HRL methods while solving complex simulated-robot tasks with only a few million samples.

  • Problem

    Existing HRL methods often require task-specific design and on-policy training, while complex tasks demand multi-level reasoning beyond atomic behaviors.

  • Method

    HIRO trains lower-level policies toward raw state goals selected by higher-level policies and uses off-policy correction for concurrent hierarchical training.

  • Results

    HIRO outperforms prior HRL algorithms and solves complex locomotion and object-interaction tasks using only a few million experience samples.

  • Takeaways & Limitations

    The results support HIRO as a generally applicable and sample-efficient approach for difficult hierarchical control tasks.

  • Takeaways & Limitations

    The authors note that HRL performance and stability remain imperfect and require further improvement.

Abstract

from arXiv · show

Hierarchical reinforcement learning (HRL) is a promising approach to extend traditional reinforcement learning (RL) methods to solve more complex tasks. Yet, the majority of current HRL methods require careful task-specific design and on-policy training, making them difficult to apply in real-world scenarios. In this paper, we study how we can develop HRL algorithms that are general, in that they do not make onerous additional assumptions beyond standard RL algorithms, and efficient, in the sense that they can be used with modest numbers of interaction samples, making them suitable for real-world problems such as robotic control. For generality, we develop a scheme where lower-level controllers are supervised with goals that are learned and proposed automatically by the higher-level controllers. To address efficiency, we propose to use off-policy experience for both higher and lower-level training. This poses a considerable challenge, since changes to the lower-level behaviors change the action space for the higher-level policy, and we introduce an off-policy correction to remedy this challenge. This allows us to take advantage of recent advances in off-policy model-free RL to learn both higher- and lower-level policies using substantially fewer environment interactions than on-policy algorithms. We term the resulting HRL agent HIRO and find that it is generally applicable and highly sample-efficient. Our experiments show that HIRO can be used to learn highly complex behaviors for simulated robots, such as pushing objects and utilizing them to reach target locations, learning from only a few million samples, equivalent to a few days of real-time interaction. In comparisons with a number of prior HRL methods, we find that our approach substantially outperforms previous state-of-the-art techniques.

1 Introduction

HIRO addresses the limited complexity and sample efficiency of existing deep RL by combining general goal-based hierarchical control with off-policy training and a correction for changing lower-level behaviors. It learns complex navigation and object-interaction behaviors that existing non-HRL methods cannot solve and outperforms prior HRL methods.

  • Motivation: Deep RL has advanced continuous-control skills, but most learned behaviors remain atomic and rarely require complex multi-level reasoning.Examples include locomotion, dexterous manipulation, and simple robot-arm manipulation.
  • Motivation: HRL uses layered policies so higher levels plan over longer timescales while only the lowest level acts on the environment.This structure is intended to address difficult tasks requiring multiple levels of temporal and behavioral abstraction.
  • Method: HIRO trains lower-level controllers toward raw environment states proposed as goals by higher-level policies, avoiding manual or multi-task design.Using raw state observations instead of learned embeddings simplifies lower-level learning and provides substantial experimental benefits.
  • Method: The method uses off-policy training for efficiency and introduces an off-policy correction to address changes in the higher-level action space caused by evolving lower-level behaviors.Goal-proposing approaches otherwise rely on less efficient on-policy reinforcement learning, while off-policy instability is amplified when policies are trained jointly.
  • Results: HIRO learns successful policies in difficult environments requiring exploratory navigation and complex object interactions, tasks described as unsolvable by existing non-HRL methods.The paper also reports superiority over other published HRL methods.

2 Background

The paper adopts standard continuous-control reinforcement learning, where agents learn behavior policies from environment interactions and transition data. Its HRL method uses TD3, building on DDPG’s actor–critic framework and stochastic behavior policies for off-policy correction.

  • Continuous-control reinforcement learning: In continuous-control RL, an agent follows a behavior policy to map observed states to continuous actions, receives rewards, and transitions or terminates.The state and action spaces are continuous, with future rewards discounted by γ.
  • Temporal-difference learning: Temporal-difference learning updates policies from state-action-reward-next-state transition tuples collected through environment interaction.The method uses TD3, a continuous-control variant of DDPG.
  • DDPG actor–critic learning: DDPG jointly learns a deterministic policy and state-action Q-function, training the Q-function with Bellman-error minimization and the policy to maximize Q-values.The policy parameters and Q-function parameters are updated through gradient-based learning.
  • Stochastic behavior policies: Although DDPG’s learned policy is deterministic, its behavior policy adds Gaussian or Ornstein-Uhlenbeck noise when collecting experience.The resulting stochastic behavior policy is used for the HRL method’s off-policy correction.

3 General and Efficient Hierarchical Reinforcement Learning

HIRO is a two-layer hierarchical reinforcement learning framework that automatically uses state-based goals for lower-level control and off-policy correction for higher-level training. This design addresses the non-stationarity caused by changing lower-level behavior while retaining generality and sample-efficient off-policy learning.

  • Framework: HIRO uses a lower-level controller trained toward parameterized goals proposed by a higher-level policy over temporally extended periods.The higher-level policy samples goals every c steps, while a fixed transition function determines intermediate goals.
  • Goal Representation: The higher-level goal represents a desired relative state change, encouraging the lower-level controller to reach an observation close to st + gt.This generic representation avoids requiring task-specific goal subspaces; evaluations use positional observations for simulated ant locomotion.
  • Goal Representation: Directly using state observations as goals and state changes as higher-level actions distinguishes HIRO from prior parameterized-reward HRL approaches.The paper identifies this as a significant distinction from methods that must train goals or higher-level action spaces separately.
  • Off-Policy Correction: Changing lower-level behavior makes old higher-level experience inconsistent with the current controller, forcing prior goal-setting HRL methods toward on-policy training.This non-stationarity arises because identical goals can produce different transitions under different lower-level controllers.
  • Off-Policy Correction: HIRO corrects stale higher-level experience by re-labeling goals to maximize the likelihood that the current lower-level controller would have produced the observed action sequence.Intermediate re-labeled goals are computed using the fixed transition function h.
  • Off-Policy Correction: 10 candidates approximate the correction objective: eight Gaussian-sampled goals plus the original goal and the observed state difference st+c − st.The candidate goals are evaluated by lower-level action-sequence log probability, and the maximal candidate is selected.

4 Related Work

Prior HRL methods commonly construct sub-goals or auxiliary rewards using domain knowledge, diversity objectives, or learned representations. HIRO instead emphasizes off-policy learning and raw-state goals, addressing non-stationarity while improving sample efficiency relative to most HRL work.

  • Classical HRL focused on discrete state domains, using visitation and transition statistics to construct heuristic sub-goals for low-level policies.
  • Recent methods provide low-level policies with auxiliary rewards based on hand-crafted domain knowledge, mutual information, or diverse task pre-training.
  • HIRO differs from most HRL methods by using off-policy learning, which improves sample efficiency but creates higher-level non-stationarity as lower-level behavior changes.
  • Unlike FeUdal Networks, HIRO represents goals and computes rewards using raw states rather than a learned state representation, which under-performed in experiments.
  • Goal-conditioned value functions are actively studied outside HRL and may provide further improvements to hierarchical methods.

5 Experiments

Experiments evaluate HIRO on challenging locomotion-and-manipulation environments against prior HRL methods and through ablations of its design choices. HIRO performs well across tasks, learns rapidly, and substantially outperforms non-HRL and most competing variants, especially on harder tasks.

  • Overall results: HIRO performs well across difficult tasks and reaches good performance on complex navigation problems within a few million environment steps.Results average reward or success rate over 10 randomly seeded trials, with environment steps measured in millions.
  • Comparisons: Against FuN, SNN4HRL, and VIME, HIRO outperforms the modified FuN variants on the evaluated tasks except Ant Gather and remains ahead of SNN4HRL.VIME performs approximately the same as SNN4HRL on the benchmark’s 500-step task.
  • Comparisons: The option-critic variants yielded no reasonable performance, possibly because continuous control is difficult and their sub-policies rely solely on external rewards.This reward structure makes learning gait policies difficult.
  • Ablations: Lower-level re-labelling can provide an initial speed-up by allowing experience collected for one goal to train behavior for alternative goals.The technique increases data available to the lower-level policy trained with a parameterized reward.
  • Ablations: Off-policy correction significantly benefits Ant Push and Ant Fall, while pre-training the lower-level policy for 2M steps and freezing it addresses higher-level non-stationarity differently.Without correction, training can perform well initially but struggles on the harder tasks; the pre-training variant uses Gaussian-sampled goals.
  • Ablations: A single non-HRL policy makes almost no progress on the environments compared with HIRO.This result highlights the importance of the hierarchical structure for these tasks.

6 Conclusion

The paper presents a general two-layer hierarchical policy that uses learned goals to communicate between levels and supports concurrent off-policy training. Experiments show that it is highly sample-efficient, outperforms prior HRL algorithms, and solves complex tasks combining locomotion and rudimentary object interaction.

  • The method trains a two-layer hierarchical policy in which higher-level policies provide learned goals as instructions to lower-level policies.
  • Both policies can be trained concurrently off-policy, enabling highly sample-efficient hierarchical reinforcement learning.
  • Experiments show that the method outperforms prior HRL algorithms and solves exceedingly complex tasks combining locomotion with rudimentary object interaction.

A Discussion on Alternative Off-Policy Corrections for High-Level Actions … B.2.3 Fall

The paper compares alternative off-policy corrections for high-level actions and reports that maximum likelihood-based action relabeling performs best empirically. It also specifies the simulator settings and navigation-task configurations used to evaluate the method, including Maze, Push, and Fall.

  • A Discussion on Alternative Off-Policy Corrections for High-Level Actions: Maximum likelihood-based action relabeling worked well empirically, while alternative correction schemes performed worse in preliminary experiments across the tested domains.The alternatives are presented as references for future work on off-policy correction for HRL.
  • A Discussion on Alternative Off-Policy Corrections for High-Level Actions: Direct importance correction was unbiased but had very high variance in continuous action domains and did not work well in practice.The method directly importance-weights samples using the expectation in Eq. 9.
  • A Discussion on Alternative Off-Policy Corrections for High-Level Actions: Importance-based action relabeling seeks a new goal whose importance weight is approximately 1, but remains biased because such a goal is not guaranteed to exist.The authors nevertheless expect this relabeling may reduce bias.
  • A Discussion on Alternative Off-Policy Corrections for High-Level Actions: Model-based relabeling uses learned forward or inverse dynamics to relabel states or goals, but fitting these models can be as difficult as a fully model-based approach.A low-level policy that eventually completes assigned goals motivates approximate inverse-model forms; this approach did not outperform the authors’ method empirically.
  • B Environment Details: Experiments used MuJoCo with dt = 0.02 and a frame skip of 5.These settings define the simulator configuration for the reported environments.
  • B.1 Gather: Gather used a simulated ant with reduced gear range (−30, 30), depth readings, activity range 10, sensor span 2π, and termination after falling or 500 steps.The reward was the default number of apples minus number of bombs.
  • B.2 Navigation: The navigation tasks used 8 × 8 × 8 block environments, ant joint torques, target observations, distance-based rewards, and 500-step episodes; Falling measured distance in x, y, and z.The Falling episode does not terminate when the ant falls.
  • B.2.1 Maze; B.2.2 Push; B.2.3 Fall: Maze confined the ant to a “⊃”-shaped corridor, evaluated reaching (0, 16); Push required moving left, pushing a block right, then reaching (0, 19); Fall required crossing a chasm using a movable block.Maze, Push, and Fall each defined success as ending within an L2 distance of 5 from the target.

C Implementation Details · C.1 Network Structure

HIRO uses a TD3-style network with hidden layers of size (300, 300), while scaling lower- and higher-level actor outputs to task-relevant action ranges. The higher-level ranges separately constrain relative position and torso orientation goals, with remaining limb-angle ranges specified in ant.xml.

  • C.1 Network Structure: The networks follow the basic structure proposed by TD3, but use layer sizes (300, 300) instead of (400, 300).This is the sole stated architectural difference from the TD3 structure.
  • C.1 Network Structure: The lower-level actor output uses tanh activation before scaling to the low-level action range of ±30.The scaling maps the activated output to the specified low-level action limits.
  • C.1 Network Structure: The higher-level actor output is scaled to an approximated range of ±10 for desired relative x and y.These bounds apply to the higher-level controller's desired relative position goals.
  • C.1 Network Structure: The higher-level actor output uses an approximated range of ±0.5 for desired relative z.This is a separate range from the relative x and y limits.
  • C.1 Network Structure: The higher-level actor output uses an approximated range of ±1 for desired relative torso orientations.The torso-orientation range is specified independently of the desired relative position ranges.
  • C.1 Network Structure: The remaining higher-level limb-angle ranges are taken from the ant.xml file.Unlike the other higher-level action dimensions, these ranges are not numerically specified in the passage.

C.2 Training Parameters · C.3 Off-Policy Correction

The implementation uses fixed off-policy training settings for both hierarchical controllers, while correcting high-level experience by selecting candidate goals that maximize lower-level action likelihood. Candidate goals include the original goal, observed state displacement, and Gaussian perturbations clipped to the high-level action range.

  • C.2 Training Parameters: Both controllers use discount γ = 0.99 and Adam optimization.The actor learning rate is 0.0001 and the critic learning rate is 0.001.
  • C.2 Training Parameters: Target networks use soft-update rate τ = 0.005 for both controllers.This is paired with a replay buffer of size 200,000 for each controller.
  • C.2 Training Parameters: Lower-level training and target updates occur every 1 environment step, versus every 10 environment steps at the higher level.No gradient clipping is used.
  • C.3 Off-Policy Correction: For each high-level transition, 10 candidate goals are selected to maximize the lower-level actions’ log-probability.The candidates include the original goal, the observed displacement st+c − st, and eight random Gaussian samples.
  • C.3 Off-Policy Correction: The eight random candidate goals are sampled around st+c − st with standard deviation 0.5 × 1/2[high-level action range].Candidates are subsequently clipped to lie within the high-level action range.

C.4 Evaluation

Learned hierarchical policies are evaluated at regular training intervals, with performance averaged over 50 random episodes.

  • Evaluation protocol: Evaluation occurs every 50,000 training steps.
  • Evaluation protocol: Performance is averaged over 50 random episodes.
  • Evaluation protocol: The evaluation procedure applies to learned hierarchical policies.

D Benchmark Details … D.3 Variational Information Maximizing Exploration

The benchmark details compare HIRO with FuN, SNN4HRL, and VIME using task-matched modifications, controlled training procedures, and specified evaluation budgets. The protocols include skill pretraining, high-level skill selection, and seed-averaged performance reporting.

  • D.1 FuN: HIRO was evaluated against FuN through equivalent modifications for transition policy gradients, directional cosine rewards, and representation-based goals.The dilated RNN component was omitted because the tasks were low-dimensional and fully observed.
  • D.1 FuN: For directional rewards, HIRO’s relative-position reward was replaced with a cosine-similarity reward equivalent to FuN’s.
  • D.2 SNN4HRL: SNN4HRL first trains diverse low-level skills with a mutual-information bonus, then trains a high-level policy to switch among skills for task rewards.The benchmark used imported environments and minimal modifications to the official open-source code.
  • D.2 SNN4HRL: SNN4HRL used TRPO with step size 0.01 and discount 0.99, 2-layer 32-unit networks, and final results averaged across 10 random seeds.Low-level training used 5 seeds, while high-level training used 2 seeds per trained low-level policy.
  • D.2 SNN4HRL: 6 skills were trained with αH = 10, 500 updates, and 25M transitions to obtain effective skill primitives.Skill training used 10 mesh divisions per unit, bilinear integration, batch size 50,000, and maximum path length 500.
  • D.2 SNN4HRL: SNN4HRL’s high-level policy selected a categorical skill every 10 time steps while keeping low-level policies fixed.Experiments used sparse and dense maze rewards and searched batch sizes of 1e4, 5e4, and 5e5 transitions; dense rewards often led to local optima.
  • D.3 Variational Information Maximizing Exploration: VIME was evaluated as a non-HRL baseline using its official implementation defaults, batch size 50,000, and 15M transitions across 5 seeds.Only the Gather task required more samples to converge.
  • D.3 Variational Information Maximizing Exploration: VIME and SNN4HRL results differ because both primarily report and compare results on SwimmerMaze and SwimmerGather.

D.4 Option-Critic Architecture

The paper explores continuous-action adaptations of the option-critic architecture, including an off-policy actor-critic formulation designed to reduce non-stationarity and gradient variance. However, these modifications did not produce reasonably working implementations on the authors’ domains, likely because low-level option policies relied only on external task rewards.

  • Continuous-action design: Continuous-action option policies use Gaussian distributions with neural-network means conditioned on state and option, plus global diagonal variance.The authors first tested a naive extension of the official implementation before modifying the critic to learn Q_U(s, ω, a).
  • Off-policy formulation: The modified critic can use both option and action samples off-policy, removing the non-stationarity problem caused by changing low-level behaviors.The implementation also uses reparameterization through the critic to estimate policy gradients with lower variance.
  • Off-policy formulation: A fully off-policy option-critic actor-critic algorithm, analogous to DDPG, is enabled by making similar approximations for the termination policy.The approach can also use off-policy state samples while enumerating all options, because the policy gradient no longer requires next-state estimates.
  • Limitations: The authors could not make the modified option-critic implementation work reasonably on their domains, likely because low-level option policies learned only from the external task reward.They identify high variance in score-function policy gradients for continuous policies as another pragmatic problem and suggest that more sophisticated policy evaluation or improvement could help.
Loading 1805.08296v4…