PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Skills for Efficient Exploration

Jonas Gehring, Gabriel Synnaeve, Andreas Krause, Nicolas Usunier

arXiv:2110.10809v1cs.LGcs.AIcs.RO

TL;DR

Pre-trained skills may improve exploration, but choosing between generality and task-specific efficiency has relied on prior knowledge, especially in continuous-control work focused on locomotion and navigation. The paper learns skills of varying complexity without task rewards and uses a three-level policy to select among them. On sparse-reward bipedal tasks, dynamic selection matches or exceeds a generic skill on all tasks and can improve over the best single skill per task.

  • Problem

    Prior skill design relies on downstream-task knowledge to balance generality against task-specific exploration benefits, and this trade-off has not been explicitly examined beyond navigation-focused continuous-control settings.

    Source: §1 · p. 1

  • Method

    The authors unsupervisedly pre-train skills of varying complexity and use a three-level policy to select a skill, its goal, and native actions on downstream tasks.

    Source: §1 · pp. 1–2

  • Results

    Dynamic goal-space selection performs equally or better than a generic skill on all tested tasks and can improve over the best single skill per task.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The benchmark and experiments demonstrate HSD-3’s efficacy on varied sparse-reward bipedal tasks, including effective policies on most tested Humanoid tasks.

    Source: §6 · p. 9

    Humanoid investigations use limited goal spaces and significant priors; acquiring exhaustive unsupervised skill sets for more complex morphologies remains future work.

    Source: §6 · p. 9

Abstract

from arXiv · show

In reinforcement learning, pre-trained low-level skills have the potential to greatly facilitate exploration. However, prior knowledge of the downstream task is required to strike the right balance between generality (fine-grained control) and specificity (faster learning) in skill design. In previous work on continuous control, the sensitivity of methods to this trade-off has not been addressed explicitly, as locomotion provides a suitable prior for navigation tasks, which have been of foremost interest. In this work, we analyze this trade-off for low-level policy pre-training with a new benchmark suite of diverse, sparse-reward tasks for bipedal robots. We alleviate the need for prior knowledge by proposing a hierarchical skill learning framework that acquires skills of varying complexity in an unsupervised manner. For utilization on downstream tasks, we present a three-layered hierarchical learning algorithm to automatically trade off between general and specific skills as required by the respective task. In our experiments, we show that our approach performs this trade-off effectively and achieves better results than current state-of-the-art methods for end- to-end hierarchical reinforcement learning and unsupervised skill discovery. Code and videos are available at https://facebookresearch.github.io/hsd3 .

1 Introduction

Pre-trained skills can improve exploration, but prior assumptions about downstream tasks create a trade-off between skill specificity and generality. The paper proposes an unsupervised hierarchy and a three-level controller, then evaluates them on sparse-reward bipedal tasks.

Motivation

  • Prior work often assumes skills should control the robot’s center of mass, fitting navigation benchmarks while leaving their value for other task classes unclear.Skill design trades off sample efficiency against generality, so no single pre-trained skill is universally superior.Source: §1 · p. 1

Approach

  • The authors propose unsupervised pre-training of skills with increasing complexity, which can be composed by a high-level policy for varied downstream tasks.Skills control parts of a robot configuration over short horizons, such as a foot’s position or the torso’s orientation.Source: §1 · p. 1
  • A three-level policy selects a skill, its target configuration, and then low-level actions, shifting the choice of prior knowledge to downstream learning.Unlike a single static pre-trained skill, this hierarchy can select different skills as needed.Source: §1 · p. 2

Results

  • Dynamic goal-space selection performs as well as or better than a generic skill on every task and can improve over the best single skill for each task.The experiments also report better results than existing skill-discovery and hierarchical reinforcement-learning approaches.Source: §1 · p. 2

Contributions

  • The paper introduces sparse-reward benchmark tasks to evaluate motor skills and hierarchical methods beyond traditional navigation settings.The experiments use simulated bipedal robots.Source: §1 · p. 2

2 Related Work

Prior hierarchical reinforcement learning separates low-level skills from higher-level decisions, but much continuous-control work favors skills suited to navigation. The paper frames the unresolved issue as balancing task-specific benefits with wider applicability.

Prior hierarchical methods

  • Prior work discovers low-level primitives from random walks, mutual-information objectives, agent or expert traces, motion capture, and dedicated pre-training tasks.Hierarchical methods can ease individual learning problems and support long-term decisions, while improved exploration is a primary benefit in modern neural systems.Source: §2 · p. 2

Skill generality and specificity

  • Many continuous-control benchmarks center on navigation, favoring directed locomotion skills that control center-of-mass translation or emphasize position features in rewards.Non-navigation work has also used tailored pre-training tasks or in-domain motion-capture data.Source: §2 · p. 2
  • The paper targets unsupervised skill learning without trace supervision or task-designed pre-training, avoiding a fixed behavioral trade-off.The accompanying framework illustration distinguishes empty-environment pre-training from downstream three-level control.Source: §2 · p. 2, Figure 1 · p. 3

3 Hierarchical Skill Learning

The framework pre-trains goal-directed skills over robot features and uses a three-level policy to choose goal spaces, goals, and native actions. Its training procedure shares low-level policy parameters across skills and adapts soft actor-critic to the hierarchical action space.

3.1 Overview

  • Skills target short-horizon configurations defined over robot-state features, with single features and combinations exposing different amounts of control to the high-level policy.Goal-directed skills use proprioceptive observations and receive no task-specific rewards.Source: §3.1 · p. 3
  • On downstream tasks, a high-level policy selects a goal space and a goal, while a pre-trained low-level policy controls the robot; high-level policies can use task-specific state unavailable to the low-level policy.The goal-space policy can switch skills within an episode, trading off generality and specificity.Source: §3.1 · p. 3

3.2 Unsupervised Pre-Training

  • A shared low-level policy is conditioned on the feature set and goal, avoiding a separate policy for every feature subset; its distance-based reward measures progress toward the selected goal.The feature set is encoded as a binary bag of words, and the feature selection and transformations constitute the prior knowledge used for unsupervised pre-training.Source: §3.2 · p. 4
  • Pre-training samples feature sets and goals by episode, uses short-horizon skills, and resets only sporadically to encourage applicability across states.The low-level policy can use any reinforcement-learning algorithm; the experiments use Soft Actor-Critic (SAC).Source: §3.2 · p. 4

3.3 Hierarchical Control

  • The downstream hierarchical policy factorizes actions into goal-space choice, goal choice, and low-level control, and the authors extend SAC to learn the two high-level policies over a joint discrete-continuous action space.A shared critic incorporates both action factors, with separate entropy temperatures for goal-space and goal policies; the goal-space entropy is computed in closed form and adjusted by subtracting log |F|.Source: §3.3 · p. 4
  • High-level actions are selected less frequently than low-level actions, reducing the number of high-level transitions; a step-conditioned critic uses transitions gathered between high-level decisions.The critic receives the number of steps since the last high-level action as an additional input.Source: §3.3 · p. 5
  • The high-level policies are parameterized separately with a shared critic, a design reported as more effective in initial experiments than the cited alternative.The losses include both entropy terms, and entropy temperatures are updated automatically to maintain target entropies.Source: §3.3 · p. 5

4 Benchmark Environments

The benchmark evaluates bipedal motor skills on diverse tasks requiring locomotion, obstacle negotiation, foot control, and balance. Sparse rewards and episode-level environmental variation make exploration central to the task design.

4 Benchmark Environments

  • The suite tests Hurdles, Limbo, Stairs, GoalWall, and PoleBalance, covering jumping, torso and foot control, balance, and locomotion.All but PoleBalance require locomotion; PoleBalance provides a constant reward until the pole falls.Source: §4 · p. 5
  • Rewards are sparse in all tasks except PoleBalance, and obstacle positions and step lengths vary randomly between episodes.Falling terminates an episode with a reward of -1.Source: §4 · p. 5
  • Hurdles, Stairs, and Gaps have prior variants, but these tasks differ in parameterization and reward sparsity.The benchmark also includes HurdesLimbo, where Hurdles and Limbo obstacles alternate.Source: §4 · p. 5, Figure 2 · p. 6

5 Experimental Results

Experiments compare fixed low-level goal spaces with hierarchical skill selection on Walker tasks, then evaluate HSD-3 on a 21-joint Humanoid. Skill performance varies by task, while HSD-3 matches or exceeds single-goal-space approaches in the reported comparisons.

5 Experimental Results

  • Walker experiments train 31 shared-parameter skills from five robot features, then train separate high-level policies for each benchmark task.High-level actions occur every five environment steps, except PoleBalance, where all three policies act at the same frequency.Source: §5 · p. 6

5.1 Trade-offs for Low-Level Skills

  • After 5M samples, the best fixed goal-space skills differ across Walker tasks; the most general skill enables progress but is usually outperformed by specific skills.Feet-position control is crucial for GoalWall, though close-to-optimal performance occurs in few runs.Source: §5.1 · pp. 6–7, Figure 3 · p. 6

5.2 Skill Selection with HSD-3

  • On Walker tasks, HSD-3 matches or beats the full goal-space policy's learning speed and significantly exceeds single-space approaches on GoalWall and Stairs.The final performances are reported in Table 1 after 7.5M samples; learning curves cover 5M interactions.Source: §5.2 · p. 7, Table 1 · p. 7, Figure 4 · p. 7
  • HSD-3 selects different goal spaces across Stairs stages, while bandit selection fails on most environments and several baselines make little progress.SAC performs well on dense-reward PoleBalance; HIRO-SAC and HIDIO make no meaningful progress, and HIRO-SAC and DIAYN-C can visit more states without converting that experience into higher returns.Source: §5.2 · p. 8, Figure 5 · p. 8

5.3 Evaluation on a Humanoid Robot

  • On the 21-joint Humanoid, HSD-3 matches or surpasses the full goal space and can outperform the best single goal space selected in hindsight.HSD-3 outperforms both single-space comparisons on HurdlesLimbo and learns Limbo faster than SD; none of the hierarchical policies makes significant progress on GoalWall or Gaps.Source: §5.3 · pp. 8–9, Figure 6 · p. 9

6 Conclusion

The paper presents HSD-3 as a way to let downstream policies balance directed exploration and fine-grained control using hierarchically pre-trained skills. Its simulated bipedal experiments demonstrate the framework on sparse-reward tasks, while Humanoid investigations remain limited to goal spaces with significant priors.

6 Conclusion

  • HSD-3 lets high-level policies trade off directed exploration and fine-grained control during downstream training, and performs effectively on sparse-reward Walker tasks.The framework is also applied to a Humanoid robot, where it learns effective policies on most tasks.Source: §6 · p. 9
  • The benchmark tasks require diverse motor skills and are intended to extend hierarchical control research beyond navigation.The task distribution is released to support further research in continuous-control motor skills.Source: §6 · p. 9
  • Humanoid investigations use limited goal spaces and therefore retain significant prior knowledge; acquiring exhaustive unsupervised skills for more complex morphologies remains future work.Source: §6 · p. 9

A Environment Details

The appendix describes sparse-reward bipedal benchmark environments, robot-specific feature spaces, and normalized goal spaces used to specify subsets of robot-state features.

A Environment Details

  • Benchmark episodes start from designated robot positions and configurations with joint-position noise uniform on [−0.1, 0.1] and joint-velocity noise 0.1·N(0,1); invalid states defined per robot prematurely end episodes with reward −1.Unless otherwise noted, episodes consist of 1,000 interaction steps.Source: A Environment Details · p. 13
  • The seven benchmark tasks vary obstacle layouts and motor demands; most require locomotion, while PoleBalance tests balance without it.The tasks use randomized obstacle positions, and rewards generally arrive after passing an obstacle or step or reaching a goal.Source: A Environment Details · p. 13, Table 2 · p. 14

B.1 Walker

  • The Walker uses joint positions, velocities, and contact forces as observations, with five primitive goal features combined into 31 candidate goal spaces.Goal features include torso position and rotation and foot positions relative to the torso; some are derived or global rather than directly observed.Source: §B.1 · p. 14

B.2 Humanoid

  • The 21-joint Humanoid uses joint-target actions and Walker-like state features, with torso and relative-foot features defining 31 candidate goal spaces.Its valid states require the torso to remain at least 0.9 meters above ground, with no body-rotation restriction.Source: §B.2 · p. 15

B.3 Goal Space Construction

  • Each goal space selects a subset of goal features and normalizes their ranges to [−1, 1], yielding normalized distance-based rewards and subgoal actions.The affine construction is intended to reduce bias from differing feature magnitudes and support shared policy parameters across goal spaces.Source: §B.3 · p. 15

C Unsupervised Pre-Training Details

Unsupervised pre-training samples feature subsets and goals in a robot-only environment, then trains a shared low-level skill policy using standard Soft Actor-Critic updates.

C Unsupervised Pre-Training Details

  • Each episode samples a goal space and goal at random, and the shared low-level policy and Q-functions receive goal-feature and goal representations as inputs.The algorithm uses standard Soft Actor-Critic updates during training.Source: C Unsupervised Pre-Training Details · p. 16
  • A goal ends when it is reached, its horizon expires, or a resampling event occurs; simulation resets follow invalid states or a fixed number of sampled goals.An invalid state receives reward −1.Source: C Unsupervised Pre-Training Details · p. 16
  • The training loop samples a feature set and normalized goal, stores transitions, and periodically updates policy and value parameters with SAC.Source: C Unsupervised Pre-Training Details · p. 16

D.1 Soft-Actor Critic for HSD-3

The high-level policy chooses both a discrete goal-space feature set and a continuous goal. Its SAC formulation shares a critic and adjusts entropy treatment for the two action types.

D.1 Soft-Actor Critic for HSD-3

  • The high-level action combines a discrete feature set with a continuous goal, using a shared critic and separate policies for feature-set choice and goal selection.Source: §D.1 · p. 16
  • Table 3 lists the hyperparameters for unsupervised pre-training.Source: Table 3 · p. 17
  • The feature-set policy is evaluated over all discrete choices, while separate automatically adjusted temperature coefficients handle discrete and feature-conditioned continuous actions.Continuous-action entropy contributions are normalized by the dimensionality of the selected feature set.Source: §D.1 · p. 17

E Training Details

The appendix details HSD-3 training and its baselines, then reports that multi-task pre-training improves returns in most benchmark environments. It also notes that exploration is difficult across environments except PoleBalance.

E Training Details

  • HSD-3 uses four-layer neural networks with skip connections and ReLU activations, concatenating multiple inputs and using a multi-head goal policy.The HSD-3 Q-function assigns separate channels to goal inputs for each goal space.Source: E Training Details · p. 17
  • The evaluation protocol averages returns over 50 deterministic trials using the same 50 environment seeds for every method.The appendix also specifies separate learning-rate and initial-temperature searches for HSD-3 and SD versus the baselines.Source: E Training Details · p. 18

E.1.1 HIRO-SAC

  • HIRO-SAC uses Soft Actor-Critic, relabels goals by maximizing low-level action log-probabilities, and updates the high-level policy less frequently than the low-level policy.The implementation reports worse performance when including DynE critic updates.Source: §E.1.1 · pp. 18–19

E.1.2 DIAYN-C

  • DIAYN-C learns a continuous representation of discrete skills, applies its discriminator to the predefined goal spaces, and uses 10 skills with a 256-unit policy for best overall performance.The selected skill-policy configuration was identified across experiments with 5 to 100 discrete skills and 256 or 1024 hidden units.Source: §E.1.2 · p. 19

E.1.3 Switching Ensemble

  • The Switching Ensemble baseline randomly selects among five SAC policies during rollouts, but uses one policy for an entire evaluation episode.Switches occur at the same frequency as high-level actions in the other hierarchical methods.Source: §E.1.3 · p. 19

E.1.4 HIDIO

  • HIDIO’s discriminator operates on the full state space, with option lengths of one step on PoleBalance and five steps elsewhere.Its discount factor is 0.99, and the low-level policy uses 256 hidden units.Source: §E.1.4 · p. 19

F.1 Walker Learning Curves

  • Exploration is challenging in every environment except PoleBalance; SAC, Switching Ensemble, and HIRO-SAC obtain positive returns with only a few seeds.Table 6 reports mean returns over environment steps, with standard deviations across nine seeds.Source: §F.1 · p. 20

F.2 Ablation: Multi-Task Pre-Training

  • Shared multi-task pre-training yields higher returns in most benchmark environments, and the single-goal-space policy prevents learning progress on GoalWall.The shared model also provides a compact representation of skills for downstream use.Source: §F.2 · p. 20

F.3 Analysis: Exploration Behavior

The analysis finds that hierarchical methods generally visit more states than SAC, but greater visitation does not necessarily yield higher returns. Pre-training goals become harder as feature sets grow, while Humanoid results vary by task and seed.

F.3 Analysis: Exploration Behavior

  • Hierarchical methods generally encounter more states than SAC, but HSD-3’s new-state counts decline as returns converge after initial fast learning on four Walker tasks.Figure 8 tracks unique hashed states; the analysis uses SimHash on full observations, which may include randomly placed objects.Source: §F.3 · pp. 20–21, Figure 8 · p. 22

F.4 Analysis: Pre-Training Performance

  • As feature sets grow, sampled goals become harder to reach partly because hypercube sampling includes unreachable configurations, such as foot-position targets outside the reachable half-moon region.The effect compounds when feature sets are combined; Figure 9 plots controllability during pre-training.Source: §F.4 · p. 22, Figure 9 · p. 23

F.5 Humanoid Learning Curves

  • HSD-3 outperforms the best single goal space on Humanoid Limbo and HurdlesLimbo, while its Stairs median is modest because one seed fails to learn.SAC has positive returns only on Stairs, where two successful runs climb the initial flight but fall before completing the course; Table 7 shows learning curves.Source: §F.5 · p. 22, Table 7 · p. 24

G Implicit Priors in HIRO

The HIRO ablation examines whether reward normalization changes its implicit preference for center-of-mass translation. The reported results distinguish the reward-weighting mechanism from the learning outcome under normalized reward and state inputs.

G Implicit Priors in HIRO

  • HIRO uses larger manual subgoal ranges for X-Y locations than for joint angles, steering its low-level policy toward translation.The ablation examines reward normalization on AntMaze.Source: G Implicit Priors in HIRO · p. 25
  • Reward normalization makes all goal-space dimensions contribute equally to the low-level learning signal and removes the implicit center-of-mass translation prior.The reward scaling vector corresponds to the high-level policy’s action scaling.Source: G Implicit Priors in HIRO · p. 25
  • With normalized rewards, only the nearby goal at (16,0) can be reached; with normalized state-representation inputs, no learning progress is made.Figure 11 reports goal-reaching fractions, while Figure 12 reports the representation-learning evaluation.Source: Figure 11 · p. 25, Figure 12 · p. 25
Loading 2110.10809v1…