Source-linked AI summary
Learning Transferable Motor Skills with Hierarchical Latent Mixture Policies
Dushyant Rao, Fereshteh Sadeghi, Leonard Hasenclever, Markus Wulfmeier, Martina Zambelli, Giulia Vezzani, Dhruva Tirumala, Yusuf Aytar, Josh Merel, Nicolas Heess, Raia Hadsell
TL;DR
HeLMS addresses the expense of learning robot tasks from scratch by learning a three-level hierarchy of discrete and continuous skills from offline data. The learned skills transfer across tasks, unseen objects, and state-to-vision policies, and are particularly useful in sparse-reward settings.
Problem
Reinforcement learning requires substantial experience, computation, and time for each task, motivating reusable skills that can be transferred to related tasks.
Source: §1 · p. 1Method
HeLMS learns a three-level hierarchy from offline data, combining a discrete high-level selector, continuous mid-level skills, and a latent-conditioned low-level controller.
Source: §1 · p. 1Results
Takeaways & Limitations
The analysis finds HeLMS most useful in sparse-reward settings, where its skills encourage directed exploration of task-relevant state-space regions.
Source: §1 · p. 1Future work could more actively exploit temporal consistency for sample-efficient transfer and improve fine-tuning of lower-level behaviors.
Source: §5 · p. 9
Abstract
from arXiv · showhide
For robots operating in the real world, it is desirable to learn reusable behaviours that can effectively be transferred and adapted to numerous tasks and scenarios. We propose an approach to learn abstract motor skills from data using a hierarchical mixture latent variable model. In contrast to existing work, our method exploits a three-level hierarchy of both discrete and continuous latent variables, to capture a set of high-level behaviours while allowing for variance in how they are executed. We demonstrate in manipulation domains that the method can effectively cluster offline data into distinct, executable behaviours, while retaining the flexibility of a continuous latent variable model. The resulting skills can be transferred and fine-tuned on new tasks, unseen objects, and from state to vision-based policies, yielding better sample efficiency and asymptotic performance compared to existing skill- and imitation-based methods. We further analyse how and when the skills are most beneficial: they encourage directed exploration to cover large regions of the state space relevant to the task, making them most effective in challenging sparse-reward settings.
1 INTRODUCTION
Task-by-task reinforcement learning is costly, motivating reusable skills; HeLMS learns a hierarchical mixture of discrete and continuous behaviors from offline data and evaluates their transfer and benefits.
1 INTRODUCTION
- The method targets costly task-by-task reinforcement learning, where experience, computation, and time are required for each task.Reusable skills could be adapted to related downstream tasks, particularly where real-world robot interaction is expensive.Source: §1 · p. 1
- Existing transferable-skill approaches commonly use two-level hierarchies with either continuous or discrete skill representations; the paper aims to combine their advantages.Examples include goal-conditioned or latent-space policies and mixture- or option-based methods.Source: §1 · p. 1
- HeLMS learns a three-level hierarchy from offline data, combining discrete behavioral modes with continuous variation across levels of abstraction.Its low level controls motor primitives, continuous mid-level skills parameterize behaviors, and a discrete high level composes and selects among them.Source: §1 · pp. 1–2
- HeLMS is evaluated for transferring skills to new tasks and objects and from state-based to vision-based policies, with improved sample efficiency and performance.The introduction also reports that ablations find benefits from both continuous and discrete components.Source: §1 · pp. 1–2
- The analysis finds hierarchical skills most useful in sparse-reward settings, where they encourage directed exploration of task-relevant state-space regions.The contributions include interpreting learned skills and assessing the utility of discrete and continuous representations.Source: §1 · pp. 1–2
2 RELATED WORK
Prior work learns reusable behaviors through task-based, unsupervised, segmented, primitive-based, hierarchical, and offline approaches. HeLMS builds on offline skill learning while explicitly combining discrete and continuous structure in a three-level hierarchy.
2 RELATED WORK
- Multi-task reinforcement learning learns behaviors across tasks, while unsupervised methods seek distinguishable skills through intrinsic objectives that often maximize visited-state entropy.Multi-task methods can require careful task-set curation because each skill represents a separate task (James et al., 2018; Riedmiller et al., 2018).Source: §2 · p. 2
- Behavioral segmentation identifies repeatable temporal patterns, while motor-primitive methods select or sequence movements for complex manipulation or locomotion.Some mixture models jointly represent low-level primitives and a high-level controller (Muelling et al., 2010; Colomé & Torras, 2018; Pervez & Lee, 2018).Source: §2 · p. 2
- Hierarchical policies typically pair low-level skills with a high-level latent- or goal-conditioned policy, using either discrete or continuous latents.HeLMS differs from the cited discrete-skill approach by explicitly modeling abstract discrete skills, temporal dependence, and lower-level variation in execution (Florensa et al., 2017).Source: §2 · p. 2
- Offline skill-learning methods commonly train two-level models with a continuous high-level encoder and a latent-conditioned low-level controller, then learn a new high-level policy at transfer.Prior encoders use trajectories, look-ahead states, goal states, or Gaussian noise; skill priors or temporal abstraction can improve transfer efficiency.Source: §2 · p. 2
- HeLMS adds a discrete high-level controller and transferable continuous mid-level skills to offline skill learning, using information asymmetry similar to NPMP.Its high-level encoder conditions on a short look-ahead trajectory, while the additional mid-level transfers abstract skills rather than learning a continuous latent policy from scratch.Source: §2 · p. 3, Figure 1 · p. 3
3 METHOD
HeLMS learns a three-level latent skill hierarchy from offline trajectories, then transfers it to downstream reinforcement-learning tasks by training a new high-level policy while freezing lower policy levels. Its discrete skills capture behavioural modes, while continuous latents vary their execution.
3 METHOD
- The two-stage setup learns a hierarchical skill space from offline trajectories, then transfers it to downstream reinforcement learning by training a new high-level controller and freezing lower policy levels.The downstream controller acts over either discrete skills or continuous latents.Source: §3 · p. 3, §3.2 · p. 5
3.1 LATENT MIXTURE SKILL SPACES FROM OFFLINE DATA
- Discrete latents capture behavioural modes, while continuous latents conditioned on them vary each behaviour’s execution for the low-level controller.The categorical skill depends on the previous timestep, yielding a prior that can be viewed as a Hidden Markov model over continuous latents.Source: §3.1 · pp. 3–4
- The variational skill selection depends on the previous skill and current input; continuous skills depend on the selected category and current input.The categorical initial-skill prior is uniform, the continuous-skill prior is fixed N(0, I), and the transition prior is learned.Source: §3.1 · p. 4
- HeLMS gives hierarchy levels different inputs: proprioception to the low level, object poses to the mid-level, and object plus proprioceptive state with short lookahead to the high level.The high-level selects skills using additional context, while the low-level executes motor primitives; the categorical high-level and added mid-level distinguish HeLMS from the cited approach (Merel et al., 2019).Source: §3.1 · p. 4
- The high-level uses a gated head to select a categorical skill, whose mid-level network samples a latent action for the low-level controller’s action distribution.The skill transition prior is learned, and the model components are trained end-to-end via the objective in Equation 3.Source: §3.1 · p. 4
3.2 REINFORCEMENT LEARNING WITH RELOADED SKILLS
- HeLMS-cat freezes both lower levels and learns a categorical controller, regularizing toward a uniform prior to encourage diversity; this can limit flexibility and requires downstream access to all lower-level inputs.HeLMS-mix instead learns a latent mixture policy, regularizes its components toward offline skill posteriors, and can use different inputs such as vision.Source: §3.2 · p. 5
4 EXPERIMENTS
The experiments evaluate learned skills in manipulation, testing whether they produce distinct behaviours and transfer across objects, tasks, modalities, and reward settings.
4.1 EXPERIMENTAL SETUP
- Experiments use a Sawyer arm in a MuJoCo manipulation environment with five object sets spanning simple rectangles to geometries that make grasping or stacking more challenging.The study evaluates transfer to different objects, tasks, and vision-based policies, alongside dense and sparse reward settings.Source: §4.1 · p. 5, §4 · p. 5
4.2 LEARNING SKILLS FROM OFFLINE DATA
- Five skills learned from offline data produce distinct behaviours, including lifting, reaching, and grasping; the learned prior favours staying in the same skill, and all skills are used.The figure shows rollouts with the discrete skill fixed and a histogram of skill use.Source: Figure 3 · p. 6, §4.2 · p. 7
4.3 TRANSFER TO DOWNSTREAM TASKS
- On unseen object sets, both HeLMS variants learn faster than baselines; versus NPMP, they perform better on sets 1 and 3, similarly on set 5, and worse on set 2.Set 2 often requires reorienting a slanted-bottom object, a behaviour uncommon in the offline data.Source: §4.3 · p. 7, Figure 4 · p. 7
- HeLMS-mix and HeLMS-cat reach higher asymptotic performance than NPMP and MPO on the pyramid task, and a rollout shows a successful solution.The task requires placing red beside green, then stacking blue on top.Source: Figure 5 · p. 7, §4.3 · p. 8
- For vision-based stacking, HeLMS improves sample efficiency over MPO with an asymmetric critic and reaches slightly better asymptotic performance.The transfer reuses the low-level controller and regularizes newly initialized vision-based skills toward state-based skills; training still assumes object states.Source: Figure 6 · p. 7, §4.3 · p. 8
4.4 WHERE AND HOW CAN HIERARCHICAL SKILL REUSE BE EFFECTIVE?
- With fully sparse reward, HeLMS is the only method to achieve non-zero reward, while all approaches learn with dense reward.The authors attribute the sparse-reward result to directed exploration that helps the agent encounter rewards.Source: §4.4 · p. 8, Figure 7 · p. 8
- At zero-shot transfer, HeLMS receives staged sparse reward an order of magnitude greater than other approaches and has nearly two orders of magnitude greater grasp-state coverage.Other methods can cover joint space more; coverage is measured as state variance over an episode, with results averaged over 1,000 episodes.Source: §4.4 · pp. 8–9, Table 1 · p. 9
4.5 ABLATION STUDIES
- Discrete-only skills improve sample efficiency in both object-set cases, while combining discrete and continuous behaviours makes a significant difference in the hard case.The results suggest discrete modes matter for challenging objects, while continuous latent variation accommodates differences in execution.Source: §4.5 · p. 9, Figure 8 · p. 9
5 CONCLUSION
HeLMS learns reusable skills from offline data and transfers them across tasks, objects, and modalities. The conclusion identifies sparse-reward directed exploration as a particularly useful setting and points to two avenues for future work.
5 CONCLUSION
- HeLMS clusters offline data into distinct, interpretable behaviours and transfers learned skills across tasks, unseen objects, and state-to-vision policies.Ablations also indicate benefits from modelling both discrete modes and continuous behavioural variation, KL-regularisation during RL, and fine-tuning the full skill mixture.Source: §5 · p. 9
- The skill hierarchy is particularly useful in sparse-reward settings, where it supports directed exploration.Source: §5 · p. 9
- Future work could more actively exploit temporal consistency for sample-efficient transfer and improve fine-tuning of lower-level behaviours.Source: §5 · p. 9
- HeLMS-cat and HeLMS-mix are identical in the cited analysis because both transfer mid-level skills and initialise a new high-level controller at RL start.Source: §5 · p. 9
A.1 ABLATIONS FOR KL-REGULARISATION
The ablation varies offline and online KL-regularisation of mid-level skills across easy and hard object-transfer cases. Online regularisation matters when offline skills use an N(0, I) prior, especially for hard transfer, but has little effect without offline regularisation.
A.1 ABLATIONS FOR KL-REGULARISATION
- With mid-level skills regularised offline to N(0, I), online KL-regularisation is important, especially for hard transfer, where HeLMS-cat outperforms weaker regularisation.Performance degrades significantly as online regularisation falls; without offline KL-regularisation, performance is insensitive across the evaluated online range of 0.01 to 100.0.Source: §A.1 · p. 14
- The authors conjecture that offline N(0, I) regularisation compacts skill latents, making online KL-regularisation useful to prevent drift and skill overlap.Without offline regularisation, skills may occupy more distant latent regions, making additional online regularisation unnecessary.Source: §A.1 · p. 14
- The ablation compares downstream performance across easy object-set 4 and hard object-set 3 transfer, varying online ηz and offline βz regularisation.Figure 9 distinguishes offline regularisation to N(0, I) with βz = 0.01 from no offline regularisation with βz = 0.Source: Figure 9 · p. 14, §A.1 · p. 14
A.2 NPMP ABLATION
The appendix reports improvements to the NPMP baseline and describes how MPO and RHPO optimise the two HeLMS transfer variants. MPO updates a categorical controller, while RHPO adapts its policy update to the latent mixture.
A.2 NPMP ABLATION
- Two NPMP changes—an N(0, I) prior and KL-regularisation during RL—substantially improve manipulation performance across all five object sets.The paper uses this modified variant in its main experiments; Figure 10 presents the NPMP ablation.Source: §A.2 · p. 15, Figure 10 · p. 15
B REINFORCEMENT LEARNING WITH MPO AND RHPO
- HeLMS-cat learns a new high-level categorical policy via MPO, whereas HeLMS-mix learns a mixture policy over continuous latents via RHPO.The appendix describes the base optimisation methods, omitting additional KL-regularisation terms introduced for transfer.Source: B REINFORCEMENT LEARNING WITH MPO AND… · p. 15
B.1 HELMS-CAT VIA MPO
- MPO updates the critic, constructs a non-parametric policy by critic-weighting sampled actions, then fits a parametric policy under trust-region constraints.For HeLMS-cat, MPO sets the high-level categorical variable yt.Source: §B.1 · pp. 15–16
B.2 HELMS-MIX VIA RHPO
- RHPO optimises HeLMS-mix in continuous latent space and applies separate trust-region constraints to the categorical policy and mixture components.Separate constraints let the optimiser independently regulate changes to categorical probabilities, component means, and component variances.Source: §B.2 · p. 16
C ELBO DERIVATION AND INTUITIONS
The ELBO for the state-conditional action distribution is expressed as a log-likelihood objective with a KL penalty over latent variables. Its structured terms weight component learning by posterior assignment and permit recursive computation over categorical skills.
C ELBO DERIVATION AND INTUITIONS
- The ELBO lower-bounds the state-conditional action log-likelihood using a posterior-to-prior KL penalty over discrete and continuous latent variables.The derivation expands the joint log probability and approximate posterior into per-timestep action, latent, and transition terms.Source: C ELBO DERIVATION AND INTUITIONS · p. 17
- Each mixture component receives reconstruction and KL terms weighted by its posterior probability, encouraging components to specialise across the state space.The objective includes the component action reconstruction loss and latent KL regulariser.Source: C ELBO DERIVATION AND INTUITIONS · p. 17
- The categorical transition KL is weighted by the previous component’s posterior, and component posteriors can be updated recursively without sampling the categorical variable.The derivation marginalises discrete values and approximates the expectation over continuous zt by sampling.Source: C ELBO DERIVATION AND INTUITIONS · pp. 17–18
D ENVIRONMENT PARAMETERS
Experiments use a MuJoCo Sawyer manipulation environment with three colored objects and five simulated object sets designed to vary geometry and interaction difficulty.
D ENVIRONMENT PARAMETERS
- The MuJoCo Sawyer environment uses red, green, and blue objects, with all state variables stacked over three frames.State variables are summarized in Table 3; action dimensions are normalized to [−1, 1] during learning, as summarized in Table 4.Source: D ENVIRONMENT PARAMETERS · p. 18, Table 3 · p. 18, Table 4 · p. 18
- RL experiments randomize physics and sample action delays of 0–2 timesteps to support robustness to small environmental changes.Physics is randomized, but visual appearance is not.Source: D ENVIRONMENT PARAMETERS · p. 18
D.1 OBJECT SETS
- Five object sets span different geometries and stacking challenges; the experiments use their simulated versions.The sets were designed to cover object geometries and affordances; Figure 11 depicts the five triplets and is reproduced from Lee et al. (2021).Source: §D.1 · p. 19, Figure 11 · p. 19
- Set 1 challenges grasping and rolling, sets 2 and 3 challenge reorientation or precise placement, and sets 4 and 5 are relatively easier.Set 4 uses rectangular prisms; set 5 has a ten-faced blue object with limited stacking surface area.Source: §D.1 · p. 19
E NETWORK ARCHITECTURES AND HYPERPARAMETERS
The experiments use mostly MLP controllers, with input representation networks and separate vision-transfer architecture choices; hyperparameters are swept selectively.
E NETWORK ARCHITECTURES AND HYPERPARAMETERS
- The authors sweep offline β and RL η coefficients, find RHPO ϵ fairly insensitive, keep other parameters fixed, and run RL with three seeds.Table 5 presents HeLMS architecture and hyperparameter details for offline training and RL.Source: E NETWORK ARCHITECTURES AND… · p. 19, Table 5 · p. 20
- Most experiments use two-layer MLPs for the high- and low-level controllers and each mid-level mixture component; vision experiments are the exception.Networks learned from scratch receive encoded inputs from representation networks.Source: E NETWORK ARCHITECTURES AND… · p. 19
- Vision policies learn both high- and mid-level components from scratch, with the mid-level KL-regularized to offline skills.State-based experiments can reuse mid-level components conditioned on object state; the critic is a three-layer MLP with a separate input representation network and concatenated action.Source: E NETWORK ARCHITECTURES AND… · p. 20
F REWARDS
The reward designs range from staged dense shaping to fully sparse task completion, using shared primitive rewards to define subtasks.
F REWARDS
- Reward functions for stacking and pyramid tasks use staged subtasks and primitives whose rewards lie in [0, 1].Primitives include shaped reach, lift, hover, and above rewards; grasp is binary, while stack, pyramid, and place_near are sparse.Source: F REWARDS · pp. 20–21
- Dense stacking rewards progress through reaching and grasping red, lifting it, hovering over blue, stacking, and moving the gripper above.The latest stage with nonzero reward is current; earlier stages receive 1, and the summed reward is scaled so the timestep maximum is 1.Source: F REWARDS · p. 21
- Sparse staged stacking sparsifies each stage until its reward exceeds 0.95, reflecting settings where meaningful shaped rewards may be difficult to specify.The staged reward gives incremental feedback as stacking subtasks are completed.Source: F REWARDS · p. 21
- The fully sparse stacking reward provides reward only when the red object is stacked on the blue object.For pyramid building, stages are sparsified at 0.95 and progress through grasping, lifting, positioning, placing, stacking, and moving the gripper above.Source: F REWARDS · pp. 21–22