Source-linked AI summary
Latent Space Policies for Hierarchical Reinforcement Learning
Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, Sergey Levine
TL;DR
The paper addresses how to learn useful hierarchical RL policies without restricting lower layers or relying on hand-designed hierarchy mechanisms. It uses maximum-entropy latent-variable policies with invertible mappings and layerwise training, reporting improved benchmark performance and solutions to sparse-reward tasks through lower-level shaping rewards.
Problem
Automated hierarchical RL must induce useful, diverse lower-level behaviors without collapsing primitives or crippling layers through restricted information or heuristic objectives.
Method
The method trains maximum-entropy latent-variable policies layerwise, using each lower layer’s latent space as the higher layer’s action space and invertible mappings to preserve expressivity.
Results
Experiments report improved performance over single-layer policies on standard benchmarks and effective solutions to more challenging sparse-reward tasks using lower-level shaping rewards.
Takeaways & Limitations
Hierarchies can be constructed by adding policy layers while allowing each layer to solve the task directly and preserving unconstrained behavior through invertibility.
Abstract
from arXiv · showhide
We address the problem of learning hierarchical deep neural network policies for reinforcement learning. In contrast to methods that explicitly restrict or cripple lower layers of a hierarchy to force them to use higher-level modulating signals, each layer in our framework is trained to directly solve the task, but acquires a range of diverse strategies via a maximum entropy reinforcement learning objective. Each layer is also augmented with latent random variables, which are sampled from a prior distribution during the training of that layer. The maximum entropy objective causes these latent variables to be incorporated into the layer's policy, and the higher level layer can directly control the behavior of the lower layer through this latent space. Furthermore, by constraining the mapping from latent variables to actions to be invertible, higher layers retain full expressivity: neither the higher layers nor the lower layers are constrained in their behavior. Our experimental evaluation demonstrates that we can improve on the performance of single-layer policies on standard benchmark tasks simply by adding additional layers, and that our method can solve more complex sparse-reward tasks by learning higher-level policies on top of high-entropy skills optimized for simple low-level objectives.
1. Introduction
The paper proposes hierarchical RL policies in which every layer directly solves the task while maximum-entropy latent variables provide diverse, controllable behaviors. Invertible latent-to-action mappings preserve expressivity, and experiments report improvements on benchmark and sparse-reward tasks.
- Automated hierarchy construction must avoid collapsing primitives or crippling lower layers through restricted observations, hand-designed features, or diversity heuristics.
- Each layer directly attempts the task and uses internal latent variables whose values determine how states map into actions.
- Invertible latent-to-action mappings let higher layers undo lower-layer action transformations, leaving every layer unconstrained in sensing and environmental influence.
- Maximum-entropy reinforcement learning encourages latent variables to encode diverse strategies that higher layers can control through the lower layer’s latent action space.
- The layerwise framework trains one latent-variable policy at a time, using either the task reward or lower-level shaping rewards.
- Experiments report state-of-the-art sample complexity on benchmark tasks, further gains from added layers, and effective solutions to sparse-reward tasks using lower-level shaping rewards.
2. Related Work
Related hierarchical RL methods use options or contextual policies, but automated end-to-end training can produce degenerate hierarchies. Prior approaches address this with entropy or mutual-information objectives, while this paper uses entropy maximization to diversify base policies.
- Options combine low-level option policies with a top-level policy, while contextual policies generalize options to continuous goals.
- End-to-end hierarchical training can learn degenerate policies that exclusively use one option and lose the benefits of hierarchy.
- Prior methods encourage hierarchy through entropy regularization, mutual information between high-level actions and state distributions, or bounds on action-level mutual information.
- This method also uses entropy maximization to obtain diverse base policies for hierarchical learning.
3. Preliminaries
The preliminaries formalize continuous-action RL as an MDP and introduce maximum-entropy optimization, which trades off expected reward against policy entropy through a temperature parameter.
- Continuous-action policy learning is formalized with state and action spaces, transition probabilities, bounded rewards, initial-state distributions, and trajectory distributions.
- The standard RL objective maximizes the expected sum of rewards over trajectories.
- Maximum-entropy RL augments expected reward with the policy’s expected entropy over visited states.
- The temperature parameter α > 0 controls the relative weight of entropy and therefore the stochasticity of the optimal policy.
- In practice, the method optimizes a discounted infinite-horizon objective and omits α explicitly by absorbing it into reward scaling.
4. Control as Inference
The paper derives maximum-entropy RL by treating control as probabilistic inference over optimal trajectories. Structured variational inference preserves the true dynamics while learning a parameterized policy, recovering the entropy-regularized objective under a uniform action prior.
- The control problem is represented as inference in a graphical model with environment dynamics, action priors, and binary optimality variables.
- The inferred posterior action distribution conditions on current and future optimality variables to produce optimal behavior.
- Setting p(O_t|s_t, a_t) = exp(r(s_t, a_t)) incorporates reward into the probability of a trajectory being optimal.
- Structured variational inference constrains the approximate distribution to the true dynamics and a parameterized policy, avoiding direct use of an intractable optimal posterior.
- With a uniform action prior, maximizing the ELBO yields the maximum-entropy RL objective up to a constant and permits standard entropy-maximizing RL algorithms.
5. Learning Latent Space Policies
The method constructs hierarchical policies by augmenting maximum-entropy reinforcement learning with latent variables and invertible mappings, then training policy layers sequentially so each layer’s latent space becomes the next layer’s action space.
- Latent-variable policy construction: The framework augments the probabilistic graphical-model view of maximum-entropy RL with latent variables and bijective action transformations.This supports stable, tractable training while allowing higher levels to control lower-level behavior through latent actions.
- Latent-variable policy construction: A latent base policy exposes its latent variable as a higher-level action, producing a policy-shaped dynamical system for learning related tasks.The combined dynamics are obtained by marginalizing the physical actions, with the latent variable taking their role at the higher level.
- Invertible sub-policies: Each sub-policy is designed to be tractable, expressive, and deterministic, using invertible transformations so higher layers retain full control of the action space.Bijectivity makes marginalization a single-point evaluation and prevents lower layers from suppressing information from higher layers.
- Layerwise hierarchy construction: Hierarchies are built layerwise by training one maximum-entropy policy, freezing it, and using its latent variables as the action space for the next policy.The process can be repeated to construct arbitrarily deep hierarchical policy representations.
- Layerwise hierarchy construction: Lower layers may optimize either the true task reward or shaping rewards, while entropy maximization yields diverse controllable behaviors for higher-level policies.A weak motion objective can produce movement in many directions controlled by lower-layer latent variables, supporting quadrupedal goal navigation.
- Algorithm summary: The algorithm sequentially learns invertible transformations with specified rewards, embeds each learned layer into the environment, and operates without requiring analytic system dynamics.Soft actor-critic is used in the implementation, but other entropy-maximizing RL algorithms can also be used.
6. Experiments
Experiments evaluate SAC-LSP on continuous-control benchmarks, hierarchical policies for Ant and Humanoid, and a sparse-reward Ant maze task. SAC-LSP matches or exceeds prior methods, stagewise hierarchy improves challenging tasks, and latent skills support direct task-reward optimization in the maze.
- Benchmark comparisons: Experiments compare SAC-LSP with PPO, DDPG, soft Q-learning, and SAC-GMM on continuous-control benchmarks.The comparison considers learning efficiency and final return, with stochastic test-time policies for maximum-entropy methods.
- Benchmark comparisons: SAC-LSP generally performs on par with or better than tested prior methods, especially on challenging high-dimensional tasks such as Humanoid.Figure 3 reports training curves using means and standard deviations from five random seeds.
- Hierarchical policies: A two-level policy can outperform a single-level policy by a large margin on Ant and Humanoid after freezing the base policy and training a second level.The performance boost is more prominent when the base policy is trained longer.
- Hierarchical policies: Stagewise training improves performance beyond simply increasing policy expressivity: an end-to-end four-layer policy is comparable on Ant but worse on Humanoid.The comparison uses a single four-invertible-layer policy with a similar parameter count to the stacked two-level policy.
- Sparse-reward maze: In the sparse-reward Ant maze, SAC-LSP reuses pretrained locomotion skills while optimizing task reward directly, producing faster learning and better final performance.Its final distance to the goal is more than four times smaller than the next-best method, which fine-tunes with a shaped reward.
7. Discussion and Future Work
The discussion presents latent-variable policies as a basis for layerwise hierarchical reinforcement learning. Invertibility preserves higher-level expressivity while allowing lower layers to use shaping rewards and higher layers to optimize task rewards.
- Discussion: The framework trains each hierarchy layer with the same algorithm, using the lower layer’s latent space as the higher layer’s action space.Policies are trained layer by layer rather than requiring all levels to be optimized jointly.
- Discussion: Invertible latent-to-action mappings let higher layers perfectly invert lower-layer behavior, preserving task-specific optimization at higher levels.This permits lower layers to be trained with heuristic shaping rewards while higher layers optimize the task reward.
- Future work: The hierarchy can be interpreted as an iterative construction of graphical models that gradually simplify task dynamics.
A.1. Common Parameters
The appendix specifies shared implementation parameters for LSP policies and identifies separate environment-specific parameter tables. These settings govern replay, training frequency, batching, and network updates.
- Common parameters: LSP training uses a replay pool of one million samples and delays training until at least 1000 samples are collected.
- Common parameters: Each training iteration contains 1000 environment steps, with all networks trained at every time step using batches of 128.
- Parameter tables: Table 1 lists parameters shared across benchmark tasks, while Table 2 lists parameters that vary by environment.
A.2. High-Level Policies
High-level policies use Gaussian action priors and, in the Ant maze, hold sampled latent actions fixed across multiple environment steps. Low-level policies use velocity-based rewards, while the maze itself provides only sparse goal reward.
- High-level policies: All high-level policies use a Gaussian action prior.
- High-level policies: For the Ant maze, the high-level latent sample is drawn once per rollout and held fixed for three environment steps.
- Reward settings: Low-level policies receive velocity-based rewards in any xy-plane direction, whereas the Ant maze gives 1000 only upon reaching the goal and 0 otherwise.The maze provides no velocity reward or control costs and terminates after goal attainment.