PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamics-Aware Unsupervised Discovery of Skills

Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, Karol Hausman

arXiv:1907.01657v2cs.LGcs.ROstat.ML

TL;DR

MBRL must learn global dynamics models that are difficult to estimate accurately and may generalize poorly outside their training distribution. DADS instead discovers predictable skills while learning their dynamics, then plans with those skills in a continuous latent space. The learned skills support zero-shot planning that outperforms standard model-based and goal-conditioned RL baselines, including on sparse-reward tasks.

  • Problem

    Global dynamics models are difficult to learn for complex systems and may not generalize well outside the states used for training.

  • Method

    DADS jointly discovers predictable, diverse skills through unsupervised mutual-information-based exploration and learns their skill-transition dynamics for latent-space planning.

  • Results

    Zero-shot planning with DADS skills outperforms model-based RL baselines, including baselines trained with test-task knowledge, and supports sparse-reward navigation.

  • Takeaways & Limitations

    Continuous, low-variance skills can be composed for downstream tasks without additional task training, including in high-dimensional environments.

  • Takeaways & Limitations

    The model-based RL comparison gives some baselines an advantage through goal-directed exploration during training, unlike DADS’s task-agnostic mutual-information exploration.

Abstract

from arXiv · show

Conventionally, model-based reinforcement learning (MBRL) aims to learn a global model for the dynamics of the environment. A good model can potentially enable planning algorithms to generate a large variety of behaviors and solve diverse tasks. However, learning an accurate model for complex dynamical systems is difficult, and even then, the model might not generalize well outside the distribution of states on which it was trained. In this work, we combine model-based learning with model-free learning of primitives that make model-based planning easy. To that end, we aim to answer the question: how can we discover skills whose outcomes are easy to predict? We propose an unsupervised learning algorithm, Dynamics-Aware Discovery of Skills (DADS), which simultaneously discovers predictable behaviors and learns their dynamics. Our method can leverage continuous skill spaces, theoretically, allowing us to learn infinitely many behaviors even for high-dimensional state-spaces. We demonstrate that zero-shot planning in the learned latent space significantly outperforms standard MBRL and model-free goal-conditioned RL, can handle sparse-reward tasks, and substantially improves over prior hierarchical RL methods for unsupervised skill discovery.

1 INTRODUCTION

Deep RL often optimizes policies or value functions for a single training task, limiting generalization, while global dynamics models are difficult to learn accurately in complex, high-dimensional systems. DADS addresses this by discovering predictable, diverse skills and learning their dynamics for model-based planning.

  • Global dynamics models are difficult to learn for high-dimensional systems with complex and discontinuous dynamics, and may generalize poorly beyond their training-state distribution.
  • DADS learns low-level skills whose behavior is explicitly optimized for predictability, making predictive models easier to learn in behavior space.
  • DADS discovers skills without supervision through autonomous exploration, rather than requiring task-specific rewards.
  • The method uses mutual-information-based exploration, supports continuous primitive spaces, and learns skill dynamics for downstream model-based planning.

2 PRELIMINARIES

Model-based RL predicts trajectories with a learned global dynamics model, but learning such a model across the entire state space is challenging. The paper instead considers temporal abstraction through skill-conditioned policies and planning in a latent skill space.

  • Model-based RL learns a dynamics model p̂(s′ | s, a) to simulate state trajectories for action sequences without further environment interaction.
  • Learning an effective model for the whole state space remains challenging despite the sample efficiency of model-based methods.
  • The framework assumes continuous state and action spaces and a stochastic Markov decision process during training.
  • At test time, the learned skill dynamics q(s′ | s, z) are used to plan in the skill space Z for a task reward defined in a second MDP.

3 DYNAMICS-AWARE DISCOVERY OF SKILLS (DADS)

DADS maximizes conditional mutual information between skills and next states so skills produce diverse yet predictable transitions. It alternates fitting a skill-dynamics model with policy optimization using an intrinsic predictability-and-diversity reward.

  • DADS maximizes I(s′; z | s), making the current skill informative about the next state given the current state.
  • The mutual-information objective increases transition diversity while reducing conditional next-state entropy under each skill.
  • The skill-conditioned policy π(a | s, z) is optimized so latent skills are maximally informative about transitions s → s′.
  • DADS models skill transitions with qφ(s′ | s, z), which variationally approximates p(s′ | s, z) and enables model-based control.
  • Training alternates between maximizing transition likelihood under qφ and optimizing π with an intrinsic reward based on log qφ(s′ | s, z) − log p(s′ | s).
  • The approximate reward favors transitions that are predictable under qφ and different from transitions produced by prior-sampled skills.
  • Entropy regularization encourages action sequences with similar state transitions to cluster under the same skill and supports robust policies.

4 PLANNING USING SKILL DYNAMICS

At test time, DADS performs model-predictive control in the latent skill space using learned skill-transition dynamics. It simulates candidate skill plans, updates their distribution by reward, and executes the first primitive before replanning.

  • The learned policy and skill-transition model enable zero-shot model-based planning for test-time rewards without downstream-task learning.
  • Unlike conventional planners, DADS samples plans of latent skills rather than action sequences.
  • Figure 3 summarizes iterative plan-distribution updates from simulated rewards followed by execution of the first primitive using π.
  • Each primitive can be held for HZ > 1 steps, yielding an effective horizon H = HP × HZ with fewer planning primitives.
  • The planner simulates trajectories with qφ, evaluates task rewards, and selects the first latent action from the best sampled plan.
  • For continuous skills, plan distributions are represented with normal distributions and refined over multiple sampling updates.

5 RELATED WORK

The paper situates its method among mutual-information skill discovery, intrinsic-motivation exploration, hierarchical reinforcement learning, and model-based planning. Unlike approaches focused on exploration or single-skill control, it learns diverse, predictable skills intended for model-based planning and composition.

  • Mutual-information skill discovery: Mutual-information skill discovery learns diverse behaviors by making skills distinguishable from one another.The paper builds on prior unsupervised model-free reinforcement learning methods using this discriminability objective.
  • Mutual-information skill discovery: DADS distinguishes skills by how they modify the system’s uncontrolled dynamics, encouraging both diversity and predictability.The paper states that predictable skills are more amenable to hierarchical composition and downstream tasks.
  • Intrinsic motivation: Information-theoretic intrinsic-motivation methods typically improve exploration, whereas DADS learns varied skills directly usable for model-based planning.The paper contrasts its objective with approaches aimed at learning a better exploration policy for a single skill.
  • Hierarchical reinforcement learning: DADS skills provide extended actions and temporal abstraction, connecting the method to hierarchical reinforcement learning approaches.The paper frames these properties as enabling more efficient exploration for solving tasks.
  • Model-based planning: The method’s second phase uses learned skill-transition dynamics models for model-based planning, following a broader literature on planning with learned dynamics.The cited model-based reinforcement learning literature includes methods designed for low-data regimes and data-efficiency gains.

6 EXPERIMENTS

DADS discovers diverse, predictable skills without extrinsic rewards and models their dynamics for planning in a learned primitive space. Across MuJoCo and navigation experiments, continuous and low-variance skills support composition and zero-shot control beyond model-based and hierarchical baselines.

  • 6.1 QUALITATIVE ANALYSIS: DADS discovers diverse skills without rewards in high-dimensional MuJoCo environments, including varied gaits, directions, and stable locomotion primitives.The environments include Half-Cheetah, Ant, and Humanoid.
  • 6.1 QUALITATIVE ANALYSIS: DADS skills are predictable and stable, discouraging behaviors such as flipping and favoring sustainable locomotion, especially for unstable Humanoids.Predictability reduces the appeal of trajectories with unreliable outcomes.
  • 6.1 QUALITATIVE ANALYSIS: Continuous skill spaces produce more diverse Ant trajectories and a smooth latent space whose trajectory orientation changes smoothly with skill.The comparison uses a two-dimensional continuous space versus 20 discrete skills.
  • 6.2 SKILL VARIANCE ANALYSIS: DADS yields lower-variance primitives than DIAYN, while adding the x-y prior reduces skill variance further and improves control over generated trajectories.High-variance DIAYN skills have limited utility for hierarchical control.
  • 6.3 MODEL-BASED REINFORCEMENT LEARNING: Zero-shot MPPI planning with DADS primitives significantly outperforms model-based RL baselines, and continuous primitives outperform discrete primitives.The baselines include random, weak-oracle, and strong-oracle model-based RL variants.
  • 6.4 HIERARCHICAL CONTROL WITH UNSUPERVISED PRIMITIVES: A meta-controller composes DADS skills to navigate Ant to different goals, whereas the same controller cannot compose DIAYN skills.The result supports the link between lower skill variance and temporal compositionality.
  • 6.5 GOAL-CONDITIONED RL: For goal navigation, DADS planning degrades smoothly with greater goal distance and remains effective for sparse rewards, unlike goal-conditioned RL outside its training distribution.With sparse rewards, goal-conditioned RL is unable to navigate, while MPPI performs comparably to dense-reward planning up to about 20 metres.

7 CONCLUSION

The authors present DADS as an unsupervised skill-learning method for hierarchical control with model-based planning. It discovers diverse, low-variance skills and composes them to outperform competitive model-based baselines on downstream tasks without task-specific training.

  • DADS is an unsupervised skill-learning algorithm designed to support model-based planning for hierarchical downstream control.
  • The method scales to high-dimensional state-spaces while discovering a diverse set of low-variance skills.
  • Without training on the specified task, learned skills can be composed to outperform competitive model-based baselines trained with test-task knowledge.

A IMPLEMENTATION DETAILS

The models are implemented with TensorFlow Agents, using TensorFlow as the underlying framework.

  • The models are implemented in open-source TensorFlow Agents based on TensorFlow.

A.1 SKILL SPACES

DADS supports discrete and continuous skill spaces, with skills sampled at episode start and conditioned policies and dynamics models used to represent their behavior.

  • Discrete skills use one-hot vectors sampled from a uniform prior, with experiments using up to 128 skills.For MuJoCo Ant, the discrete-skill experiments use D = 20.
  • Continuous skills sample z from Uniform(−1, 1)^D, with latent dimensions ranging from 2 to 5 across Ant and Humanoid settings.The skills are sampled once at the beginning of each episode and fixed thereafter.
  • The policy uses SAC and generally excludes global center-of-mass coordinates so learned skills are agnostic to agent location.The policy and critic use two hidden layers, although larger networks can improve skill expressivity.
  • Skill dynamics are modeled as q(Δs | s, z), predicting state changes rather than full successor states for easier learning.Global coordinates are excluded from the input, while Δx and Δy can still be predicted.
  • The dynamics output uses a four-expert mixture of Gaussian distributions whose parameters are predicted from the state and skill.
  • Training uses shorter horizons for Ant and longer horizons for Humanoid, collecting 2000 and 4000 steps per iteration, respectively.Longer Humanoid episodes help filter unstable skills, while longer Ant episodes add no value.

A.5 PLANNING AND EVALUATION SETUPS

Evaluation fixes a 200-step horizon while varying planning samples with latent-space and planning-horizon settings. The appendix frames DADS as matching trajectory and primitive dynamics distributions through an information-bottleneck objective.

  • Planning and evaluation: Evaluation uses a 200-step horizon for all models, with 10–200 planning-distribution samples depending on latent-space and planning-horizon sizes.For HP = 1, HZ = 10, and a 2D latent space, the setup uses 50 samples and sets MPPI’s coefficient γ to 10.
  • Planning and evaluation: Hierarchical controllers use PPO for discrete skills and SAC for continuous skills, selecting meta-actions every 10 steps.The meta-controller uses hidden layers of size (128, 128).
  • Planning and evaluation: The PETS baseline uses an ensemble of 3 neural networks, a planning horizon of 20, and MPPI because it outperformed CEM in the experiments.
  • Graphical-model formulation: The desired world model approximates p(s′ | s, z), enabling model-based planning in behavior space without requiring exact actions.
  • Graphical-model formulation: DADS optimizes the policy distribution in the trajectory model to reduce its KL divergence from the desired primitive-dynamics model.The formulation uses a reverse information projection and compares multi-information terms across the two graphical models.
  • Graphical-model formulation: The resulting information-bottleneck objective maximizes I(s_t+1; z | s_t) while entropy regularization corresponds to minimizing I(a_t; {s_t, z}).Setting δ_t = 0 neglects the unchangeable information of the fixed, unknown underlying dynamics.

C APPROXIMATING THE REWARD FUNCTION

The section develops tractable approximations for the reward objective despite intractable transition marginals and posteriors. The chosen approximation samples skills from the prior, while alternative variational approaches introduce improper lower bounds or weaken the skill interpretation.

  • Chosen approximation: The reward approximation uses qφ(s′ | s, z)p(z) to estimate the intractable marginal transition distribution p(s′ | s).Sampling from p(z) is inexpensive and requires only a forward pass through qφ; the approximation is considered reasonable for some locomotion problems.
  • Alternative approximations: Approximating p(z | s) with qβ(z | s) may improve sampling for some problems, but still yields an improper lower bound for R(π | qφ).The passage presents this as an alternative whose approximation quality may be better than sampling from the prior in some settings.
  • Variational derivation: The derivation introduces variational posteriors qα(z | s′, s) and qφ(s′ | s, z), then further decomposes the remaining entropy terms.The action posterior can be chosen uniformly over the bounded action space, producing an entropy-regularized-RL analogy under deterministic dynamics.
  • Alternative approximations: Variational approximations to p(z | s) make the mutual-information lower bound improper because p(z | s) is intractable.The issue remains despite the mathematical machinery and additional assumptions used to bypass the intractability of p(s′ | s).
  • Alternative approximations: Resampling z from p(z) at every timestep makes p(z | s) equal to p(z), but weakens the interpretation of z as a temporally extended skill.Empirically, this modification significantly throttled learning and did not yield useful or interpretable skills.

D INTERPOLATION IN CONTINUOUS LATENT SPACE

DADS supports continuous latent-space interpolation: interpolating between skill vectors corresponds to interpolating between Ant trajectories. The learned skill dynamics also provide robust long-horizon state predictions.

  • Continuous latent space: Interpolating between skill vectors in DADS’s continuous primitive space produces corresponding interpolation in trajectory space.The figure shows three interpolations from z = [1.0, 1.0] to alternative latent vectors.
  • Model prediction: Skill-dynamics prediction error rises more slowly with horizon than the norm of actual position for the Ant x−y prior.The passage identifies this as evidence of cooperation between the learned primitives and skill-dynamics.
  • Model prediction: Skill-dynamics provide robust state predictions over long planning horizons.Figure 14 compares actual colored trajectories with dotted-black trajectories predicted by skill-dynamics and reports normalized coordinate prediction error.
Loading 1907.01657v2…