PaperlayerView exact revision on arXiv

Source-linked AI summary

Diversity is All You Need: Learning Skills without a Reward Function

Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, Sergey Levine

arXiv:1802.06070v6cs.AIcs.RO

TL;DR

DIAYN addresses how reinforcement-learning agents can acquire useful skills without a reward function. It learns latent-conditioned skills by maximizing mutual information between skills and states with maximum-entropy policies, and reports diverse behaviors, benchmark-solving skills, faster downstream learning, and hierarchical-task performance.

  • Problem

    Acquiring useful skills autonomously without any reward signal is challenging, yet such skills could support exploration, long-horizon tasks, and settings where reward evaluation requires supervision.

  • Method

    DIAYN learns latent-conditioned policies by maximizing mutual information between skills and states while using maximum-entropy reinforcement learning.

  • Results

    DIAYN learns diverse behaviors and often solves benchmark tasks with a learned skill without receiving the true task reward, while pretrained skills accelerate downstream learning and support hierarchical reinforcement learning.

  • Takeaways & Limitations

    Unsupervised skill discovery can provide reusable skills for rapid task adaptation, hierarchical reinforcement learning, and imitation learning.

Abstract

from arXiv · show

Intelligent creatures can explore their environments and learn useful skills without supervision. In this paper, we propose DIAYN ('Diversity is All You Need'), a method for learning useful skills without a reward function. Our proposed method learns skills by maximizing an information theoretic objective using a maximum entropy policy. On a variety of simulated robotic tasks, we show that this simple objective results in the unsupervised emergence of diverse skills, such as walking and jumping. In a number of reinforcement learning benchmark environments, our method is able to learn a skill that solves the benchmark task despite never receiving the true task reward. We show how pretrained skills can provide a good parameter initialization for downstream tasks, and can be composed hierarchically to solve complex, sparse reward tasks. Our results suggest that unsupervised discovery of skills can serve as an effective pretraining mechanism for overcoming challenges of exploration and data efficiency in reinforcement learning.

1 INTRODUCTION

DIAYN addresses the challenge of acquiring useful skills without reward supervision by maximizing an information-theoretic objective. The learned skills can emerge as diverse behaviors, solve benchmark tasks without true task rewards, and support downstream reinforcement learning.

  • Motivation: Unsupervised skills may improve exploration in sparse-reward environments and provide primitives for long-horizon hierarchical reinforcement learning.They may also reduce supervision when reward evaluation requires human feedback.
  • Motivation: DIAYN targets useful skill acquisition when the reward function is unknown or unavailable.The method seeks skills that are individually distinct and collectively explore large parts of the state space.
  • Approach: The method maximizes an information-theoretic discriminability objective with a maximum-entropy policy.Its central idea is to make skills distinguishable while encouraging broad behavioral coverage.
  • Results: DIAYN produces diverse behaviors such as running, jumping, and backflips in simulated robotic tasks.The paper reports behaviors including forward running, backflips, skipping backward, and face flops.
  • Results: In several reinforcement learning benchmarks, learned skills solve the task despite never receiving the true task reward.The paper also reports downstream use through hierarchical reinforcement learning, rapid adaptation, and imitation learning.

2 RELATED WORK

DIAYN connects skill discovery to information theory and diversity maximization while targeting low-supervision skill acquisition for downstream reinforcement learning. Its distinction is a general information-theoretic objective that avoids manually designed distance metrics and supports diverse policies rather than a single intrinsically motivated policy.

  • Hierarchical reinforcement learning: Prior hierarchical reinforcement learning commonly learned skills and a meta-controller to maximize a single known reward function.Joint training can leave options that the meta-policy does not select without reward signals for improvement.
  • Hierarchical reinforcement learning: DIAYN uses a random meta-policy during unsupervised learning so all skills receive learning signals.The paper presents this as a way to prevent the degeneracy associated with unselected options.
  • Information theory: Information-theoretic prior work includes maximum-entropy reinforcement learning, empowerment, and discriminability objectives related to mutual information.DIAYN maximizes mutual information between states and skills, interpreting this as empowerment for a hierarchical agent whose actions are skills.
  • Diversity maximization: Diversity-maximization work often seeks better solutions, whereas DIAYN seeks complex skills with minimal supervision for efficiency, imitation learning, and hierarchical reinforcement learning.The paper aims to reduce objective-function queries rather than manually optimize a task solution.
  • Intrinsic motivation: Unlike intrinsic-motivation methods that learn a single policy, DIAYN learns many diverse policies.The paper also claims scalability to more complex tasks through algorithmic design choices.

3 DIVERSITY IS ALL YOU NEED

DIAYN learns task-independent skills during an unsupervised exploration stage, then uses them to facilitate supervised task learning. It conditions policies on latent skills, distinguishes them through visited states, and combines information maximization with entropy-based exploration.

  • Unsupervised skill discovery: DIAYN separates unsupervised skill discovery from a later supervised stage that maximizes task reward.Because the skills are learned without prior task knowledge, they can be reused across different tasks.
  • Design principles: Each skill is a latent-conditioned policy intended to control the states visited by the agent.Different skills should visit different, distinguishable states, while states rather than invisible actions identify behavior.
  • Objective: The objective maximizes mutual information between skills and states, minimizes conditional mutual information between skills and actions given states, and maximizes mixture-policy entropy.These terms encourage state control, state-based discrimination, and exploratory action selection.
  • Objective: F(θ) = I(S; Z) + H[A | S] − I(A; Z | S) combines skill-state dependence, mixture entropy, and a penalty for action-based discrimination.The equivalent form exposes the entropy of the skill prior, state-based inference, and per-skill action entropy.
  • Implementation: DIAYN implements the objective with Soft Actor-Critic and a policy conditioned on a latent skill variable.Skills are sampled at episode start, and the discriminator is trained to infer each skill from visited states.
  • Stability: On gridworlds, the unique analytical optimum evenly partitions states among skills, while empirical results show robustness to random seeds.The paper notes that convergence guarantees in continuous approximate settings remain difficult.

4 EXPERIMENTS

DIAYN learns diverse, task-agnostic skills without reward signals and applies them to downstream adaptation, hierarchical control, and imitation. Experiments show diverse locomotion and navigation behaviors, faster policy learning, and improved performance on sparse-reward tasks.

  • What skills does DIAYN learn?: DIAYN learns diverse skills across tasks from 2 DOF point navigation to 111 DOF ant locomotion, including running, jumping, flipping, and curved walking.The method discovers multiple distinct task-solving skills without rewards and produces diverse primitive behaviors in continuous-control environments.
  • Training dynamics and state overlap: Skills become increasingly diverse during training, while state-based discrimination still permits skills to overlap when they eventually become distinguishable.In the hallway experiment, skills leave initially unrewarded regions to reach states that make them mutually distinguishable.
  • Why use a fixed prior?: A fixed skill prior lets DIAYN continue sampling all skills, whereas VIC converges to sampling only a handful and discovers less diverse skills.The comparison measures effective skill count as exp(H[Z]) throughout training.
  • Accelerating learning with policy initialization: Initializing policy and value networks with DIAYN-pretrained weights accelerates downstream learning compared with random initialization.The comparison uses half cheetah, hopper, and ant, assumes unsupervised pretraining is free, and averages results across five random seeds.
  • Using skills for hierarchical RL: A meta-controller that composes DIAYN skills outperforms competitive baselines on challenging hierarchical tasks, including sparse-reward navigation.The meta-controller chooses which skill to execute for a fixed number of steps, enabling cheetah hurdle jumping and ant navigation.
  • Imitating an expert: DIAYN qualitatively imitates three of four half-cheetah expert behaviors, succeeding on standing upright, flipping, and faceplanting but failing on handstands.The expert trajectories are synthetically generated using a different random seed, ensuring they are not produced by the currently available skills.

5 CONCLUSION

DIAYN learns diverse skills without reward functions and uses them for adaptation, hierarchical reinforcement learning, and imitation. The authors suggest that learned skills can simplify complex action spaces and support future combinations with other methods.

  • DIAYN learns diverse skills for complex tasks, often solving benchmark tasks with one learned skill without receiving task reward.
  • The method supports rapid adaptation to new tasks, hierarchical reinforcement learning, and expert imitation.
  • DIAYN may make task learning easier by replacing a complex action space with a set of useful skills.
  • The pseudo-reward baseline keeps rewards non-negative and encourages the agent to stay alive, except in environments where ending quickly is desirable.

B OPTIMUM FOR GRIDWORLDS

The gridworld analysis gives analytic optima for DIAYN and explains how entropy and discriminability shape preferred state partitions. DIAYN favors partitions with short borders and bottleneck states, while some environments favor removing the survival-oriented baseline.

  • For an N × N gridworld, the DIAYN objective admits an analytic solution under up/down/left/right movement dynamics.Agents remain in place when attempting to move outside the gridworld.
  • With no entropy regularization, two skills that partition the state space evenly maximize the un-regularized DIAYN objective.The partition makes H[Z | S] = 0 while the uniform skill prior maximizes H[Z] = log(2).
  • The log p(z) baseline can be removed in environments such as mountain car where ending the episode quickly is desirable.
  • With regularization, an even partition is within O(1/N) of the optimum, and its distance from optimal decreases as the gridworld grows.
  • The entropy and discriminability terms conflict at skill borders, favoring partitions with short borders and bottleneck states.For dimensions N < M, splitting along the first dimension yields a better objective than splitting along the second.

C EXPERIMENTAL DETAILS

The experiments use standard continuous-control environments and modified hurdle and navigation tasks. In 2D navigation, DIAYN skills covered the state space sufficiently for the hierarchical policy to choose one skill and complete the task.

  • Experiments use HalfCheetah-v1, Ant-v1, Hopper-v1, MountainCarContinuous-v0, and InvertedPendulum-v1.
  • The cheetah hurdle environment adds regularly spaced boxes to HalfCheetah-v1 to create an obstacle-avoidance task.The boxes have height 0.25m, width 0.1m, and depth 1.0m.
  • The ant navigation environment modifies Ant-v1 with four corner waypoints and sparse rewards for reaching them in sequence.Each waypoint reward is +1, with a maximum possible reward of +5.
  • In 2D navigation, DIAYN skills provided sufficient state-space coverage for the hierarchical policy to complete the task by selecting a single skill.

D.1 TRAINING OBJECTIVES

DIAYN’s training objective balances entropy and skill discriminability, producing increasingly diverse skills without collapsing to deterministic policies. The method’s behavior is stable across random seeds, while entropy regularization controls the exploration–discriminability trade-off.

  • The discriminability term continues increasing while the entropy regularizer plateaus, indicating increasingly diverse skills without deterministic-policy collapse.The plot reports means and standard deviations across 5 seeds while learning 20 skills in half cheetah.
  • Repeating the experiment across 5 random seeds shows little effect on DIAYN’s training dynamics.
  • Increasing entropy regularization makes skills visit more diverse states, but excessive regularization makes skills difficult to discriminate.With small α, skills move far in different directions but explore less of the state space.

D.3 DISTRIBUTION OVER TASK REWARD

DIAYN skills learned without task rewards exhibit diverse task-reward outcomes and can perform well on multiple benchmark objectives. Learning the skill distribution instead of fixing it substantially reduces the effective number of skills.

  • A wide distribution of benchmark rewards demonstrates diverse learned behaviors, including hopping, standing, forward hopping, backward diving, and immediate falling.For hopper, some skills receive at least 1000 reward, while others receive between 100 and 1000 or fall immediately.
  • DIAYN learns forward-moving half-cheetah and hopper skills that achieve large benchmark task rewards despite never receiving reward during training.The same environments also produce backward-moving skills with much lower task reward.
  • Across running, jumping, and distance-from-origin rewards, DIAYN learns some high-performing skills, whereas VIME performs poorly on all three tasks.
  • Learning p(z) rather than fixing it reduces the effective number of sampled skills, with the reduction reaching 10x in the reported environments.The comparison evaluates half cheetah, inverted pendulum, and mountain car.

E.2 EFFECT OF LEARNING p(z)

The effective number of skills quantifies how broadly the learned skill distribution is used. Learning p(z) reduces this number substantially compared with a fixed uniform distribution.

  • With fixed p(z), its entropy remains constant at log(50) ≈3.9, whereas learning p(z) allows the distribution’s effective skill count to decline.
  • Learning p(z) reduces the effective number of skills by a factor of 10x in the reported environments.The effective number is obtained by exponentiating the entropy of p(z).

F.1 CLASSIC CONTROL TASKS

Without rewards, DIAYN discovers multiple distinct solutions and varied behaviors in classic-control and locomotion environments. These skills can also support imitation by selecting a skill whose state distribution matches an expert trajectory.

  • F.1 CLASSIC CONTROL TASKS: DIAYN learns multiple distinct reward-free skills that solve inverted pendulum and mountain car tasks.Inverted-pendulum skills balance at different positions, while mountain-car skills use varied reversal timing and velocity strategies.
  • F.1 CLASSIC CONTROL TASKS: Reward-free locomotion skills include diverse behaviors in half cheetah, hopper, and ant.Ant skills move in multiple directions, flip onto its back, and rotate in place.
  • G IMITATION LEARNING: Across 600 imitation tasks, DIAYN more closely matches the expert than all baselines.
  • G IMITATION LEARNING: For imitation, the learned discriminator estimates which skill most likely generated an expert trajectory.The retrieved skill is motivated as an M-projection of the expert state distribution onto the family of skill-induced state distributions.

G.1 IMITATION LEARNING EXPERIMENTS

The imitation-learning experiments synthetically generate “expert” trajectories using a different random seed, ensuring they are not produced by currently available skills. The results also provide evidence that maximum-entropy policies and a fixed p(z) distribution enabled scaling to more complex tasks.

  • Expert trajectories are synthetically generated by running the algorithm with a different random seed.The seed differs from those used to produce the currently available skills.
  • A different seed ensures the synthetic trajectories are not produced by any currently available skills.
  • Maximum-entropy policies and fixing the p(z) distribution are identified as factors enabling the method to scale to more complex tasks.
Loading 1802.06070v6…