Source-linked AI summary
Variational Option Discovery Algorithms
Joshua Achiam, Harrison Edwards, Dario Amodei, Pieter Abbeel
TL;DR
Variational option discovery asks how agents can learn diverse, reward-free skills whose meanings are discovered during training. The paper connects these methods to variational autoencoders, introduces VALOR and a decoder-driven curriculum, and reports that the curriculum stabilizes learning and enables many more behavioral modes, while information-theoretic objectives can produce trivial behaviors in complex environments.
Problem
The paper studies how agents can discover diverse options without task rewards while assigning context meanings through training rather than predefined reward-grounded instructions.
Method
The paper derives VALOR from the variational autoencoder connection, using policies to encode contexts into trajectories and decoders to recover contexts, then increases contexts through curriculum learning.
Results
The curriculum stabilizes and speeds learning for VALOR, VIC, and DIAYN and can allow a single agent to learn up to hundreds of behavioral modes.
Takeaways & Limitations
Variational option discovery produces interesting behaviors across several environments, but learned hierarchies were not shown to outperform task-specific policies learned from scratch.
Takeaways & Limitations
Purely information-theoretic objectives can produce trivial context-encoding behaviors in complex environments rather than differences aligned with human intuition.
Abstract
from arXiv · showhide
We explore methods for option discovery based on variational inference and make two algorithmic contributions. First: we highlight a tight connection between variational option discovery methods and variational autoencoders, and introduce Variational Autoencoding Learning of Options by Reinforcement (VALOR), a new method derived from the connection. In VALOR, the policy encodes contexts from a noise distribution into trajectories, and the decoder recovers the contexts from the complete trajectories. Second: we propose a curriculum learning approach where the number of contexts seen by the agent increases whenever the agent's performance is strong enough (as measured by the decoder) on the current set of contexts. We show that this simple trick stabilizes training for VALOR and prior variational option discovery methods, allowing a single agent to learn many more modes of behavior than it could with a fixed context distribution. Finally, we investigate other topics related to variational option discovery, including fundamental limitations of the general approach and the applicability of learned options to downstream tasks.
1 Introduction
The paper frames reward-free option discovery as learning diverse skills without task-specific rewards, then introduces VALOR and a decoder-driven curriculum as its two main contributions.
- Reward-free option discovery learns skills through environmental interaction without maximizing cumulative reward for a particular task.
- The paper connects variational option discovery to variational autoencoders, with policies encoding contexts into trajectories and decoders recovering contexts.Contexts acquire their meanings through training, with each context corresponding to a distinct option.
- The paper presents VALOR and curriculum learning as its two main algorithmic contributions.
- VALOR decodes complete trajectories to encourage dynamical modes such as moving in a circle rather than goal-attaining modes such as reaching X.
- The curriculum increases the number of contexts when decoder-measured performance is strong enough on the current context set.
2 Related Work
Related work situates variational option discovery among option learning, universal policies, intrinsic motivation, variational autoencoders, and novelty search.
- Option Discovery: Prior option-discovery work includes policy-gradient methods that learn options during task solving or from demonstrations for task-specific acceleration.
- Universal Policies: Universal-policy approaches use extrinsic rewards and hand-crafted instruction spaces, whereas variational option discovery is unsupervised and discovers its own instruction space.
- Intrinsic Motivation: The paper distinguishes intrinsic-motivation methods from universal-policy learning and identifies knowledge fade when states cease to be interesting to intrinsic reward.
- Variational Autoencoders: Variational autoencoders learn probabilistic encoders and decoders by optimizing the evidence lower bound under a latent-variable prior.
- Novelty Search: Diversity-based option discovery is related in spirit to novelty search, which seeks behavior diversity using a characterization function.
3 Variational Option Discovery Algorithms
Variational option discovery treats contexts as latent skill identifiers encoded into trajectories and recovered by a decoder, connecting the framework to β-VAEs. VALOR extends this view with trajectory decoding, while curriculum learning increases the context set as decoder performance becomes strong enough.
- Variational Option Discovery: The policy samples a fixed episode context and generates a trajectory whose behavior should uniquely identify that context without reward-defined semantics.The decoder is trained from known context–trajectory pairs, while policy reinforcement favors trajectories that make contexts distinguishable.
- Connections to Prior Work: Variational option discovery has a one-to-one correspondence with the β-VAE objective, mapping contexts to data, trajectories to latent representations, and policy–environment dynamics to an encoder.The decoder corresponds to the VAE decoder, and policy entropy corresponds to KL regularization against a prior induced by a uniform random policy.
- Connections to Prior Work: VIC maximizes a variational mutual-information bound between contexts and terminal behavior, while DIAYN relates to the general objective through per-state decoding with a fixed context distribution.The comparison treats VIC in its canonical fixed, state-independent form with final-state decoding; DIAYN removes the constant context-prior term when its distribution is fixed.
- VALOR: VALOR directly optimizes the autoencoding objective and decodes contexts from complete trajectories rather than only final states or per-timestep decompositions.Its decoder omits actions, uses a bidirectional LSTM, and processes trajectory observations to distinguish behaviors that may share states or state sets reached in different orders.
- Curriculum Approach: Curriculum learning starts with a small number of contexts and increases that number when decoder-measured performance is sufficiently strong, producing faster and more stable convergence.The number of contexts is bounded by a hyperparameter Kmax.
4 Experimental Setup
The experiments compare VALOR, VIC, and DIAYN, with and without curriculum learning, across locomotion and additional simulated-robot environments. They also examine context embeddings and the downstream usefulness of learned behaviors.
- Experimental Questions: The study asks whether curriculum learning improves variational option discovery training, how distinct the learned behaviors are, and whether they support downstream control.These questions are evaluated across VALOR, VIC, and DIAYN.
- Test Environments: Core comparisons use a custom 2D point agent, HalfCheetah, Swimmer, and a customized Ant environment without contact forces in observations.Additional experiments use a dextrous hand, the toddler humanoid environment, and a modified Ant-Maze downstream-control setting.
- Implementation and Training: All three methods are implemented with vanilla policy gradient and recurrent neural-network policy architectures, differing from the original RL algorithms used for VIC and DIAYN.The paper also investigates curriculum generation and learned context embeddings as training techniques.
5 Results
The experiments show that embeddings and curriculum learning improve VALOR training, while variational option discovery can produce many distinct behaviors but faces challenges in complex environments and downstream-task transfer.
- Context embeddings consistently improve the speed and stability of VALOR training, while uniform context distributions become harder to optimize as K increases.
- Curriculum learning with embeddings enables substantially faster mastery of Kmax = 64 contexts than uniform context distributions, with results consistent across algorithms.
- The comparison evaluates VALOR, VIC, and DIAYN using behavior scores for Cheetah, Swimmer, Ant, and Ant3 across K = 64 contexts.
- The curriculum approach does not consistently change behavioral diversity, but it makes score distributions more consistent across random seeds.
- Hand was comparatively easy, whereas Toddler remained difficult: after 15000 iterations, only K = 40 behaviors had been learned, and the behaviors were extremely unnatural.
- The Point experiments produced hundreds of decoder-distinct behaviors, although these may represent increasingly fine-grained divisions of already-learned behaviors.
- Interpolating context embeddings produced some reasonably smooth behavioral interpolations, suggesting that VALOR learns general-purpose universal policies.
- The study found no clear evidence that hierarchies using variationally discovered options outperform task-specific policies learned from scratch.
6 Conclusions
The empirical examination found that variational option discovery produces varied behaviors and evidence of policy universality, but struggles in very high-dimensional control and can encode contexts trivially. Evidence for the usefulness of hierarchies built on these methods remained unclear.
- Variational option discovery produced interesting behaviors across Cheetah, Ant, and Hand, but struggled in the very high-dimensional Toddler environment.
- Mode interpolation and hierarchy experiments provided evidence that learned policies are universal in meaningful ways.
- The experiments found no clear evidence that hierarchies built on variational options provide the investigated benefits.
- Purely information-theoretic objectives can produce trivial context encoding, such as tiling a narrow state-space volume with contexts.
- Future methods should make decoders distinguish trajectories according to meaningful behavioral differences aligned with human intuition.
A VAE-Equivalence Proof
The objective combines entropy regularization with a second term that is constant for a uniform random policy and fixed episode length, so that term can be removed without changing optimization.
- The objective’s first term provides entropy regularization.
- For a uniform random policy and fixed episode length, the second term is independent of the policy and can be removed.
B Implementation Details
The experiments optimize all three variational option discovery algorithms with vanilla policy gradient, using shared training settings alongside algorithm- and environment-specific details.
- Vanilla policy gradient optimizes the reinforcement objective for VALOR, VIC, and DIAYN.Policy optimization details vary slightly between algorithms and environments.
- VALOR and VIC use one advantage-function formulation, while DIAYN uses a separate formulation.
- The value baseline Vψ(st, c) is learned with one gradient descent step per iteration.
- The entropy-gradient calculation ignores the policy’s role in the trajectory distribution.This approximation yields the same entropy regularization used by Mnih et al. [2016].
- Shared settings include 1000 paths per epoch, γ = 0.97, learning rate 1e−3, and an LSTM(64)-MLP(32) policy.The entropy coefficient is β = 1e−3 where applicable, and context embeddings have dimension 32.
- VALOR uses a bidirectional LSTM decoder, whereas VIC and DIAYN use MLP decoders of size (180, 180).The VALOR decoder has cell size 64 in each direction.
C Additional Analysis for Best Practices
Figure 6 examines context embeddings and curriculum learning in HalfCheetah, tracking decoder performance and the curriculum’s changing context count over training.
- The analysis compares learning with and without context embeddings for K ∈{8, 16, 32, 64}.Learning curves are shown using E[log PD(c|τ)].
- Curriculum learning with Kmax = 64 is compared against uniform sampling with K = 64.Curriculum spikes and dips occur when K changes according to Eq. 5.
- The curriculum and uniform curves are directly comparable only after curriculum reaches K = Kmax.The dashed red line marks that point.
- The curriculum’s current K is plotted throughout the runs, with all curves averaged over three random seeds.
D Complete Experimental Results for Comparison Study
The comparison evaluates VALOR, VIC, and DIAYN with uniform or curriculum context distributions across several robotics environments using optimization curves, behavior scores, and trajectory traces.
- The study compares VALOR, VIC, and DIAYN under uniform and curriculum distributions across point mass, cheetah, swimmer, and ant environments.Because behavior discovery is unsupervised, the analysis uses multiple measures rather than one quantitative metric.
- Behavior scores include Cheetah final x-coordinate, Swimmer and Ant final distance from origin, and Ant z-axis rotations.Scores use T = 1000 test trajectories although training uses T = 250.
- Bar charts show behavior IDs against log-scale scores, while histograms show score bins against log-scale behavior counts.Black bars represent score standard deviations; composite charts combine seeds 0, 10, and 20.
- Each single seed corresponds to one policy with K = 64 behaviors.
- Point traces use x and y ranges [−1.3, 1.3], while Ant traces use [−15, 15].Point traces use T = 65, and Ant traces use T = 1000.
- Figure 7 reports that curriculum generally speeds and stabilizes learning, except for DIAYN and VIC in the point environment.
- Random-reward experiments use the same core-comparison settings except that learned decoder rewards are replaced by random rewards.The experiments include a matched comparison and a K = 1 condition.
E.1 Results from RR1
Random-reward training produced inconsistent or uninteresting behavior across the evaluated locomotion environments. Cheetah showed opposing running modes, while Swimmer and Ant exhibited little motion or high variability, and RR1 did not outperform RR2.
- Cheetah: Cheetah results were nearly bimodal across seeds, with forward-running and backward-running behaviors emerging.The composite results appeared reasonable, but individual seeds were weak and inconsistent.
- Swimmer: Swimmer showed almost no motion under random rewards.
- Ant: Ant behaviors varied extremely across seeds and were inconsistent with the evaluated score functions.The reported standard deviations were very large for every behavior.
- RR1 versus RR2: RR1's multimodal policies did not produce significantly better learned behaviors than RR2's unimodal policies.Using one random reward function instead of several did not yield useful or consistent behavior under the score functions.
- Conclusion: Random rewards based on Eq. 6 did not result in interesting behavior in the environments considered.The authors note that another functional form for random rewards might perform better.