Source-linked AI summary
Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings
John D. Co-Reyes, YuXuan Liu, Abhishek Gupta, Benjamin Eysenbach, Pieter Abbeel, Sergey Levine
TL;DR
Long-horizon hierarchical reinforcement learning with delayed or sparse rewards requires effective reusable skills and higher-level reasoning. SeCTAR learns continuous trajectory representations with mutually consistent policy and predictive decoders, then plans in the learned latent space; experiments show strong performance across long-horizon, sparse-reward, and multi-step tasks. Its scope assumes access to an evaluable reward function and relies on updating the trajectory distribution through exploration.
Problem
Long-horizon reinforcement learning with delayed or sparse rewards needs reusable temporally extended skills and effective higher-level reasoning.
Method
SeCTAR learns continuous trajectory latents with a policy decoder that executes skills and a state decoder that predicts their outcomes, enabling model-based planning over latent skills.
Results
SeCTAR outperforms several prior and flat reinforcement learning methods on tasks requiring long-horizon reasoning, sparse rewards, and multi-step compound skills.
Takeaways & Limitations
Trajectory-level representations and outcome prediction support hierarchical control without additional reinforcement learning for a high-level policy.
Takeaways & Limitations
The approach assumes access to a reward function evaluable on arbitrary states and requires periodically updating its training trajectory distribution through exploration.
Abstract
from arXiv · showhide
In this work, we take a representation learning perspective on hierarchical reinforcement learning, where the problem of learning lower layers in a hierarchy is transformed into the problem of learning trajectory-level generative models. We show that we can learn continuous latent representations of trajectories, which are effective in solving temporally extended and multi-stage problems. Our proposed model, SeCTAR, draws inspiration from variational autoencoders, and learns latent representations of trajectories. A key component of this method is to learn both a latent-conditioned policy and a latent-conditioned model which are consistent with each other. Given the same latent, the policy generates a trajectory which should match the trajectory predicted by the model. This model provides a built-in prediction mechanism, by predicting the outcome of closed loop policy behavior. We propose a novel algorithm for performing hierarchical RL with this model, combining model-based planning in the learned latent space with an unsupervised exploration objective. We show that our model is effective at reasoning over long horizons with sparse rewards for several simulated tasks, outperforming standard reinforcement learning methods and prior methods for hierarchical reasoning, model-based planning, and exploration.
1. Introduction
SeCTAR addresses long-horizon, sparse-reward reinforcement learning by learning reusable continuous trajectory skills and using their predicted outcomes for higher-level planning. Its hybrid approach combines model-free skill acquisition with model-based control and outperforms several prior methods across delayed and sparse-reward tasks.
- Motivation: Long-horizon tasks with extremely delayed or sparse rewards challenge standard reinforcement learning and motivate reusable temporally extended lower-level skills.Such skills let higher-level policies reason at a coarser temporal abstraction and support multiple tasks.
- Motivation: Prior approaches rely on hand-specified objectives or top-down training, restricting generality, exploration, optimization, or skill reuse.Top-down methods may provide solutions tailored to a single task.
- Approach: SeCTAR learns continuous trajectory representations bottom-up without requiring hand-specified subgoals or objectives.The method replaces a discrete skill set with a continuous latent space of behaviors.
- Approach: Its latent-conditioned policy generates temporally extended behaviors while a latent-conditioned model predicts their outcomes, enabling model-based higher-level control.Low-level behaviors are trained model-free, while higher-level behavior uses model-based planning over closed-loop skill outcomes.
- Results: SeCTAR performs delayed and sparse-reward tasks, including object interaction and waypoint navigation, while outperforming TRPO, VIME, FeUdal Networks, and option critic.The evaluation covers hierarchical, exploration-driven, and standard reinforcement learning baselines.
2. Background
This section frames reinforcement learning as maximizing discounted rewards in environments with unknown dynamics, especially when rewards are delayed over long horizons. It then introduces variational inference and variational autoencoders as the representation-learning background.
- Reinforcement learning seeks to maximize the expected discounted sum of rewards.
- The policy maps states to action distributions, while environment transitions follow unknown dynamics and rewards are defined over states and actions.
- The targeted reinforcement learning problems involve long horizons and delayed rewards, assuming the reward function can be evaluated on arbitrary states.
- Variational inference uses a tractable proxy q(z | x) to estimate the intractable posterior p(z | x).
- The evidence lower bound combines expected reconstruction quality with a KL-divergence penalty, and variational autoencoders optimize this bound.
3. Self-Consistent Trajectory Autoencoder
SeCTAR learns continuous latent representations of trajectory segments with paired state and policy decoders that are trained to remain consistent. These representations support latent-space hierarchical planning, while iterative entropy-based exploration supplies increasingly useful training trajectories.
- Trajectory representations: SeCTAR represents trajectory segments with continuous latent variables, enabling planning over extended state sequences rather than individual states and actions.The model targets long-horizon tasks by composing shorter trajectories within complete episodes.
- Trajectory representations: The model uses a trajectory-level VAE with a recurrent state decoder that predicts state sequences and a policy decoder that generates behavior through environmental execution.An encoder maps trajectories to a latent distribution, after which the latent is decoded both directly into states and through a conditioned policy.
- Self-consistency: Joint training enforces consistency between predicted state-sequence distributions and trajectories generated by the latent-conditioned policy.The policy decoder is trained by penalizing KL divergence between the policy- and state-decoder trajectory distributions, alongside likelihood-based VAE training.
- Hierarchical control: A meta-controller selects latent variables sequentially, executes each conditioned policy for T steps, and uses model-predictive control to plan over the effective horizon H/T.The state decoder supplies predicted trajectory segments and rewards for latent-space planning.
- Exploration: Because fixed trajectory data limits exploration in long-horizon tasks, SeCTAR alternates model fitting with an explorer policy that targets relevant regions and diverse behaviors.The explorer maximizes entropy of the marginal trajectory distribution rather than only the conditional action entropy of the policy.
4. Related Work
SeCTAR learns continuous, unsupervised trajectory-based skills rather than relying on fixed options, hand-engineered subgoals, or proxy rewards. Its trajectory-level entropy objective provides a distinct exploration mechanism for hierarchy induction.
- SeCTAR learns a continuous set of skills without supervision by representing trajectories and optimizing entropy over trajectory distributions.
- Prior options methods typically use a finite, predetermined number of temporally extended subpolicies, which may not suit complex continuous-control domains.
- Hand-engineered subgoals and pseudo-rewards are common ways prior work acquires skills, whereas SeCTAR does not assume a proxy reward function.
- SeCTAR’s exploration method maximizes entropy of marginal trajectory distributions, providing density-based exploration at the trajectory level.
- Related trajectory-segment models address planning but lack an exploration mechanism and do not consider hierarchical tasks.
5. Experiments
Experiments evaluate SeCTAR on four sparse-reward, long-horizon domains, testing unsupervised exploration, hierarchical control, and consistency of latent-space predictions. The method explores more broadly, outperforms the baselines, and produces mostly consistent interpolated trajectories.
- Experimental Setup: The evaluation covers 2D navigation, wheeled locomotion, object manipulation, and swimmer navigation, spanning sparse rewards, continuous actions, interaction skills, and exploration challenges.
- Unsupervised Exploration: In unsupervised training, the explorer policy covers a significantly larger portion of the state space than a randomly initialized policy.For manipulation, it learns to pick up and move objects; in locomotion and navigation, it explores different state-space regions.
- Hierarchical Control: The full evaluation compares SeCTAR with TRPO, A3C, VIME, a model-based method, FeUdal Networks, and option-critic across five randomly generated goal configurations per environment.
- Hierarchical Control: SeCTAR significantly outperforms prior methods in task performance and sample complexity on tasks requiring sequential, long-horizon reasoning with delayed and sparse rewards.The comparison includes model-free, model-based, exploration, and hierarchical baselines.
- Hierarchical Control: Trajectory-level models outperform the model-based baseline on block manipulation, which the authors attribute to predicting outcomes of closed-loop skills rather than arbitrary actions.The authors suggest this reduces susceptibility to compounding prediction errors.
- Model Analysis: Interpolated latent codes produce mostly consistent state-decoder and policy-decoder trajectories, supporting generalization to unseen behaviors and a structured latent space.
6. Conclusion
The paper combines trajectory representation learning with model-based planning in a continuous behavior space for hierarchical reinforcement learning. Experiments report improved performance on long-horizon, sparse-reward, multi-step tasks relative to prior and flat methods.
- SeCTAR combines trajectory representation learning with model-based planning in a continuous latent space of behaviors.
- The method is evaluated on tasks requiring long-horizon reasoning, sparse-reward handling, and multi-step compound skills, where it outperforms several prior and flat reinforcement-learning methods.
A. Experimental Details
The experiments use multilayer neural policies, recurrent trajectory encoder and state decoder components, and an eight-dimensional diagonal Gaussian latent representation.
- Policies use three fully connected layers with 400, 300, and 200 hidden units and ReLU activations, outputting categorical or Gaussian distributions.
- The encoder is a two-layer bidirectional LSTM with 300 hidden units, mean-pooled over time before producing Gaussian parameters.
- SeCTAR uses an 8-dimensional diagonal Gaussian distribution for the latent variable z.
B. Baseline Details
The baselines use distinct implementations and hyperparameter settings, with correctness checks reported for several environments.
- TRPO uses the rllab implementation, while VIME uses the OpenAI implementation with a batch size of 100 × task horizon and step size 0.01.
- MPC uses learning rate 0.001, batch size 512, and 2048 simulated paths per action, with correctness verified on half-cheetah.
- Option Critic replaces DQN with PPO and sweeps the number of options, reward multiplier, and entropy bonuses.Correctness was verified on cartpole, hopper, and cheetah.
- Feudal and A3C implementations are based on chainerRL, with sweeps over β, tmax, and gradient clipping.