PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Imitation of Diverse Behaviors

Ziyu Wang, Josh Merel, Scott Reed, Greg Wayne, Nando de Freitas, Nicolas Heess

arXiv:1707.02747v2cs.LG

TL;DR

The paper addresses the tension between supervised imitation's brittleness and GAIL's limited behavioral diversity. It combines a trajectory VAE with GAIL, yielding robust, diverse imitation across biped and humanoid control tasks while learning structured behavior embeddings.

  • Problem

    Supervised imitation is brittle after trajectory divergence, whereas GAIL is more robust with fewer demonstrations but can suffer mode collapse and lose behavioral diversity.

  • Method

    The method combines a trajectory VAE that learns semantic policy embeddings with an adversarial imitation objective based on GAIL.

  • Results

    The approach learns robust, diverse behaviors on a 2D biped and a 62 DoF humanoid, while its embeddings support smooth policy interpolation.

  • Takeaways & Limitations

    A single embedding-based controller can robustly execute diverse behaviors and encode new trajectories for one-shot imitation.

  • Takeaways & Limitations

    The analysis includes an assumption that the inference model computes the true posterior distribution.

Abstract

from arXiv · show

Deep generative models have recently shown great promise in imitation learning for motor control. Given enough data, even supervised approaches can do one-shot imitation learning; however, they are vulnerable to cascading failures when the agent trajectory diverges from the demonstrations. Compared to purely supervised methods, Generative Adversarial Imitation Learning (GAIL) can learn more robust controllers from fewer demonstrations, but is inherently mode-seeking and more difficult to train. In this paper, we show how to combine the favourable aspects of these two approaches. The base of our model is a new type of variational autoencoder on demonstration trajectories that learns semantic policy embeddings. We show that these embeddings can be learned on a 9 DoF Jaco robot arm in reaching tasks, and then smoothly interpolated with a resulting smooth interpolation of reaching behavior. Leveraging these policy representations, we develop a new version of GAIL that (1) is much more robust than the purely-supervised controller, especially with few demonstrations, and (2) avoids mode collapse, capturing many diverse behaviors when GAIL on its own does not. We demonstrate our approach on learning diverse gaits from demonstration on a 2D biped and a 62 DoF 3D humanoid in the MuJoCo physics environment.

1 Introduction

The paper targets imitation policies that are both robust to trajectory deviations and capable of representing diverse behaviors from few demonstrations. It combines supervised generative modeling with adversarial imitation to address their complementary limitations.

  • Versatile embodied agents must reproduce diverse, adaptive motor behaviors, but this remains a long-standing AI challenge.
  • The model's base VAE maps demonstration trajectories into structured semantic embeddings and supports smooth interpolation of policies and reaching behaviors.
  • Supervised imitation can support one-shot learning but requires large datasets for non-trivial tasks and becomes brittle after trajectory divergence.
  • GAIL improves robustness with fewer demonstrations through environment interaction, but adversarial training can cause mode collapse and insufficient behavioral diversity.
  • The proposed approach combines VAE and GAIL properties to obtain robust policies that capture diverse behaviors, including on a 2D biped and humanoid.

2 Background and Related Work

The background frames imitation learning as generative modeling under environment and action constraints, contrasting likelihood-based behavioral cloning with adversarial methods. It reviews VAEs, GANs, and GAIL as models with different tractability, robustness, and diversity trade-offs.

  • Generative models: VAEs optimize a variational lower bound using latent-variable inference, while autoregressive models factorize likelihood into conditional terms.
  • Generative models: GANs distinguish generated from real samples but suffer mode collapse, unlike maximum-likelihood VAEs and autoregressive models, which preserve different modeling trade-offs.
  • Imitation learning: Imitation learning seeks a control policy that mimics behavior supplied through demonstration while generating trajectories through environment interaction.
  • Behavioral cloning: Behavioral cloning maximizes action likelihood when dynamics are unavailable, but small errors accumulate during sequential rollout unless corrective behaviors appear in training data.
  • GAIL: GAIL uses a GAN-based reward measuring similarity between policy-generated and expert trajectories, then trains the policy with reinforcement learning.
  • Related work: The imitation-learning literature includes behavioral cloning, apprenticeship learning, inverse reinforcement learning, and physics-based controllers requiring substantial human insight.

3 A Generative Modeling Approach to Imitating Diverse Behaviors

The approach combines a stochastic VAE for trajectory-conditioned policy representations with a conditional GAIL objective. Conditioning adversarial training on latent embeddings aims to preserve diverse behaviors while improving robustness.

  • 3.1 Behavioral cloning with variational autoencoders suited for control: A stochastic VAE maps demonstration sequences to latent embeddings and decodes both state and action trajectories.Its bidirectional LSTM encoder produces the embedding, while separate action and state decoders reconstruct trajectory behavior.
  • 3.1 Behavioral cloning with variational autoencoders suited for control: The action decoder predicts continuous actions from the current state and embedding, while the state decoder autoregressively models state components with a conditional WaveNet.The state decoder conditions on the embedding and previous state and uses a mixture of Gaussians output.
  • 3.2 Diverse generative adversarial imitation learning: The method conditions GAIL’s discriminator on VAE-generated embeddings and integrates the adversarial objective over the variational posterior.This produces conditional rewards tailored to different embedded trajectories rather than a single unconditional reward.
  • 3.2 Diverse generative adversarial imitation learning: The learned latent space supports one-shot imitation from unlabeled trajectories and continuous interpolation between behaviors.The policy is initialized from the VAE policy and optimized with TRPO while the VAE parameters remain fixed.
  • 3.2 Diverse generative adversarial imitation learning: Latent conditioning reduces mode-collapse pressure because each embedding defines a distinct conditional imitation objective, making conditional distributions closer to unimodal when embeddings remove ambiguity.The paper motivates this through an objective involving Jensen-Shannon divergence between conditional data and generator distributions.

4 Experiments

Experiments evaluate the approach on a Jaco arm, a 2D biped, and a complex humanoid, showing semantic embeddings, diverse imitation, improved robustness, and behavior transitions.

  • Experimental scope: The evaluation spans a 9 DoF Jaco arm, a 9 DoF planar walker, and a 62 DoF humanoid in MuJoCo.The reaching task tests the VAE architecture and embedding space, while the locomotion tasks test the full learning procedure.
  • 4.1 Robotic arm reaching: The Jaco VAE reproduces test demonstrations and maps convex combinations of trajectory embeddings to interpolated reaching trajectories.Latent-space interpolation corresponds to interpolation in task endpoint space, supporting the semantic structure of the embeddings.
  • 4.2 2D Walker: On the diverse 2D walker dataset, BC alone performs poorly, while unconditioned GAIL meshes behaviors together and produces dramatically less diversity.The experiment uses 1200 demonstration trajectories, each 400 steps or 10 seconds long.
  • 4.2 2D Walker: Adversarial training greatly improves walker-controller reliability and accuracy in matching demonstration speeds.Quality is measured by the absolute difference between demonstration and imitation average speeds.
  • 4.2 2D Walker: Walker embedding spaces cluster according to movement speed, while nearby trajectories also exhibit similar movement styles across different speeds.The model also successfully imitates previously unseen trajectories, preserving their movement style.
  • 4.3 Complex humanoid: For the humanoid, adversarial training dramatically improves stability, and the controller often transitions robustly between behaviors by blending embeddings over 20 control steps.The humanoid evaluation measures the percentage of episodes in which the agent falls before the episode ends.

5 Conclusions

The approach combines latent-variable density modeling with GAIL to learn structured behavior embeddings and robust controllers for diverse behaviors. It works across varied control problems, including a challenging simulated humanoid.

  • The model learns semantically structured behavior embeddings, a robust multi-task controller, and an encoder supporting one-shot imitation.These capabilities are learned from a moderate number of demonstration trajectories.
  • The approach combines VAEs' favorable density-modeling properties with GAIL's complementary imitation-learning properties.
  • The approach works on diverse control problems and scales to a simulated humanoid with many degrees of freedom.

A.1 Jaco

The random reaching policies were trained with DDPG to reach randomly selected workspace positions in short simulations.

  • DDPG trained the random reaching policies to reach random positions in the workspace.
  • Each simulation lasted 2.5 secs or 50 steps.
  • Hyper-parameters and network configuration are provided in Table 1.

A.2 Walker

Walker demonstration policies were trained for distinct target speeds, including policies conditioned on context labels. Additional hyper-parameter and network details are referenced in Tables 1, 3, and 4.

  • Target speeds were selected from four values: -1, 0, 1, and 3 m/s.
  • For each individual target speed, 12 policies were trained.
  • Additional sets of 12 policies were trained to achieve three target speeds selected by context labels.One set used -1, 0, and 1; another used -1, 0, and 3.
  • Further hyper-parameter and network configurations are listed in Tables 1, 3, and 4.

A.3 Humanoid

The humanoid experiments use a 62 DoF body with 56 actuated joint angles and a freely translating and rotating root. Training and evaluation draw on controllers representing multiple movement styles, while the discriminator uses mostly end-effector features.

  • The humanoid has 56 actuated joint angles and a freely translating and rotating root.Its actions are torques applied to the joint angles.
  • Training trajectories come from six controllers imitating styles including simple walk, cat, chicken, drunk, and old.Evaluation uses five independently trained controllers with partially overlapping styles.
  • The discriminator receives a subset of features consisting mostly of end-effector positions.
Loading 1707.02747v2…