Source-linked AI summary
Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets
Karol Hausman, Yevgen Chebotar, Stefan Schaal, Gaurav Sukhatme, Joseph Lim
TL;DR
Imitation learning traditionally depends on isolated demonstrations, limiting its use when robots must perform multiple skills. The paper jointly segments and imitates skills from unstructured, unlabeled demonstrations with a multi-modal policy. Simulations show that the method separates demonstrations into tasks and imitates them with one policy.
Problem
Traditional imitation learning relies on isolated skill demonstrations, making it difficult to scale to robots executing combinations of skills.
Method
The framework jointly segments mixed, unstructured demonstrations and learns a multi-modal stochastic policy conditioned on latent intentions.
Results
The method separates demonstrations into individual skills and learns one multi-modal policy that imitates the segmented skills in simulation.
Takeaways & Limitations
Learning from unstructured, unlabeled demonstrations is presented as a step toward scalable robot learning systems.
Takeaways & Limitations
Future work includes autonomously discovering the number of tasks and evaluating the method on real robots.
Abstract
from arXiv · showhide
Imitation learning has traditionally been applied to learn a single task from demonstrations thereof. The requirement of structured and isolated demonstrations limits the scalability of imitation learning approaches as they are difficult to apply to real-world scenarios, where robots have to be able to execute a multitude of tasks. In this paper, we propose a multi-modal imitation learning framework that is able to segment and imitate skills from unlabelled and unstructured demonstrations by learning skill segmentation and imitation learning jointly. The extensive simulation results indicate that our method can efficiently separate the demonstrations into individual skills and learn to imitate them using a single multi-modal policy. The video of our experiments is available at http://sites.google.com/view/nips17intentiongan
1 Introduction
The paper addresses imitation learning’s reliance on isolated, structured demonstrations, which limits scaling to robots executing multiple skills. It proposes jointly segmenting and imitating skills from unstructured, unlabeled demonstrations with a multi-modal policy.
- Traditional imitation learning uses isolated demonstrations of particular skills, requiring substantial user effort and robot-tailored data for each skill.
- The proposed framework learns to imitate skills from unstructured and unlabeled demonstrations of various tasks.
- Learning from unstructured demonstrations requires mapping observations to state-action pairs, segmenting skills, and imitating each segment.
- The method learns a multi-modal stochastic policy that imitates automatically segmented tasks from unstructured and unlabeled demonstrations.
2 Related Work
Related work spans behavioral cloning, inverse reinforcement learning, multi-task imitation, skill segmentation, and GAN-based methods. This paper distinguishes its approach by jointly separating mixed demonstrations into skills and learning one policy for them.
- Behavioral cloning replicates demonstrations but suffers from covariance shift, whereas inverse reinforcement learning estimates a reward explaining expert behavior.
- Prior multi-task work communicates a new task through a single demonstration, while this paper separates mixed expert demonstrations and avoids new demonstrations at test time.
- Related multi-task inverse reinforcement learning methods use Bayesian or EM-based clustering, while this approach targets higher-dimensional domains with neural networks.
- Unlike approaches that later use reinforcement learning, the proposed method does not assume access to the expert reward function.
- The method builds on GAN and imitation-learning connections while extending disentangled representations to multi-modal policies.
3 Preliminaries
The preliminaries formulate imitation learning through inverse reinforcement learning and its GAN reinterpretation. A policy generator is trained against a discriminator that distinguishes generated from expert state-action samples.
- The paper formulates learning in a finite-horizon Markov decision process with states, actions, dynamics, rewards, initial-state distribution, discount factor, and horizon.
- In imitation learning, expert trajectories are assumed to come from a policy optimizing an unknown reward, which inverse reinforcement learning seeks to recover.
- The GAN formulation treats the policy as a generator that fools a discriminator distinguishing imitated samples from demonstrated expert samples.
- The discriminator is trained on expert and generator samples, while its output supplies a reward signal for policy optimization.
4 Multi-modal Imitation Learning
The multi-modal framework models mixed demonstrations from multiple expert policies using a latent intention variable. It combines adversarial imitation with intention inference so different intention values select different skills.
- The framework treats unstructured demonstrations as a mixture of expert policies and aims to segment their tasks while learning one policy that imitates all of them.
- A latent intention input selects a specific policy mode corresponding to one skill in the demonstrations.
- Trajectories are augmented with intention variables, and the reward depends on the selected intention because different tasks have different reward functions.
- The objective seeks high entropy without a specified intention while collapsing to a particular task when intention is specified.
- A latent intention cost rewards state-action pairs that make intention inference easier, using a neural network to represent p(i|s, a).
- Relation to InfoGAN: The InfoGAN-related formulation uses mutual information between latent variables and generated state-action pairs to prevent mode collapse and encourage interpretable behavioral diversity.
5 Implementation
The implementation addresses training instability by smoothing the discriminator’s reward signal and monitoring latent-intention prediction for policy collapse. It represents intentions with categorical or continuous latent variables, trading fixed cardinality for interpolation flexibility.
- Training stability: High discriminator accuracy can flatten the generator’s reward signal and make the TRPO gradient vanish.Gaussian instance noise was most effective for controlling discriminator performance and producing a smoother generator-policy reward signal.
- Training stability: Gaussian noise is annealed as the generator policy improves toward the end of training.The noise is added to discriminator inputs to avoid degenerate generative distributions.
- Policy collapse: A high or increasing intention-prediction loss indicates that the policy has collapsed to a uni-modal behavior across latent intentions.Increasing the latent intention cost can help prevent this collapse.
- Latent intention: Categorical and continuous latent variables represent intention, with continuous variables avoiding a preset number of skills and supporting interpolation.Categorical variables use a softmax layer, while continuous variables use a uniform distribution.
6 Experiments
The experiments test whether the framework can segment and imitate unstructured demonstrations across multi-target, multi-task, high-dimensional, and hierarchical robotics settings. Results show that latent intention costs generally enable multimodal policies, while performance becomes more difficult for continuous intentions and high-dimensional tasks.
- Experimental setup: The experiments evaluate skill segmentation and multimodal imitation on Reacher, Walker-2D, Humanoid, and Gripper-pusher tasks.The evaluation covers multi-target imitation, different behaviors, high-dimensional policies, and hierarchical task structure.
- Multi-target imitation: With the latent intention cost, categorical intentions reach all Reacher targets, while continuous intentions discover two modes for two targets but only two modes for four targets.The four-target continuous-intention case is described as significantly more difficult.
- Multi-target imitation: Without the latent intention cost, the Reacher baseline collapses to one target for two-target tasks and two modes for four-target tasks.The four-target baseline can still learn two modes because the demonstrations are diverse.
- Multi-target imitation: The latent intention cost separates Reacher skills during training, with distinct intention-conditioned behavior emerging around the 1000-th iteration.The resulting policy consistently reaches the target associated with each intention, unlike the standard GAN setup.
- Multi-task and hierarchical imitation: For a single Reacher target, different latent intentions produce different successful trajectories, and in Gripper-pusher changing intention after grasping switches execution to pushing.The Gripper-pusher result illustrates extracting grasping and pushing options for hierarchical reinforcement learning.
- Multi-task and high-dimensional imitation: On Humanoid, the method nearly matches expert performance for running forwards and balancing but remains suboptimal for running backwards, whereas the baseline collapses to balancing.Walker-2D results show autonomous segmentation and imitation of running forwards, running backwards, and jumping.
7 Conclusions
The method learns a multi-modal stochastic policy that segments unstructured, unlabeled demonstrations into tasks and imitates the resulting skills. The authors identify autonomous task-count discovery and real-robot evaluation as future work.
- The method learns a multi-modal stochastic policy that imitates automatically segmented tasks from unstructured and unlabeled demonstrations.It uses policy intentions to perform different tasks and was evaluated in simulation across scenarios involving segmentation and multi-modal imitation.
- The evaluations show that the method segments demonstrations into different tasks and learns one multi-modal policy that imitates the segmented skills.
- Future work includes autonomously discovering the number of tasks in the demonstration pool and evaluating the method on real robots.The authors also plan to learn a hierarchical policy over the discovered intentions.