Source-linked AI summary
Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations
Ajay Mandlekar, Danfei Xu, Roberto Martín-Martín, Silvio Savarese, Li Fei-Fei
TL;DR
Imitation learning struggles to generalize beyond demonstrated behaviors, while annotated demonstrations can be costly and difficult to scale to physical robots. GTI exploits intersecting demonstration trajectories through a two-stage stochastic-imitation and goal-directed-learning process, and demonstrates novel start-goal behavior in simulated and real-world long-horizon manipulation tasks.
Problem
Imitation-learning methods require large amounts of annotated demonstrations and still face an open challenge in generalizing beyond demonstrated behaviors.
Method
GTI uses intersecting trajectories to train a stochastic policy, then distills rollouts from it into a goal-directed policy for novel start-goal configurations.
Results
GTI reproduced demonstrated behaviors and generalized to novel start-goal configurations in simulated domains and a challenging real-world kitchen domain.
Takeaways & Limitations
Under an hour of human demonstrations was sufficient to learn novel and unseen goal-directed behavior on long-horizon manipulation domains.
Abstract
from arXiv · showhide
Imitation learning is an effective and safe technique to train robot policies in the real world because it does not depend on an expensive random exploration process. However, due to the lack of exploration, learning policies that generalize beyond the demonstrated behaviors is still an open challenge. We present a novel imitation learning framework to enable robots to 1) learn complex real world manipulation tasks efficiently from a small number of human demonstrations, and 2) synthesize new behaviors not contained in the collected demonstrations. Our key insight is that multi-task domains often present a latent structure, where demonstrated trajectories for different tasks intersect at common regions of the state space. We present Generalization Through Imitation (GTI), a two-stage offline imitation learning algorithm that exploits this intersecting structure to train goal-directed policies that generalize to unseen start and goal state combinations. In the first stage of GTI, we train a stochastic policy that leverages trajectory intersections to have the capacity to compose behaviors from different demonstration trajectories together. In the second stage of GTI, we collect a small set of rollouts from the unconditioned stochastic policy of the first stage, and train a goal-directed agent to generalize to novel start and goal configurations. We validate GTI in both simulated domains and a challenging long-horizon robotic manipulation domain in the real world. Additional results and videos are available at https://sites.google.com/view/gti2020/ .
I. INTRODUCTION
Imitation learning struggles to generalize beyond demonstrated behaviors, especially in combinatorial long-horizon tasks. GTI exploits intersections between demonstration trajectories to compose unseen behaviors through a two-stage imitation-learning process.
- Imitation learning requires broad demonstration coverage to generalize, which is unrealistic for long-horizon multi-stage manipulation with combinatorial task instances.
- GTI leverages intersections between trajectories that share states to compose known trajectory segments into novel paths between unseen task configurations.
- GTI Stage 1 trains a stochastic policy that generates diverse behaviors by making divergent choices at trajectory intersections.
- GTI Stage 2 trains a goal-directed policy from rollouts of the Stage 1 agent to achieve controllable new behaviors.
- The approach is presented as learning novel, unseen goal-directed behavior on long-horizon manipulation domains from under an hour of human demonstrations.
II. RELATED WORK
Prior long-horizon imitation methods use one-shot representations or hierarchical planning, but they do not explicitly exploit compositional structure at intersections between demonstrations. GTI instead uses diverse behaviors from imitation to train new goal-conditioned policies.
- One-shot imitation learning: One-shot imitation represents tasks with videos, image keyframes, or state-space trajectories and optimizes likelihood, distribution matching, or related objectives.
- Hierarchical planning with imitation: Hierarchical imitation methods learn a high-level planner and low-level goal-conditioned controller to handle longer horizons through subgoals.
- Limitations of prior methods: Existing hierarchical methods do not explicitly exploit compositional structure in demonstrations containing state intersections.
- Compositional alternatives: Demonstration segmentation and recomposition methods recover subtasks, whereas GTI avoids explicitly modeling demonstration temporal structure.
- GTI perspective: GTI uses a goal-agnostic imitation agent to generate new trajectories, then trains goal-conditioned policies from those trajectories.
III. PROBLEM FORMULATION
The problem is formulated as goal-conditioned robot manipulation in an MDP, where each initial-state and goal pairing defines a task instance. GTI targets unseen pairings by exploiting intersections between demonstration trajectories to recombine their implied paths.
- MDP formulation: Robot manipulation is modeled as a discrete-time infinite-horizon MDP with state, action, transition, reward, discount, and initial-state components.
- Task instances: Each pair of an initial state and goal set defines a distinct task instance.
- Trajectory-intersection assumption: The demonstration dataset is assumed to contain trajectories that intersect at certain states.
- Compositional generalization: Intersecting trajectories implicitly provide information about unseen start-goal pairs through novel combinations at their shared states.
- Objective: GTI seeks a policy that solves new initial-state and goal pairs not demonstrated in the dataset by exploiting variability after trajectory intersections.
IV. METHOD
GTI uses a two-stage approach to achieve compositional generalization from intersecting demonstrations.
- GTI extracts information from intersecting demonstrations through a two-stage approach.
Stage 1: Multimodal Imitation Learning to Generate Novel Behaviors
Stage 1 learns a stochastic, multimodal imitation policy that reproduces diverse demonstrations and composes behaviors at trajectory intersections to generate novel paths.
- The Stage 1 policy reproduces diverse demonstrations and composes behavior before and after trajectory intersections.
- GTI decomposes imitation into predicting future states from current observations and training a goal-conditioned policy using those predictions.
- Stage 2 data collection samples random starts and latent goals, then records H-length rollouts from the Stage 1 policy for later training.
- The cVAE learns p(s_t+H|s_t) from demonstration image sequences to generate possible future observations.
- A Gaussian Mixture Model prior produces diverse, multimodal image distributions for future-goal generation.
- The low-level controller conditions on latent goals rather than high-dimensional goal images and is trained to predict H-step action sequences.
Stage 2: Goal-Directed Imitation of Novel Behaviors
Stage 2 turns Stage 1's compositional rollouts into a goal-directed policy that can control newly generated behaviors.
- Stage 2 uses Stage 1 to generate new demonstrations through compositional generalization, then learns goal-directed control.
- In PointCross and PointCrossStay, GTI Stage 1 is shown alongside demonstrations, BC, GCBC, and cVAE goal predictions to assess reproduced and novel behaviors.
- The final policy is trained by behavioral cloning, using each rollout’s final state as a fixed goal for predicting actions throughout the trajectory.
- Single-step goal-directed behavioral cloning keeps Stage 2 policy capacity separate from the quality of Stage 1-generated data.
- PandaKitchen combines two initial configurations and two goals, with intersecting task sequences enabling novel start-goal combinations.
V. EXPERIMENTS
GTI is evaluated in simulation for controlled analysis and on real-world long-horizon manipulation tasks, against BC and GCBC baselines.
- The evaluation covers simulated environments and real-world long-horizon tasks to separate algorithmic analysis from physical execution effects.
- GTI is compared with Behavioral Cloning and Goal-Conditioned Behavioral Cloning.
A. Simulation Experiments
The simulation evaluates whether GTI can use trajectory intersections to reproduce demonstrated behavior while generating unseen routes across start and goal combinations. Compared with BC and GCBC, GTI handles multimodal outcomes and avoids collapse at conflicting bottleneck states.
- Simulation setup: The simulations test Stage 1 policies on low-dimensional navigation tasks designed to assess imitation and novel trajectory generation.The tasks are PointCross and PointCrossStay.
- Evaluation: Evaluation averages four metrics across 10 start locations and 100 rollouts per location: Goal Reach Rate, Seen Behavior, Unseen Behavior, and Occupancy.Occupancy is 100% when both lower goals are reached, 50% when only one is reached, and 0% when neither is reached.
- PointCross results: On PointCross, BC reaches goals consistently but collapses to one goal per start, while GCBC reaches seen combinations but fails on unseen combinations.BC therefore has 50% occupancy, whereas GCBC has 50% success rate and no unseen behavior.
- PointCross results: GTI generates both seen and unseen behavior on PointCross, reaching both lower goals from both upper-left and upper-right starting states.This demonstrates composition across start and goal combinations not represented in the demonstrations.
- PointCrossStay results: On PointCrossStay, BC and GCBC reach no goals because conflicting supervision at the origin favors staying, whereas GTI escapes and reaches both goals from both starts.GTI models diverse future states near the origin using the cVAE and GMM prior.
B. Real World Robot Manipulation Experiments
The real-robot experiments test GTI on multimodal reaching and long-horizon kitchen manipulation using visual teleoperation data. GTI exhibits diverse Stage 1 behavior and its Stage 2 goal-directed policy generalizes to unseen start-goal combinations.
- PandaReach: PandaReach tests whether an unconditioned policy visits left and right goals equally; GTI does so, while BC consistently visits only one goal.The experiment uses 20 demonstrations for each goal location, and the result is reported over 40 rollouts.
- PandaKitchen: PandaKitchen provides demonstrations for A0 to AG and B0 to BG, then evaluates unseen combinations A0 to BG and B0 to AG in long-horizon cooking tasks.The demonstrated tasks respectively reheat bread in the oven and retrieve bread from a covered container to serve it on a plate.
- Stage 1 results: Stage 1 GTI solves 71.4% of A0 instances and visits AG and BG equally, while B0 success is 46.7% and successful rollouts reach novel goals 42% of the time.The authors hypothesize that lower B0 success reflects the longer sequence required for the B0-to-AG task.
- Stage 2 results: Stage 2 GTI achieves at least 50% success on all four start-goal combinations, including novel A0-to-BG and B0-to-AG tasks, outperforming GCBC trained on the original demonstrations.Stage 2 is trained solely on rollouts generated by the Stage 1 GTI policy.
VI. CONCLUSION
GTI leverages trajectory intersections to generalize from task demonstrations to unseen combinations of initializations and goals. It reproduces demonstrated behaviors and generalizes to novel configurations in simulated and real-world kitchen domains.
- GTI leverages trajectory crossings to generalize from task demonstrations to unseen combinations of task initializations and desired goals.
- GTI reproduces behaviors from demonstrations while generalizing to novel start and goal configurations.
- The method was demonstrated in simulated domains and a challenging real-world kitchen domain.