PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Conditioned Imitation Learning over Unstructured Data

Corey Lynch, Pierre Sermanet

arXiv:2005.07648v2cs.ROcs.AIcs.CLcs.CV

TL;DR

The paper addresses the difficulty of specifying diverse robotic tasks and training instruction-following agents from structured data or limited language labels. It introduces end-to-end language-conditioned visuomotor imitation learning over mostly unstructured demonstrations, then combines the policy with pretrained language models. The resulting policy performs long-horizon manipulation and follows thousands of synonym instructions without new demonstrations, while language annotation costs less than 1% of total data.

  • Problem

    Robotic imitation learning commonly relies on task IDs or goal images, while instruction-following methods often assume structured observations, actuators, or language unsuitable for complex robotics.

  • Method

    The method combines free-form text conditioning with multitask imitation learning, learning pixels, language understanding, and continuous control end-to-end from unstructured unlabeled demonstrations.

  • Results

    The policy performs multiple natural-language manipulation tasks over long horizons, while pretrained language models enable thousands of synonym instructions without new robot demonstrations.

  • Takeaways & Limitations

    Multicontext imitation reduces language annotation to less than 1% of total data and supports scalable language-conditioned robot learning.

  • Takeaways & Limitations

    The work is limited to task-agnostic control in a single simulated environment with a fixed set of objects and lacks autonomous policy improvement.

Abstract

from arXiv · show

Natural language is perhaps the most flexible and intuitive way for humans to communicate tasks to a robot. Prior work in imitation learning typically requires each task be specified with a task id or goal image -- something that is often impractical in open-world environments. On the other hand, previous approaches in instruction following allow agent behavior to be guided by language, but typically assume structure in the observations, actuators, or language that limit their applicability to complex settings like robotics. In this work, we present a method for incorporating free-form natural language conditioning into imitation learning. Our approach learns perception from pixels, natural language understanding, and multitask continuous control end-to-end as a single neural network. Unlike prior work in imitation learning, our method is able to incorporate unlabeled and unstructured demonstration data (i.e. no task or language labels). We show this dramatically improves language conditioned performance, while reducing the cost of language annotation to less than 1% of total data. At test time, a single language conditioned visuomotor policy trained with our method can perform a wide variety of robotic manipulation skills in a 3D environment, specified only with natural language descriptions of each task (e.g. "open the drawer...now pick up the block...now press the green button..."). To scale up the number of instructions an agent can follow, we propose combining text conditioned policies with large pretrained neural language models. We find this allows a policy to be robust to many out-of-distribution synonym instructions, without requiring new demonstrations. See videos of a human typing live text commands to our agent at language-play.github.io

I. INTRODUCTION

The paper targets scalable robotic imitation learning with free-form language, high-dimensional visual input, and continuous control. It combines multitask imitation, unstructured demonstrations, and pretrained language models to support long-horizon instruction following with minimal language annotation.

  • Motivation: Open-world imitation learning requires scalable task specification because task IDs, goal images, and target configurations are often impractical to provide.The paper frames free-form natural language as a way for untrained users to specify varied robot behavior.
  • Motivation: Prior instruction-following systems often assume 2D observations, simplified actuators, or synthetic predefined language, limiting applicability to complex robotics.The proposed setting combines pixel observations, 8-DOF continuous control, and long-horizon object manipulation.
  • Approach: The method learns perception, language understanding, and multitask control end-to-end while using large unstructured, unlabeled demonstrations.Its multicontext imitation approach combines free-form text conditioning with multitask imitation learning.
  • Approach: A single policy is trained for image or language goals, with language conditioning used at test time.The workflow relabels teleoperated play data into hindsight goal images, pairs a small amount of play with hindsight instructions, and trains one policy.
  • Results: Less than 1% of total data requires language annotation, while the policy can follow multiple natural-language instructions over long horizons in a simulated 3D tabletop.Example instruction sequences include opening a door, picking up a block, and pressing a button.
  • Results: Combining language-conditioned policies with large pretrained language models improves manipulation performance and supports thousands of synonym instructions in 16 languages without new demonstrations.This extends instruction coverage beyond the finite set of language used during training.

II. RELATED WORK

Prior work addresses robotic skill learning, task-agnostic control, and instruction following, but existing methods often rely on structured demonstrations or constrained task descriptions. This work connects unstructured imitation data with natural-language conditioning through multicontext learning and pretrained language representations.

  • Conventional imitation learning commonly learns skills from structured, isolated human demonstrations, making large-scale generalist learning difficult.
  • Multicontext learning addresses this mismatch by sharing a latent goal space across heterogeneous task descriptions, enabling transfer from plentiful contexts to scarce language supervision.
  • Instruction-following approaches often assume 2D observations, simplified actuators, or synthetic predefined language, limiting applicability to complex robotics.
  • Relabeled imitation learning can exploit unstructured, unlabeled demonstrations by converting long play trajectories into short-horizon goal-image demonstrations.
  • Language-conditioned imitation cannot directly relabel visited states as natural-language goals because language goals are not equivalent to observation-space goals.

IV. LEARNING TO FOLLOW HUMAN INSTRUCTIONS FROM UNSTRUCTURED DATA

The approach combines large unstructured play data with sparse hindsight language annotations, then trains a shared imitation policy across contextual datasets. Multicontext imitation provides the mechanism for learning language-conditioned behavior without labeling every demonstration.

  • IV. LEARNING TO FOLLOW HUMAN INSTRUCTIONS FROM UNSTRUCTURED DATA: The pipeline collects a large unstructured play dataset and relabels it into goal-image demonstrations.
  • A. Pairing unstructured demonstrations with natural language: A small number of random windows from play data are paired after-the-fact with hindsight instructions from human annotators.
  • A. Pairing unstructured demonstrations with natural language: The paired corpus asks annotators what instruction would move the agent from the first frame to the last frame of each sampled window.
  • IV. LEARNING TO FOLLOW HUMAN INSTRUCTIONS FROM UNSTRUCTURED DATA: Only a fraction of play windows need language labels because multicontext imitation transfers control learned from goal-image data to language conditioning.
  • A. Pairing unstructured demonstrations with natural language: Algorithm 1 represents each contextual dataset with demonstrations and contexts, using one encoder per context type and a single latent goal-conditioned policy.
  • A. Pairing unstructured demonstrations with natural language: Training loops over contextual datasets, encodes each context into the shared latent space, accumulates imitation loss, and updates the policy and encoders end-to-end.

B. Multicontext Imitation Learning

Multicontext Imitation Learning unifies heterogeneous task descriptions through a shared latent goal space. This lets abundant, cheaply collected contexts provide control learning while sparse labeled contexts provide scalable task conditioning.

  • For this paper, MCIL enables language-conditioned imitation to use self-supervised imitation from unstructured demonstrations.
  • MCIL generalizes contextual imitation learning to multiple heterogeneous task-description types.
  • MCIL trains one latent goal-conditioned policy across datasets and one parameterized encoder per dataset.
  • The encoders map raw task descriptions, such as task identifiers, goal images, or language, into a common latent goal space.
  • Each training step samples contextual examples, encodes their contexts, averages latent goal-conditioned imitation losses, and updates all modules end-to-end.
  • MCIL uses the cheapest data source for most control learning while learning task conditioning from a small labeled set.

C. LangLfP: following image or language goals.

LangLfP applies multicontext imitation to goal-image and language demonstrations, learning perception, language understanding, and control jointly. At test time it follows free-form instructions from pixels, while pretrained language embeddings extend synonym robustness.

  • LangLfP is an MCIL instance that learns pixel perception, natural-language understanding, and control end-to-end without auxiliary losses.
  • LangLfP jointly trains on relabeled goal-image demonstrations and demonstrations paired with hindsight language instructions.
  • At test time, LangLfP uses onboard pixel observations and free-form task descriptions to control the robot in closed loop.
  • Pretrained language embeddings relate unseen synonym instructions to instructions observed during training.
  • The experiments evaluate language-conditioned multitask imitation in a simulated 3D Playroom with continuous visual control and 18 task families.
  • The evaluation compares language-only, goal-image-only, and combined play-and-language methods, including LangBC, LfP, and LangLfP.

A. Human Language Conditioned Visual Manipulation

The evaluation tests language-conditioned manipulation across individual and long-horizon tasks, showing that LangLfP supports free-form instructions, benefits from unstructured demonstrations, and enables interactive language assistance.

  • 68.6% success on individual free-form instructions and 52.1% on 925 four-stage instructions demonstrate language-conditioned manipulation across short and long horizons.The Chain-2, Chain-3, and Chain-4 benchmarks evaluate multi-stage task sequences.
  • LangLfP matches prior goal-image-conditioned LfP across benchmarks while requiring language annotation for only ∼0.1% of demonstration data.
  • LangLfP outperforms LangBC on every benchmark, including when labeled and unlabeled training sources are restricted to the same size.The comparison attributes the improvement to training over unlabeled demonstrations through MCIL.
  • Performance scales linearly with model size for policies trained on unstructured play, whereas conventional-demonstration models peak and then decline.The pattern holds when datasets are restricted to the same size.
  • Natural language conditioning lets human operators give interactive guidance when the agent gets stuck, including extra substeps that help recover the task.An operator can revise commands during execution, such as moving back, opening a door further, and pressing a button.

B. Instruction Following with Large Pretrained Language

Pretrained language representations improve language-guided manipulation and make policies robust to unseen synonym instructions, while the evaluation also documents failure modes and scope boundaries.

  • Models: TransferLangLfP systematically outperforms LangLfP and LfP, showing positive transfer from pretrained sentence embeddings to robotic manipulation.
  • Models: Pretrained language models improve convergence of language-guided robotic control policies.
  • Models: Only TransferLangLfP is robust to out-of-distribution synonyms, and multilingual pretraining extends this robustness to 16 languages without vocabulary overlap in training.
  • C. Limitations and Future Work: At test time, policies can encounter compounding errors, such as awkward arm configurations that cause repeated attempts and timeouts.The paper suggests more stable rotation representations or more varied play collection as possible mitigations.
  • C. Limitations and Future Work: Language assistance can help the human recover the agent from awkward configurations during execution.
  • C. Limitations and Future Work: LangLfP remains a goal-directed imitation method and lacks autonomous policy improvement.The paper identifies combining imitation pretraining with reinforcement learning as future work.
  • C. Limitations and Future Work: The demonstrated scope is task-agnostic control in one simulated environment with fixed objects, under an i.i.d. training and test-task assumption.Generalization to unseen rooms or objects remains an open question.
  • Models: The method combines any language-conditioned policy with large pretrained language models to follow many new synonym instructions without collecting demonstrations for each synonym.

APPENDIX

The appendix describes how teleoperated play is relabeled into goal-conditioned examples, sparsely paired with hindsight language, and used in an end-to-end multicontext imitation framework.

  • Algorithm 2 converts unsegmented observation-action play streams into many examples by extracting windows of varying sizes and treating each window’s final observation as its goal.
  • Algorithm 3 samples a small number of relabeled play trajectories and asks a human overseer for hindsight instructions that make each trajectory optimal.The number of language-labeled pairs K is much smaller than the play dataset size.
  • The implementation trains perception, language understanding, and control end-to-end as one neural network using relabeled play, language-paired play, and multicontext imitation.
  • Raw images and proprioception are mapped to a combined 72-dimensional perceptual embedding before control prediction.The visual component contributes a 64-dimensional embedding.
  • The goal-image encoder maps a goal observation through the perception module and a two-layer ReLU MLP to produce a latent goal representation.

E. Language understanding module

The language module maps text or images into a shared latent goal space for a multicontext LMP policy, which executes language goals in closed loop in the simulated playground.

  • From scratch, the language encoder tokenizes text, embeds subwords, averages the sequence, and maps it through an MLP to a 32-dimensional goal-space embedding.
  • Transfer learning uses fixed 512-dimensional MUSE sentence embeddings trained on generic multilingual corpora before an MLP maps them onward.
  • The paper treats pretrained semantic embedding choices as interchangeable possibilities and leaves exploration of alternatives to future work.
  • Multicontext LMP adapts Latent Motor Plans to accept either image or language goals while retaining separate latent goal and plan spaces.
  • The LMP models a posterior over recognized plans, a conditional prior over possible plans, and a goal- and plan-conditioned policy that reconstructs actions.
  • Training replaces image-goal conditioning with latent goals from the image and language encoders and shares the posterior, prior, policy, and perception networks.
  • At test time, the agent encodes a human language goal and repeatedly feeds the current observation and latent goal to its closed-loop policy; the operator may type a new goal anytime.

I. 3D simulated environment

The experiments use a simulated 3D tabletop environment with an 8-DOF robot, multimodal observations, and play data paired with diverse hindsight language instructions.

  • I. 3D simulated environment: The environment includes a desk with manipulable doors, a drawer, a block, colored buttons, and a nearby trash bin.Physics are simulated with MuJoCo, and the robot has an 8-DOF arm and parallel gripper.
  • I. 3D simulated environment: Pixel experiments use 200x200x3 RGB images and 8-DOF proprioceptive state, while state experiments provide proprioception, object pose, and environment sensors.Both experiment types also expose the raw natural-language string at each timestep.
  • I. 3D simulated environment: The action space contains 8-DOF Cartesian position, Euler orientation, and gripper-angle controls, quantized into 256 bins during training.Stochastic outputs use discretized logistic distributions over the quantization bins.
  • I. 3D simulated environment: The play data contains approximately 7 hours of relabeled play, divided into about 10M short-horizon windows spanning 1–2 seconds.The dataset is used as the basis for relabeled goal-image-conditioned learning.
  • I. 3D simulated environment: 10K play windows are paired with natural-language hindsight instructions, collected by asking overseers to describe how to reproduce each clip.The interface uses looping 1–2-second videos; overseers were asked to provide diverse descriptions focused on main actions.
  • I. 3D simulated environment: The test dataset pairs demonstrations from each of 18 evaluation tasks with hindsight instructions, and 74.3% of test instructions appear at least once in the play-language dataset.Conversely, 85.08% of instructions in the play-language dataset never appear in the test set.

O. Restricted play dataset

The restricted-play evaluation compares methods trained on matched data under long-horizon multitask conditions, including a neutral-start benchmark that forces reliance on language.

  • O. Restricted play dataset: For controlled comparison, the play dataset is subsampled from approximately 7 hours to approximately 1.5 hours before relabeling.This matches the aggregate size of the multitask demonstration dataset.
  • O. Restricted play dataset: All baseline methods use the same architecture and differ only in their training-data source.The baselines maximize the same generalized contextual imitation objective, MCIL.
  • O. Restricted play dataset: Each multi-stage rollout starts from a neutral scene, samples one natural-language instruction per stage, and advances only after the current subtask succeeds.An 8-second failure timeout ends the episode, and performance is the average percentage of subtasks completed.
  • O. Restricted play dataset: Neutral initialization removes correlations between the initial arm pose and task identity, creating a fairer but substantially harder evaluation.It forces the agent to infer and solve tasks from language rather than from the starting state.
  • O. Restricted play dataset: Play-trained LangLfP remains far more robust to neutral starting positions than LangBC, whose performance degrades quickly after the first instruction.The figure caption summarizes the same contrast between relabeled-play and conventional-demonstration training.
  • O. Restricted play dataset: LangLfP composes previously unbenchmarked tasks zero shot by chaining language commands such as “pick up the object” and “put the object in the trash.”The same approach also executes “put the object on top of the shelf,” although neither task is included in the 18-task benchmark.

Q. Play scales with model capacity

Performance scales differently with model capacity and language variation: play-trained models benefit from larger capacity, while pretrained-language transfer improves robustness to unseen instructions and languages.

  • Q. Play scales with model capacity: Performance steadily improves with model size for play-trained models but peaks and declines for models trained on equally sized conventional demonstrations.The authors interpret play diversity as allowing larger capacity to be used more effectively.
  • Q. Play scales with model capacity: The operator can adapt instructions and add substeps when the robot becomes stuck, including moving back and opening a door farther before pressing a button.This example illustrates interactive recovery during execution of an initial instruction.
  • Q. Play scales with model capacity: The synonym benchmark replaces words in evaluation instructions with unseen synonyms, producing 14,609 out-of-distribution instructions.The substitutions occur only at test time and target the same fixed set of training behaviors.
  • Q. Play scales with model capacity: TransferLangLfP degrades only slightly on instructions translated into 16 languages, whereas LangLfP degrades materially when instructions have no training overlap.The multilingual benchmark contains approximately 240K out-of-distribution instructions across all 18 tasks.
  • Q. Play scales with model capacity: For state-input models, increasing the language-pair dataset from 5K to 10K provides marginal performance benefit.The scaling study compares pixel- and state-input models across language-pair dataset sizes.
Loading 2005.07648v2…