PaperlayerView exact revision on arXiv

Source-linked AI summary

Interactive Language: Talking to Robots in Real Time

Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, Pete Florence

arXiv:2210.06407v1cs.ROcs.AIcs.LG

TL;DR

Interactive Language addresses the challenge of building real-world robots that can follow diverse natural-language instructions and accept new commands in real time. It combines scalable language-labeled demonstration collection with behavioral cloning and interactive human guidance. The resulting policy addresses 87,000 unique commands at an estimated 93.5% success rate, while the released Language-Table assets support further research.

  • Problem

    Prior work rarely demonstrates real-world robots that capably respond to large numbers of rich, diverse language commands and interactive in-the-moment instructions.

  • Method

    The framework scales robot demonstration collection, applies event-selectable hindsight language relabeling, and trains language-conditioned policies with behavioral cloning.

  • Results

    87,000 unique commands are addressed at an estimated 93.5% success rate, and real-time language guidance supports precise long-horizon rearrangement goals.

  • Takeaways & Limitations

    Real-time language feedback enables humans to guide a single learned policy through complex long-horizon rearrangements and simultaneous multi-robot instruction.

  • Takeaways & Limitations

    The approach addresses only real-time language-guided manipulation, not broader human-robot collaboration settings such as intention detection or physically collaborative task completion.

Abstract

from arXiv · show

We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajectories, a produced policy can proficiently execute an order of magnitude more commands than previous works: specifically we estimate a 93.5% success rate on a set of 87,000 unique natural language strings specifying raw end-to-end visuo-linguo-motor skills in the real world. We find that the same policy is capable of being guided by a human via real-time language to address a wide range of precise long-horizon rearrangement goals, e.g. "make a smiley face out of blocks". The dataset we release comprises nearly 600,000 language-labeled trajectories, an order of magnitude larger than prior available datasets. We hope the demonstrated results and associated assets enable further advancement of helpful, capable, natural-language-interactable robots. See videos at https://interactive-language.github.io.

I. INTRODUCTION

Interactive Language targets real-world robots that respond to diverse, open-vocabulary commands in real time, addressing a setting that prior work rarely demonstrates at scale. The framework combines scalable data collection, language relabeling, behavioral cloning, and interactive guidance, yielding broad short-horizon skills and compositional long-horizon behavior.

  • Scale and capabilities: 87,000 unique commands are addressed at an estimated 93.5% success rate with continuous 5Hz visuolinguomotor control.The policies can also chain raw skills to reach hundreds of thousands of long-horizon goals in the environment.
  • Interactive guidance: Real-time language feedback lets a human guide precise, multi-minute rearrangements such as making a smiley face from blocks.The demonstrated setting studies interactive guidance rather than only short-horizon execution.
  • Interactive guidance: A single human can simultaneously guide multiple real-time robots through long-horizon tasks.This capability is presented as a consequence of real-time language competency.
  • Framework: Interactive Language combines high-throughput robot data collection, event-selectable hindsight language relabeling, and straightforward language-conditioned behavioral cloning.The framework is designed as a simple, scalable recipe for learning large repertoires of language-conditionable skills.
  • Released assets: Nearly 600,000 diverse demonstrations across simulation and the real world make Language-Table an order of magnitude larger than prior natural-language-conditioned imitation datasets.The release includes datasets, models, hardware and environment descriptions, a simulated analogue, and a research benchmark.

III. PROBLEM SETUP

The paper formulates interactive language guidance as learning a policy that maps visual observations and unconstrained human language to continuous robot actions. It uses offline demonstrations and a simple language-conditioned behavioral-cloning framework, with Language-Table providing associated benchmark assets.

  • Problem formulation: The target policy πθ(a|s, l) maps RGB observations and open-vocabulary language to physical-robot actions.The formulation allows humans to introduce new language at any time at the policy’s natural visuo-linguo-motor rate.
  • Problem formulation: Each offline demonstration pairs a variable-length observation-action trajectory with a second-person language command describing the full trajectory.The trajectory contains sequential observation-action pairs from the robot.
  • Learning framework: Interactive Language combines scalable varied-data collection with straightforward language-conditioned behavioral cloning.The framework is intended to be generically applicable to real-time natural-language-interactable robots.

A. Data Collection

Interactive Language scales data collection through continuous teleoperation and event-selectable hindsight language relabeling, then trains a transformer policy with behavioral cloning. The operation reveals language annotation, rather than teleoperation, as its main throughput bottleneck.

  • Data Collection: Operators continuously teleoperate long-horizon behaviors without task definitions, segmentation, or episodic resets, producing semi-structured raw trajectories for relabeling.The collection procedure is designed to maximize the flow of demonstrated behavior into learning.
  • Language Relabeling: Event-Selectable Hindsight Relabeling asks annotators to identify K=24 coherent behaviors in full videos and phrase each as a natural-language command.Annotators mark each behavior’s start and end frames instead of labeling randomly selected windows.
  • Language Relabeling: Event-selectable relabeling yields better-matched data than random-window relabeling, with fewer complex compound instructions and more compositionally directed instructions.Both strategies tend to describe contact-rich behaviors.
  • Throughput and Bottlenecks: 18.06% of raw data underwent annotation before model training, leaving 5.5x more unlabeled collected data than annotated data despite 16x as many hindsight annotators as robots.The authors identify crowdsourced language annotation, rather than robot teleoperation, as the main bottleneck.
  • Policy Learning: LAVA maps video and language to continuous actions using cross-attention, temporal transformers, and a residual MLP, while training uses deterministic behavioral cloning with mean squared error.Language acts as the query over multi-scale visual tokens, and the temporal output predicts the next action.

V. LANGUAGE-TABLE: DATASETS AND ENVIRONMENT

Language-Table releases large real and simulated language-conditioned trajectory datasets alongside a simulated environment and benchmark. The benchmark covers five task families and 696 variations, while SPL provides an efficiency-sensitive metric that tracks real-world performance ordering.

  • Datasets: Language-Table provides human-relabeled training data and underlying human-teleoperated data in both simulation and the real world.It is released as a suite of datasets together with a simulated multi-task language-conditioned control environment and benchmark.
  • Datasets: The real and simulated training datasets are an order of magnitude larger than comparable previously available datasets.The released data includes language-labeled trajectories and their underlying teleoperated collections.
  • Environment: The simulated environment models tabletop manipulation with an xArm6, a 2D-constrained cylindrical end-effector, a wooden board, and eight blocks spanning four colors and six shapes.Simulation and real collection use high-rate third-person human teleoperation.
  • Benchmark: The benchmark evaluates 5 task families across 696 unique task variations using thresholded success and Success weighted by Path Length (SPL).SPL trades off task success against the efficiency of the path used to succeed.
  • Benchmark: Policy hyperparameters ordered by SPL have thus far been ordered similarly in real-world performance, validating the simulated benchmark’s relevance to real-world robotics.This correspondence is presented as a degree of validation rather than a guarantee of transfer.

VI. POLICY RESULTS AND DISCUSSION

The policy addresses diverse short-horizon language commands and supports human-guided long-horizon rearrangements, while real-time feedback substantially improves performance over open-loop plans.

  • Short-horizon language skills: 93.5% expected average success over 87,588 unique language instructions demonstrates broad short-horizon open-vocabulary skill coverage.The estimate uses 20 randomly selected instructions and 10 trials each, with a 95% CI of [90.08%,96.92%].
  • Long-horizon goal reaching: 85.0% expected average success was achieved across 11 families of precise long-horizon rearrangement goals with real-time human guidance.The evaluation comprised 60 trials from 20 sampled goals, with a 95% CI of [69.35%,100.00%].
  • Real-time feedback: Performance deteriorated from 85% to 25% when real-time language feedback was removed, indicating its importance for contact-rich tasks.The open-loop condition committed to the full sequence and order of commands in advance.
  • Ablations: Simulation ablations compare the LAVA transformer with a FiLM-conditioned ResNet-18 baseline and vary the amount of training data.The study uses SPL because it better indicates qualitative performance than unweighted average success.
  • Multi-robot control: One operator guided four robots simultaneously, showing that short-horizon skill competency can relax the assumption of undivided operator attention.The capability was demonstrated with robots equipped with Interactive Language policies.

C. Simulation: Architecture and data ablation

Simulation ablations attribute significant SPL gains to the LAVA architecture, while increasing training data continues to help with diminishing returns.

  • Architecture: LAVA produces significant gains over the prior FiLM-conditioned ResNet architecture on SPL in the multi-task simulation benchmark.Results include confidence intervals over three seeded training runs.
  • Evaluation metric: SPL is treated as a better indicator of qualitative performance than unweighted average success in this setup.SPL incorporates path length into the success-weighted evaluation.
  • Data scaling: Performance shows diminishing returns but does not plateau across successive doublings of training data.The authors connect this scaling pattern to the complexity of open-vocabulary visuomotor learning.
  • Research assets: The work provides the Language-Table dataset and environment as assets for further research on real-time-conditionable visuomotor robots.These assets accompany the framework and analysis.

APPENDIX

The appendix describes the real-world sensing setup and Language-Table data sources, including simulated teleoperation and hindsight relabeling at substantial scale.

  • Real-world setup: Real-world policies use 320x180 single-camera RGB images and language, with asynchronous data batched into pseudo-synchronous 5 Hz training pairs.The hardware includes UFACTORY xArm6 arms and an Intel RealSense D415 camera.
  • Dataset overview: Language-Table includes datasets spanning both simulation and real-world collection.The appendix organizes the available data across these two settings.
  • Simulation data: 8,318 simulated teleoperation episodes yielded 85.5 hours of raw data from six teleoperators following long-horizon prompts.The average episode length was 36.8 ±15 seconds.
  • Simulation annotation: 181,020 hindsight-relabeled simulated trajectories contained 78,623 unique instructions after annotation by 64 crowdsourced annotators.The relabeling used the interface described in Appendix E.
  • Real-world data: 23,498 real-world episodes from 11 teleoperators across four robots yielded 3,865 hours of raw data.The average episode length was 9.9 minutes ±5.6 seconds, and 16,417 episodes entered policy training.

4) Real-World-Relabeled:

The appendix details real-world relabeled data and the Language-Table simulation benchmark, which evaluates five task families across 696 unique conditions using RGB and language inputs.

  • Real-World-Relabeled: 414,798 real-world hindsight-relabeled trajectories contained 119,959 unique instructions, with 298,782 trajectories and 87,140 instructions used for training.The data were annotated by 64 crowdsourced annotators.
  • Language-Table environment: The simulated environment models a 6DoF xArm6 with 320x180 RGB input, eight blocks, and delta 2D Cartesian end-effector actions.Policies are trained and evaluated from RGB and language input only, while the environment additionally exposes state observations.
  • Benchmark design: Five simulated evaluation families span 696 unique task conditions and use hand-defined success criteria.The families are block2block, block2abs, block2rel, block2blockrel, and separate.
  • Task families: The block2block family contains 56 conditions for pushing one block to another, with success based on thresholded source-target distance.The conditions cover 8 source blocks and 7 target blocks.
  • Task families: The block2abs family contains 72 conditions for pushing blocks to nine absolute board locations.Success is based on thresholded distance between the block and target location.
  • Task families: The block2rel family contains 64 conditions for pushing blocks to eight relative offset directions.Success uses thresholded distance to the invisible target offset location.
  • Task families: The block2blockrel family contains 448 conditions for placing a source block at one of eight offsets relative to a target block.This is the largest listed family, combining 8 source blocks, 7 target blocks, and 8 directions.

E. Event selectable hindsight relabeling details

Event-selectable hindsight relabeling turns long-horizon demonstrations into open-vocabulary action instructions, while the LAVA architecture fuses language with visual and temporal information to predict actions.

  • Event selectable hindsight relabeling details: Labelers watch each complete long-horizon video, segment 12 medium-horizon and 12 short-horizon actions, and describe them with open-vocabulary instructions.Temporal segmentation tools let labelers mark each action’s beginning and end; horizon definitions are left to rater discretion.
  • Model architecture details: LAVA combines perception, language, vision-language fusion, temporal fusion, and policy-output modules.The architecture is described as a transformer-based visuo-linguo-motor network.
  • Model architecture details: The Language-Attends-to-Vision block uses sentence embeddings as queries over flattened multi-scale visual tokens, then temporally fuses the per-frame outputs.The block applies cross-attention to each image in a four-frame context history.
  • Model architecture details: The policy average-pools the fused representation and uses a residual MLP followed by a linear projection to produce 2D actions.The residual MLP contains two residual blocks before the final action projection.

G. Training details

Policies are trained at scale with fixed optimization settings, while ablations examine language finetuning, temporal fusion, and batch size; halving batch size reduces performance after 350k steps.

  • Training details: Policies train for approximately 500,000 steps on a TPUv3 8x8 pod, taking about 18 hours at roughly 7.6 steps per second.Training uses Adam, learning rate 1e-3, batch size 4096, and normalized action labels.
  • Ablations: The ablations evaluate CLIP finetuning, temporal fusion, and batch size in the Language-Table simulator.The reported ablation results use three seeds after 350k steps.
  • Ablations: Without in-domain CLIP finetuning, Language-Table simulation performance drops only a few percent, but real-robot behavior differs much more strongly qualitatively.The pretrained text encoder remains fixed during policy training in this ablation.
  • Ablations: Reducing batch size from 4096 to 2048 causes a performance drop after 350k steps.This result is reported for the batch-size ablation.

I. Extended Related Work

The paper contrasts open-loop and closed-loop LLM subgoal planning with its focus on diverse, real-time-interactable short-horizon behaviors, especially where fine spatial detail challenges text-only planning.

  • Extended Related Work: Open-loop LLM planners generate subgoal sequences without replanning, whereas closed-loop planners update plans using textual descriptions of the current scene.Both formulations rely on language-conditioned policies and are discussed as approaches to long-horizon tasks.
  • Extended Related Work: The paper argues that purely textual scene descriptions make accurate fine-grained spatial subgoals difficult for tasks requiring precise spatial detail.Visual language models are identified as promising, but matching human perception and cognition remains difficult.
  • Extended Related Work: Its complementary focus is obtaining diverse short-horizon behaviors that can be interactively conditioned in real time, with autonomous planning proposed as future combination work.The passage positions combining autonomous long-horizon planning with the demonstrated short-horizon recipe as a strong candidate for future work.
Loading 2210.06407v1…