PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents

Michael Ahn, Debidatta Dwibedi, Chelsea Finn, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Karol Hausman, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Sean Kirmani, Isabel Leal, Edward Lee, Sergey Levine, Yao Lu, Isabel Leal, Sharath Maddineni, Kanishka Rao, Dorsa Sadigh, Pannag Sanketi, Pierre Sermanet, Quan Vuong, Stefan Welker, Fei Xia, Ted Xiao, Peng Xu, Steve Xu, Zhuo Xu

arXiv:2401.12963v2cs.ROcs.AIcs.CLcs.CVcs.LG

TL;DR

Embodied foundation models need much more real-world robotic experience than lab datasets provide. AutoRT uses foundation models to orchestrate robot task proposals and collection across a fleet. Across more than 20 robots, it collected 77,000 trials; training on the collected data demonstrated novel capabilities and improved generalization over state-of-the-art models.

  • Problem

    Training broadly capable embodied agents requires far more real-world robotic experience than datasets collected in well-defined lab settings provide.

    Source: §1 · p. 1

  • Method

    AutoRT uses foundation models to propose tasks from visual observations and user prompts, then accounts for robot capabilities and human-teleoperator availability when directing collection.

    Source: §1 · p. 2

  • Results

    AutoRT collected 77,000 real-world trials with over 20 robots, and training on the collected data demonstrated novel capabilities and improved generalization over state-of-the-art models.

    Source: §1 · p. 2, §6 · pp. 9–10

  • Takeaways & Limitations

    The authors present a robot constitution as a concrete application of rules for robot behavior and identify joint model improvement and data collection as a future direction.

    Source: §6 · p. 10

    When scripted or learned collection policies handle only simpler tasks or succeed less often in unseen settings, successful-episode throughput falls.

    Source: §6 · p. 10

Abstract

from arXiv · show

Foundation models that incorporate language, vision, and more recently actions have revolutionized the ability to harness internet scale data to reason about useful tasks. However, one of the key challenges of training embodied foundation models is the lack of data grounded in the physical world. In this paper, we propose AutoRT, a system that leverages existing foundation models to scale up the deployment of operational robots in completely unseen scenarios with minimal human supervision. AutoRT leverages vision-language models (VLMs) for scene understanding and grounding, and further uses large language models (LLMs) for proposing diverse and novel instructions to be performed by a fleet of robots. Guiding data collection by tapping into the knowledge of foundation models enables AutoRT to effectively reason about autonomy tradeoffs and safety while significantly scaling up data collection for robot learning. We demonstrate AutoRT proposing instructions to over 20 robots across multiple buildings and collecting 77k real robot episodes via both teleoperation and autonomous robot policies. We experimentally show that such "in-the-wild" data collected by AutoRT is significantly more diverse, and that AutoRT's use of LLMs allows for instruction following data collection robots that can align to human preferences.

1 INTRODUCTION

AutoRT addresses the need for far more real-world robotic experience than lab datasets provide by orchestrating robots that propose and carry out tasks. Its evaluation collected 77,000 trials across more than 20 robots, with both teleoperated and autonomous execution.

1 INTRODUCTION

  • The central bottleneck is obtaining much more real-world robotic experience than lab datasets provide for agents expected to handle high-level goals across diverse settings.Such agents must formulate plans and carry them out with available skills and resources.Source: §1 · p. 1
  • AutoRT lets robots assess feasible tasks, choose available skills, and decide when to rely on human teleoperators.The system is designed for a fleet spread across locations with more robots than human supervisors.Source: §1 · p. 2
  • AutoRT uses a foundation-model orchestrator to propose tasks from user prompts and visual observations, while accounting for teleoperator availability and robot capabilities.Its evaluation examines diverse, task-appropriate data and preference alignment through prompting and critiquing with a robot constitution.Source: §1 · p. 2
  • Across 7 months, 4 office buildings, and over 20 robots, AutoRT collected 77,000 real-world trials using teleoperation and autonomous execution.The evaluation also reports supervision at a scale of one person per 3–5 mobile manipulators.Source: §1 · p. 2

2 RELATED WORK

Prior work covers autonomous collection in constrained settings and human-assisted collection in varied environments, alongside LLM-driven planning and agent behavior. AutoRT combines autonomous and teleoperated collection for novel tasks in real-world environments.

2 RELATED WORK

  • Prior autonomous collection commonly targets grasping, pushing, or pick-and-place in constrained labs, while human demonstrations can cover varied environments but scale poorly with human availability.AutoRT combines teleoperated and autonomous episodes according to the available supervision.Source: §2 · p. 2
  • Unlike LLM work that generates plans for user-given instructions or explores simulated worlds, AutoRT generates robot instructions for real-world execution, where reliability and safety matter.The passage identifies Xian et al. (2023) as prior work proposing self-generated instructions and Voyager as the closest comparison.Source: §2 · p. 2

3 PROBLEM STATEMENT

The problem is to collect diverse real-world robot data for new skills in unfamiliar environments with limited human supervision. A single system must map observed states to feasible language tasks and collection policies while balancing throughput and safety.

3 PROBLEM STATEMENT

  • The goal is large-scale, in-the-wild collection of diverse real-world robot data for new skills in new environments.Source: §3 · p. 2
  • The setting assumes a mobile, object-manipulating fleet in populated buildings, no prior knowledge of layouts or objects, and more robots than human supervisors.Source: §3 · p. 3
  • The system must map each observed state to feasible language tasks and executable collection policies, accounting for throughput, task diversity, and safety.Policies may be autonomous or human-teleoperated.Source: §3 · p. 3

4 AUTORT: EXPLORING AND EXECUTING IN THE WILD

AutoRT explores unfamiliar environments, uses foundation models to propose and filter manipulation tasks, and selects collection policies subject to robot rules and available supervision. Its pipeline includes additional environment controls because foundation models have no safety guarantees.

4 AUTORT: EXPLORING AND EXECUTING IN THE WILD

  • AutoRT describes scenes with an open-vocabulary object detector, uses an LLM to generate sensible and safe goals, then uses another LLM to determine execution.Source: §4 · p. 3

4.1 EXPLORATION: NAVIGATING TO THE TARGET

  • Robots navigate toward sampled regions of interest using a natural-language map that connects VLM object embeddings with positions from depth sensing and SLAM.The map is generated once per environment and shared across robots collecting there.Source: §4.1 · p. 3

4.2 ROBOT CONSTITUTION

  • The Robot Constitution gives the LLM rules for safe task generation, including foundational rules, safety rules, embodiment limits, and optional human guidance.Safety and embodiment rules discourage, among other things, interactions with people or animals and tasks beyond the robot’s payload or single-arm capabilities.Source: §4.2 · pp. 3–4

4.3 TASK GENERATION

  • For task generation, a VLM describes the scene and five objects, then an LLM proposes manipulation tasks using the constitution and scene description.The task-generation prompt is adjusted to match the sampled collection policy; the LLM is not fine-tuned for this use case.Source: §4.3 · p. 4

4.4 AFFORDANCE

  • A prompted LLM filters proposed tasks by assigning each to a suitable collection policy or rejecting it, then randomly selects among accepted tasks.The filter considers policy capabilities and can reject tasks that exceed the robot’s embodiment.Source: §4.4 · p. 5

4.5 DATA COLLECTION

  • Episodes sample among collection policies, score diversity, and reset; the system prioritizes teleoperation throughput to retain action diversity.The instance uses teleoperation, scripted picking, and RT-2; the teleoperation sampling probability reflects human-supervision availability.Source: §4.5 · p. 5

4.6 GUARDRAILS

  • Because foundation models lack safety guarantees, AutoRT adds traditional robot-environment controls as another safety layer.Source: §4.6 · p. 5

5 EXPERIMENTAL EVALUATION

Across a seven-month, four-building deployment, AutoRT collected a large multi-policy robot dataset and evaluated its diversity, task generation, safety filtering, and model-training utility.

5 EXPERIMENTAL EVALUATION

  • 53 robots collected 77,000 episodes over seven months, with a peak of over 20 simultaneous robots and more than 6,650 unique instructions.Each human supervised three to five robots at once; stationary robots could be watched at a ratio of up to one human to eight robots.Source: §5 · pp. 5–6

5.1 DIVERSITY SCORING

  • AutoRT task instructions had higher average language-embedding distances than hand-designed tasks from Language Table, BC-Z, and RT-1.Language diversity was measured using L2 distances between normalized 512-dimensional Universal Sentence Encoder embeddings.Source: §5.1 · p. 7, Table 2 · p. 7
  • AutoRT data was more visually diverse than RT-1 across collection categories, with teleoperation scoring higher than the scripted policy.Visual diversity was measured by distance to the nearest k-means centroid; higher distances indicate greater novelty relative to prior data.Source: §5.1 · p. 7, Figure 5 · p. 8

5.2 TASK GENERATION

  • AutoRT’s guided and unguided tasks were 1.5x more likely to be feasible than templated language, and high-level guidance steered tasks toward requested themes.The evaluation generated 75 tasks across five scenes; Table 3 compares feasibility and relevance across task-generation methods.Source: §5.2 · pp. 7–8, Table 3 · p. 8

5.3 AFFORDANCE AND ROBOT CONSTITUTION

  • After affordance filtering, 200/214 tasks were acceptable (93%); the LLM rejected 17/31 unsuitable tasks (55%), with all 14 misses during teleoperation sampling.Teleoperators rejected the missed tasks during collection, providing a human-in-the-loop safety check and intervention data.Source: §5.3 · pp. 8–9
  • Constitutional prompting increased the rate of safe tasks, with the best results when used in both task generation and affordance filtering.The test used deliberately adversarial scenes; constitutional prompting also achieved high recall on unsafe tasks.Source: Table 4 · p. 9, §5.3 · p. 9

5.4 MODEL TRAINING

  • 0% to 12.5% on picking from different heights and 10% to 30% on wiping after RT-1 was co-fine-tuned with AutoRT data.The gains were modest; in a teleoperation-only ablation, the model could no longer pick from different heights.Source: Table 5 · p. 9, §5.4 · p. 9

6 CONCLUSION, LIMITATIONS, AND FUTURE WORK

AutoRT combines robot-fleet data collection with policy and navigation mechanisms, while identifying limitations in policy robustness, reasoning, data sparsity, and safety compliance. The authors point to coordinated model improvement and data collection as a future direction.

6 CONCLUSION, LIMITATIONS, AND FUTURE WORK

  • 77k real-world demonstrations collected by over 20 robots in 7 months across 4 buildings, using autonomous execution and human help.The system also introduces a robot constitution defining foundational rules, safety constraints, and embodiment.Source: §6 · p. 9
  • AutoRT relies on scripted and learned policies, and lower success on unseen tasks reduces successful-episode throughput; diverse data also leaves fewer samples per task.The authors additionally cite perception bottlenecks, weak task- and embodiment-specific reasoning, and the fact that prompting does not guarantee safety-rule compliance.Source: §6 · p. 10
  • The authors propose treating model improvement and data collection as one goal, using proximal skills and directed collection to improve sample efficiency.The system assumes data collection is decoupled from the control policy, while the authors say the two would likely need to evolve in tandem for best policy improvement.Source: §6 · p. 10

A ROBOT AND SYSTEM SETUP

  • Each robot is a mobile manipulator that explores bounded regions without advance object-location information; its policy graph links subpolicies through state-and-data-dependent transitions.Objects were also scattered in environments to broaden the task-generation coverage that otherwise could be biased toward what a robot sees naturally.Source: A ROBOT AND SYSTEM SETUP · p. 15

B NAVIGATION SAMPLING

  • Navigation targets are scored against an average text embedding of object names; sampling temperature β=1 maintained variation, while larger β was recommended for targeted collection.The object-name list was gathered once and left unchanged during the project.Source: B NAVIGATION SAMPLING · pp. 15–16

C GUARDRAILS

AutoRT pairs physical and human oversight with constitution-based prompts and task-mode choices. Its prompt variants also include versions that remove rules or encourage unsafe proposals for adversarial experiments.

C GUARDRAILS

  • Robots pause when joint force exceeds a threshold, can be disengaged by physical E-stop, and require line-of-sight human supervision unless the workspace is barricaded.Unsafe objects are proactively removed, and human experts sanity-check generated tasks during demonstrations.Source: C GUARDRAILS · p. 16

D PROMPTS

  • The prompts supply foundational, safety, embodiment, and optional guidance rules, including restrictions on harm, hazardous objects, heavy lifting, and tasks requiring two arms.The example task prompts request either creative manipulation tasks or, in a separate format, pick-only tasks.Source: D PROMPTS · pp. 16–17
  • The robot prompt offers scripted pick, teleoperation, RT-2, or rejection modes, with examples assigning tasks according to capabilities and constraints.For example, peeling a banana is rejected as bimanual, while closing a drawer is assigned to RT-2.Source: D PROMPTS · pp. 18–19

D.1 PROMPTS FOR ADVERSARIAL EXPERIMENTS

  • Adversarial prompt variants remove constitutional rules, and one teleoperation prompt explicitly encourages unsafe tasks for the affordance filter to capture.A minimal affordance-filtering prompt also omits the constitution rules.Source: §D.1 · pp. 19–20

E OPTIMIZING VISUAL DIVERSITY

A pilot used spoken visual-diversity scores as feedback while a human adjusted scenes between collection episodes. The resulting scenes included more distractors and unconventional object arrangements.

E OPTIMIZING VISUAL DIVERSITY

  • The pilot’s score feedback led the supervisor to add distractors and create askew tables and unconventional object arrangements, enabling faster iteration during collection.The figure shows example environments before and after adjustment based on visual diversity.Source: Figure 7 · p. 21, E OPTIMIZING VISUAL DIVERSITY · p. 21

F MODEL IMPROVEMENT EVALUATION TASKS

The evaluation sampled manipulation tasks for picking at different heights and for wiping with a table, sponge, and cloth. Each task was attempted twice.

F MODEL IMPROVEMENT EVALUATION TASKS

G QUALITATIVE EXAMPLES

The examples show AutoRT comparing VLM-generated scene descriptions and task-generation approaches, including relevance and feasibility tests and constitutional prompting. They illustrate both generated manipulation instructions and rejections tied to safety rules.

G QUALITATIVE EXAMPLES

  • Table 7 compares generated tasks using two VLMs for scene and nearby-object descriptions; FlexCap gives more descriptive object descriptions, particularly for color.Source: Table 7 · p. 22
  • Table 8 presents Section 5.2 examples testing relevance and feasibility, including generated and teleoperated instructions for manipulating fire extinguishers and scissors.Source: Table 8 · p. 23, G QUALITATIVE EXAMPLES · p. 24
  • Table 9 compares tasks from adversarial testing across unsafe-task generation, minimal affordance, and constitutional affordance prompts.The examples include rejections for heavy objects and sharp-object interactions.Source: Table 9 · p. 24, G QUALITATIVE EXAMPLES · p. 24
  • The examples contrast proposed actions with rejections: moving a fire extinguisher is shown as a task, while lifting a table is rejected as too heavy and scissors use is rejected under a sharp-object rule.Source: G QUALITATIVE EXAMPLES · p. 24

H SCRIPTED PICK

The scripted picking policy locates a named object, moves toward it, closes the gripper, and lifts it. The authors report that an LLM-generated first draft needed hand edits for edge cases and long-running collection.

H SCRIPTED PICK

  • The LLM-generated first draft was edited by hand to improve comments and robustness in edge cases for long-running data collection.The authors say early code-generation explorations produced good first attempts that often missed edge cases.Source: H SCRIPTED PICK · p. 25
  • The policy looks up the named object and stops the episode if it cannot find it.Source: H SCRIPTED PICK · p. 25
  • After a pick, the policy stops when within 0.1 of the raised target or when the robot has not moved for five timesteps.Source: H SCRIPTED PICK · p. 25
  • The policy stops early when lifting to 25 cm above the object is infeasible near the robot’s kinematic limit.Source: H SCRIPTED PICK · p. 25
  • When close to the target, the policy closes the gripper and marks the object picked, then raises the target pose by 25 cm.It closes incrementally within 0.05 distance and fully closes within 0.02 distance or after 10 unmoved timesteps.Source: H SCRIPTED PICK · p. 25

I TRAJECTORY DIVERSITY

The trajectory examples contrast scripted and teleoperated motion, with teleoperation described as more diverse. AutoRT also produced a small increase in teleoperation throughput by reducing manual resets.

I TRAJECTORY DIVERSITY

  • Teleoperated trajectories are described as more diverse than scripted-motion trajectories.Source: Figure 8 · p. 26
  • AutoRT produced a small increase in teleoperation throughput by using fewer manual resets than typical collection.Robots could navigate to a new scene rather than wait for a reset; the collection aimed to exceed a one-person-to-one-robot baseline.Source: Figure 9 · p. 26
Loading 2401.12963v2…