PaperlayerView exact revision on arXiv

Source-linked AI summary

RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, Brianna Zitkovich

arXiv:2307.15818v1cs.ROcs.CLcs.CVcs.LG

TL;DR

Robots need to combine web-scale semantic knowledge with grounded low-level control, but vision-language models and robotic actions use different output formats. RT-2 co-fine-tunes vision-language models with robotic trajectories by representing actions as text tokens, achieving stronger generalization and emergent semantic capabilities across 6k evaluations.

  • Problem

    It remains unclear how robots can acquire web-scale semantic capabilities while producing grounded low-level actions required for robotic control.

  • Method

    RT-2 co-fine-tunes vision-language models on robotic trajectories and Internet-scale vision-language tasks, representing robot actions as text tokens.

  • Results

    Across 6k evaluations, RT-2 improves generalization over objects, scenes, and instructions, averaging approximately 2x the performance of RT-1 and MOO.

  • Takeaways & Limitations

    Web-scale vision-language pretraining transfers generalizable visual and semantic concepts to robotic policies, enabling emergent instruction interpretation and rudimentary reasoning.

  • Takeaways & Limitations

    Web-scale pretraining does not add new physical motions, leaving the model’s skills limited to those represented in the robot data.

Abstract

from arXiv · show

We study how vision-language models trained on Internet-scale data can be incorporated directly into end-to-end robotic control to boost generalization and enable emergent semantic reasoning. Our goal is to enable a single end-to-end trained model to both learn to map robot observations to actions and enjoy the benefits of large-scale pretraining on language and vision-language data from the web. To this end, we propose to co-fine-tune state-of-the-art vision-language models on both robotic trajectory data and Internet-scale vision-language tasks, such as visual question answering. In contrast to other approaches, we propose a simple, general recipe to achieve this goal: in order to fit both natural language responses and robotic actions into the same format, we express the actions as text tokens and incorporate them directly into the training set of the model in the same way as natural language tokens. We refer to such category of models as vision-language-action models (VLA) and instantiate an example of such a model, which we call RT-2. Our extensive evaluation (6k evaluation trials) shows that our approach leads to performant robotic policies and enables RT-2 to obtain a range of emergent capabilities from Internet-scale training. This includes significantly improved generalization to novel objects, the ability to interpret commands not present in the robot training data (such as placing an object onto a particular number or icon), and the ability to perform rudimentary reasoning in response to user commands (such as picking up the smallest or largest object, or the one closest to another object). We further show that incorporating chain of thought reasoning allows RT-2 to perform multi-stage semantic reasoning, for example figuring out which object to pick up for use as an improvised hammer (a rock), or which type of drink is best suited for someone who is tired (an energy drink).

1. Introduction

RT-2 addresses the difficulty of transferring web-scale vision-language capabilities to robots by tokenizing robot actions as text and co-training them with Internet-scale vision-language tasks. The resulting models combine robot-learned physical skills with web-derived visual, linguistic, semantic, and reasoning capabilities, improving generalization and enabling emergent command interpretation.

  • Motivation: Web-scale models offer broad downstream capabilities, but collecting comparable robotic interaction data is unlikely and directly applying them to robots is difficult.The paper contrasts billions of web tokens and images with the impracticality of collecting millions of robotic trials.
  • Method: RT-2 tokenizes low-level robot actions as text and trains vision-language models jointly on robotic trajectories and Internet-scale visual-language tasks.This creates multimodal sentences in a shared token format, allowing one model to produce both natural-language outputs and robot actions.
  • Capabilities: RT-2 transfers web-derived visual and linguistic knowledge into robotic policies, improving generalization to novel objects and semantically varied instructions.The approach combines physical motions learned from robot data with image and text interpretation learned from web data.
  • Results: 6k robotic evaluations show significant generalization improvements and emergent semantic reasoning, including interpreting unseen commands and selecting objects by size or spatial relation.The paper also reports chain-of-thought-enabled multi-stage reasoning, such as choosing a rock as an improvised hammer or an energy drink for someone who is tired.

2. Related Work

Prior work spans vision-language modeling, robot-learning generalization, and pre-training for robotic manipulation. This work focuses on extending vision-language models to closed-loop control by predicting robot actions and leveraging broader pre-training data for generalization.

  • Vision-language models: Vision-language models include representation-learning models such as CLIP and visual-language models that map vision and text inputs to free-form text.These categories are identified as especially relevant among several VLM types.
  • Vision-language models: This work extends VLM capabilities to robotics closed-loop control by enabling them to predict robot actions and leverage existing VLM knowledge for generalization.The focus is on combining vision-language capabilities with action prediction rather than only producing text.
  • Generalization in robot learning: Large and diverse datasets have enabled robot generalization to novel object instances, novel semantic object categories, and unseen environments.Prior studies demonstrate generalization across these three axes.
  • Generalization in robot learning: Unlike most prior work, this study targets one model that generalizes across unseen conditions along all three axes by leveraging broader pre-trained data than the robot observes.The cited approach treats broad pre-training exposure as a key ingredient for unified generalization.
  • Pre-training for robotic manipulation: Pre-training for robotic manipulation has often used visual representations initialized through ImageNet supervision, data augmentation, or objectives tailored to robotic control.These methods typically initialize the encoder for the robot’s camera observations.

3. Vision-Language-Action Models

RT-2 adapts web-pretrained vision-language models into vision-language-action models by representing robot actions as output tokens and co-fine-tuning on robotics and web data. The resulting models include PaLI-X- and PaLM-E-based variants, with vocabulary constraints for executable actions and cloud deployment supporting real-time control frequencies.

  • Model family: RT-2 adapts PaLI-X and PaLM-E into VLA models ranging from billions to tens of billions of parameters.The resulting variants are named RT-2-PaLI-X and RT-2-PaLM-E.
  • Action representation: RT-2 represents each robot action dimension as a discretized output token, concatenating the tokens into a space-separated target string alongside language tokens.The action space includes 6-DoF positional and rotational end-effector displacement and extension level, with 256 tokens reserved for actions.
  • Training recipe: Co-fine-tuning robotics data with original web data improves generalizable policies by combining abstract visual concepts with low-level robot actions.Naïve fine-tuning on robot data alone does not provide this same exposure to web-scale visual concepts.
  • Output constraint: RT-2 constrains decoding to valid action tokens for robot-action tasks while retaining the full natural-language vocabulary on standard vision-language tasks.This distinction ensures outputs are executable on the real robot without restricting ordinary vision-language responses.
  • Deployment: 55B parameters is the largest model trained, with RT-2-PaLI-X-55B running at 1-3 Hz and its 5B version running at around 5 Hz.The models use a cloud service to support suitable control frequency and serve multiple robots.

4. Experiments

The experiments evaluate RT-2’s real-world generalization and emergent capabilities.

  • The experimental focus is RT-2’s real-world generalization and emergent capabilities.The experiments are organized around questions concerning these two areas.

1. How does RT-2 perform on seen tasks and more importantly, generalize over new objects, backgrounds, and environments?

RT-2 performs similarly to RT-1 on more than 200 seen tasks, while its main advantage appears in generalization to unseen objects, backgrounds, and environments. These gains suggest that Internet-scale vision-language pretraining transfers more generalizable visual and semantic concepts to robotic control.

  • Seen tasks: The evaluation covers more than 200 seen tasks spanning picking, knocking, placing, moving, and opening or closing drawers.The seen-task suite follows RT-1 and includes 36 picking, 35 knocking, 35 upright-placing, 48 moving, 18 drawer, and 36 picking-related tasks.
  • Seen tasks: RT-2 models perform similarly to RT-1 on seen tasks, while other baselines achieve lower success rates.The comparison includes RT-2-PaLI-X, RT-2-PaLM-E, RT-1, VC-1, and R3M-based policies.
  • Unseen evaluations: RT-2’s advantage is most pronounced in generalization experiments involving unseen objects, backgrounds, and environments.Evaluations distinguish easy and hard cases; hard cases use harder-to-grasp or unique objects, more varied backgrounds and novel objects, and a visually distinct office desk environment.
  • Interpretation: The generalization gains are attributed to transferring more generalizable visual and semantic concepts from Internet-scale pretraining data.The paper frames vision-language-action models as transferring web knowledge into end-to-end robotic policies.
  • Ablations: Co-fine-tuning improves generalization over fine-tuning with robotic data alone, whereas training a very large model from scratch performs poorly even at 5B parameters.The ablation compares co-fine-tuning, robotic-data-only fine-tuning, and training from scratch.

5. Limitations

RT-2’s web-scale pretraining improves semantic and visual generalization but does not expand the robot’s repertoire of physical motions. Although large VLA models can run in real time, their high computational cost may limit high-frequency control.

  • 5. Limitations: Web-scale VLM pretraining improves semantic and visual generalization, but RT-2 does not learn new motions from the additional experience.Its physical skills remain limited to the distribution of skills represented in the robot data.
  • 5. Limitations: Large VLA models can run in real time, but their high computation cost may become a major bottleneck for high-frequency control.The passage identifies quantization and distillation as possible ways to enable higher-rate inference or lower-cost hardware.

6. Conclusions · A. Contributions •

The paper presents VLAs trained by combining vision-language pretraining with robotic data, using text-token robot actions in two RT-2 instantiations. This approach produces performant policies, better generalization, and emergent capabilities inherited from web-scale vision-language pretraining.

  • 6. Conclusions: The authors propose training vision-language-action models by combining vision-language model pretraining with robotic trajectory data.The approach incorporates robotic actions into the model’s output format as text tokens.
  • 6. Conclusions: RT-2-PaLM-E and RT-2-PaLI-X are two VLA instantiations cofine-tuned to output robot actions represented as text tokens.
  • 6. Conclusions: The resulting models produce very performant robotic policies while achieving significantly better generalization performance.
  • 6. Conclusions: The models acquire emergent capabilities inherited from web-scale vision-language pretraining.
  • 6. Conclusions: The paper argues that this simple and general approach offers a promising route for robotics to benefit directly from improved vision-language models.
  • A. Contributions •: The work’s contributions include designing and executing training procedures, evaluating models in simulation and the real world, and running ablations for algorithm design choices.

B. Datasets … F. Evaluation Details

The paper combines large-scale multilingual vision-language data with mobile-manipulation demonstrations, co-fine-tunes several multimodal architectures, and evaluates against robotics and visual foundation baselines. Training mixtures, task formulations, and model-specific configurations vary across RT-2 variants and Language-Table experiments.

  • B. Datasets: 1B WebLI image-text pairs, filtered from around 10B examples across 109 languages, form the bulk of the vision-language training data.The mixture also includes captioning and visual question answering datasets.
  • B. Datasets: Mobile-manipulation demonstrations are annotated with natural-language instructions spanning seven listed skills, including object manipulation, drawer operation, and receptacle placement.The robotics dataset is based on Brohan et al. (2022).
  • B. Datasets: RT-2-PaLI-X assigns the robotics dataset about 50% of the training mixture, while RT-2-PaLM-E assigns it about 66%.These weights are used for co-fine-tuning.
  • B. Datasets: Language-Table co-fine-tuning predicts actions, instructions, arm positions, timestep counts, and task success from image frames and text instructions.The five prediction tasks include action prediction from two consecutive frames and a text instruction.
  • C. Baselines: Baselines use the exact same robotic data and include RT-1, which tests the value of VLM pre-training, and VC-1, a robotics-oriented visual foundation model.RT-1 is described as a transformer-based model, while VC-1 provides pre-trained visual representations designed for robotics tasks.
  • D. VLMs for RT-2: PaLI-X processes image sequences with ViT-22B and feeds projected image tokens into a 32B-parameter, 50-layer encoder-decoder backbone.The backbone jointly processes text and image embeddings to generate output tokens auto-regressively.
  • D. VLMs for RT-2: PaLM-E uses a decoder-only LLM with ViT-4B visual projections, while PaLI-3B uses ViT-G/14 and UL2-3B for Language-Table.PaLM-E accepts multimodal inputs by projecting robot data into language-token space and concatenating continuous variables with textual input.

F.1. Evaluation Scenarios · F.2. Evaluation Instructions

RT-2 is evaluated on challenging semantic scenarios targeting reasoning, symbol understanding, and human recognition. The evaluation uses natural-language instructions for unseen settings and quantitative emergent capabilities, with quantitative instructions repeated five times.

  • F.1. Evaluation Scenarios: The evaluation scenarios measure emergent capabilities in reasoning, symbol understanding, and human recognition.These scenarios are designed for quantitative study of RT-2’s emergent capabilities.
  • F.2. Evaluation Instructions: Table 2 contains natural-language instructions for evaluations involving unseen objects, backgrounds, and environments.These instructions cover generalization across unseen visual and environmental conditions.
  • F.2. Evaluation Instructions: Each unseen-object, background, and environment instruction was run 1-5 times, depending on the total number of instructions in its evaluation set.The number of repetitions varied with evaluation-set size.
  • F.2. Evaluation Instructions: Table 3 lists the natural-language instructions used for quantitative emergent evaluations.These instructions correspond to the quantitative semantic scenarios described for RT-2.
  • F.2. Evaluation Instructions: Each quantitative emergent evaluation instruction was run 5 times.The repetition count is fixed for the quantitative evaluation set.
  • F.1. Evaluation Scenarios: Figure 8 visualizes a subset of the evaluation scenes, while the full quantitative instruction list appears in Table 3.The visualized instructions are only a subset of the complete instructions listed in Appendix F.2.

G. Example Failure Cases · H. Quantitative Experimental Results · H.1. Overall Performance, for Section 4.1

RT-2 shows strong overall generalization, matching or exceeding baselines on seen tasks and outperforming them on several unseen settings. However, it still fails on unseen object dynamics and other real-world manipulation cases.

  • G. Example Failure Cases: RT-2 fails to generalize to unseen object dynamics in some Language Table cases despite correctly attending to instructions and reaching the first target object.The failure arises from difficulty controlling dynamics that differ substantially from those represented in training.
  • G. Example Failure Cases: The authors hypothesize that scaling datasets across diverse environments and objects could improve generalization to robot-environment interaction dynamics.They specifically suggest including more diverse examples of similar pushing dynamics.
  • G. Example Failure Cases: Despite promising qualitative and quantitative emergent performance, RT-2 exhibits numerous notable failure cases under its current training dataset composition and training method.The paper introduces these cases as a limitation of the current system.
  • H.1. Overall Performance, for Section 4.1: RT-2 performs as well as or better than baselines on seen tasks.This summarizes the overall comparison reported in Table 4.
  • H.1. Overall Performance, for Section 4.1: RT-2 significantly outperforms baselines when generalizing to unseen objects, backgrounds, and environments.These comparisons are part of the quantitative overall evaluation results.
  • H. Quantitative Experimental Results: Table 4 reports quantitative overall results for two RT-2 instantiations and baselines across seen training tasks and unseen generalization evaluations.The unseen evaluations measure generalization to novel objects, novel backgrounds, and novel environments.

H.2. Emergent Evaluation, for Section 4.2 · H.3. Size and Training Ablations, for Section 4.3

RT-2 outperforms RT-1 on emergent instructions without additional robotic demonstrations, leveraging web-scale vision-language pretraining. Ablations show that larger models and co-fine-tuning improve performance over smaller models and alternative training strategies.

  • H.2. Emergent Evaluation, for Section 4.2: RT-2 performs 2x to 3x better than RT-1 on new instructions without additional robotic demonstrations.These results are reported in Table 5’s quantitative emergent evaluations.
  • H.2. Emergent Evaluation, for Section 4.2: The emergent evaluation results show that RT-2 leverages capabilities from web-scale vision-language pretraining.The paper presents these findings as evidence that pretraining benefits robotic control.
  • H.2. Emergent Evaluation, for Section 4.2: Table 5 reports quantitative emergent-evaluation performance for RT-2 and baselines.The table organizes the comparison underlying the reported emergent results.
  • H.3. Size and Training Ablations, for Section 4.3: Model size plays an important role in performance across the ablations.This conclusion is drawn from the quantitative results detailed in Table 6.
  • H.3. Size and Training Ablations, for Section 4.3: Co-fine-tuning outperforms fine-tuning, which outperforms training from scratch.The ordering is reported across the training-approach ablations.
  • H.3. Size and Training Ablations, for Section 4.3: Table 6 reports ablations across model size and training approach.The ablations examine parameter count and training strategy in relation to generalization.

I. Additional Chain-Of-Thought Reasoning Results

This section presents additional chain-of-thought reasoning rollouts from RT-2-PaLM-E, illustrated in Figure 10. The accompanying note clarifies that relevant PaLM-E pretraining examples lacked the low-level actions evaluated here.

  • Additional Chain-Of-Thought Reasoning Results: Additional chain-of-thought reasoning rollouts were obtained with RT-2-PaLM-E.These examples are presented in Figure 10.
  • Additional Chain-Of-Thought Reasoning Results: PaLM-E-12B pretraining included robot images for high-level VQA planning tasks, but none of those examples included the low-level actions evaluated here.The note states that those images could resemble generalization scenarios while distinguishing them from the evaluated low-level actions.
Loading 2307.15818v1…