Source-linked AI summary
RT-1: Robotics Transformer for Real-World Control at Scale
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, Brianna Zitkovich
TL;DR
Robotics lacks the broad, task-agnostic training evidence that has supported general models in other domains, despite robotics’ need for strong generalization. RT-1 combines large, diverse real-robot demonstrations with an efficient Transformer architecture and shows strong performance across instructions and new scenarios. Its scope remains limited by imitation learning, compositional instruction generalization, and relatively non-dexterous tasks.
Problem
Robotics still lacks evidence that broad, task-agnostic training and high-capacity models can support generalization despite the difficulty of collecting real-world robotic data.
Method
RT-1 combines an efficient Transformer architecture with a broad dataset of approximately 130k demonstrations collected from 13 robots over 17 months.
Results
RT-1 performs over 700 instructions at 97% success rate and generalizes to new tasks, objects, and environments better than previously published baselines.
Takeaways & Limitations
RT-1 can absorb heterogeneous simulation and other-robot data without sacrificing original-task performance while improving generalization to new scenarios.
Takeaways & Limitations
RT-1 cannot yet generalize to completely new motions and is evaluated on a large but not very dexterous set of manipulation tasks.
Abstract
from arXiv · showhide
By transferring knowledge from large, diverse, task-agnostic datasets, modern machine learning models can solve specific downstream tasks either zero-shot or with small task-specific datasets to a high level of performance. While this capability has been demonstrated in other fields such as computer vision, natural language processing or speech recognition, it remains to be shown in robotics, where the generalization capabilities of the models are particularly critical due to the difficulty of collecting real-world robotic data. We argue that one of the keys to the success of such general robotic models lies with open-ended task-agnostic training, combined with high-capacity architectures that can absorb all of the diverse, robotic data. In this paper, we present a model class, dubbed Robotics Transformer, that exhibits promising scalable model properties. We verify our conclusions in a study of different model classes and their ability to generalize as a function of the data size, model size, and data diversity based on a large-scale data collection on real robots performing real-world tasks. The project's website and videos can be found at robotics-transformer1.github.io
1 INTRODUCTION
RT-1 addresses the difficulty of general-purpose robot learning by combining broad real-world data with a high-capacity architecture designed for efficient control. Its experiments show strong performance across many instructions and improved generalization to new scenarios.
- Motivation: Robotics datasets are typically narrow and task-specific, unlike the broad pretraining datasets that enabled general models in vision and NLP.This limits interplay among tasks and motivates task-agnostic robotic learning.
- Challenges: Building general robotic models requires both broad, sufficiently diverse datasets and architectures capable of absorbing heterogeneous data.The paper identifies dataset assembly and model design as the two central challenges.
- Approach: RT-1 encodes images, language instructions, and motor commands into compact tokens, enabling Transformer inference fast enough for real-time control.The architecture is designed to handle high-dimensional inputs and outputs efficiently.
- Results: RT-1 performs over 700 training instructions at 97% success rate and generalizes to new tasks, distractors, and backgrounds 25%, 36%, and 18% better than the next best baseline, respectively.It also executes SayCan tasks with as many as 50 stages and can incorporate simulation or other-robot data while retaining original-task performance.
2 RELATED WORK
RT-1 builds on prior language-conditioned and multitask robotic learning while extending Transformer policies toward broad, robust real-world manipulation. The paper distinguishes this focus from earlier systems with narrower task coverage or per-task learning.
- Transformer policies: Transformer policies use language commands to specify and generalize to robotic tasks, and RT-1 frames vision-language-to-action prediction as sequence modeling.This extends Transformer use beyond language conditioning alone.
- Scope of prior work: Earlier large-scale policies such as Gato and Behavior Transformer were limited in the breadth of their real-world robotic tasks.The cited comparison contrasts RT-1’s emphasis on generalizable real-world manipulation at scale with narrower systems.
- Foundations: RT-1 builds on a longer robotics literature spanning grasping, language understanding, goal reaching, and multitask or parameterized policy learning.These foundations include both pipelined and end-to-end approaches.
3 PRELIMINARIES
The paper formulates language-conditioned robot learning as sequential decision-making from visual observations and explains Transformers and imitation learning as core preliminaries. Policies generate actions iteratively until task termination.
- Robot learning: At each timestep, a policy receives a language instruction and image history, then samples an action distribution to control the robot.The interaction continues iteratively until a termination condition is achieved.
- Robot learning: An episode contains the instruction and the sequence of observations and actions from the initial timestep through termination, receiving a binary success reward.The learning objective is to maximize expected average reward over instructions, starting states, and transition dynamics.
- Transformers: A Transformer maps an input token sequence to outputs using self-attention layers and fully connected neural networks.RT-1 uses this sequence model to parameterize the policy.
- Imitation learning: Imitation learning trains a policy from successful demonstration episodes using behavioral cloning to minimize the negative log-likelihood of demonstrated actions.The demonstrations include images and language instructions paired with actions.
4 SYSTEM OVERVIEW
RT-1 is an efficient, language-conditioned architecture that converts visual and textual inputs into compact tokens and discretized robot actions. It is designed for closed-loop, real-time control across mobile-manipulation tasks.
- Robot and data: The system uses mobile manipulators with seven-degree-of-freedom arms, two-fingered grippers, and mobile bases in kitchen-based environments.Training data consists of human-provided demonstrations annotated with textual descriptions and grouped into skills and objects.
- Architecture: RT-1 uses short image sequences and natural-language instructions to produce an action at each timestep.The policy operates in closed loop for practical robotic control.
- Architecture: A pretrained convolutional network, FiLM conditioning, Token Learner, and Transformer convert image-text inputs into compact representations and discretized action tokens.This pipeline combines pretrained visual processing with language conditioning and temporal attention.
- Action space: The action space includes seven arm dimensions, three base dimensions, and a discrete mode for arm control, base control, or episode termination.Arm dimensions are x, y, z, roll, pitch, yaw, and gripper opening; base dimensions are x, y, and yaw.
5 RT-1: ROBOTICS TRANSFORMER
RT-1 combines language-conditioned visual tokenization, token compression, and autoregressive action prediction in a Transformer controller designed for real-time robotic operation. It is trained and evaluated across diverse instructions, skills, objects, environments, and large-scale demonstrations.
- Model architecture: RT-1 takes six-image histories and language instructions, conditioning EfficientNet-B3 visual features with Universal Sentence Encoder embeddings through FiLM layers.The image-language tokenizer produces 81 vision-language tokens from the EfficientNet representation.
- Model architecture: TokenLearner reduces each image’s 81 visual tokens to 8, yielding 48 history tokens for an 8-layer decoder-only Transformer.The Transformer has 19M parameters and outputs action tokens.
- Model architecture: RT-1 discretizes arm, base, and mode actions into 256 bins and trains their causal sequence prediction with categorical cross-entropy.The action space includes seven arm variables, three base variables, and a three-mode control variable.
- Real-time inference: Real-time control requires at least 3Hz operation and less than 100ms model inference, motivating token reduction and reuse across overlapping windows.The two inference-speed techniques provide measured speedups of 2.4 and 1.7 times, respectively.
- Data collection: The primary dataset contains approximately 130k demonstrations collected by 13 robots over 17 months across office-kitchen robot classrooms.The collection spans multiple tasks, objects, and environments to support generalization and robustness to distractors and backgrounds.
- Skills and instructions: RT-1 performs over 700 language instructions spanning skills such as picking, placing, drawer manipulation, upright placement, and opening jars.Object diversity was expanded especially for picking to test generalization across varied objects.
6 EXPERIMENTS
RT-1 is evaluated against Gato and BC-Z across seen-task performance, zero-shot generalization, robustness, realistic kitchen scenarios, heterogeneous data, and long-horizon execution. Across these evaluations, RT-1 shows strong performance and generalization, while results also expose the importance of dataset breadth and robot diversity.
- Experimental comparisons: RT-1 is compared with Gato, BC-Z, and BC-Z XL using the same collected data, isolating differences between model architectures.The evaluation covers overall performance, generalization, and robustness rather than differences in task sets or datasets.
- Seen and unseen tasks: 97% of more than 200 seen-task instructions are performed successfully by RT-1.Seen-task evaluation still varies object placement and other environmental factors, requiring generalization to realistic variability.
- Seen and unseen tasks: 76% of never-before-seen instructions are performed successfully, 24% more than the next-best baseline.The unseen instructions recombine previously observed skills and objects in novel ways.
- Robustness: RT-1 successfully executes 83% of distractor-robustness tasks and 59% of background-robustness tasks, exceeding the next-best alternative by 36% and 18%.Background tests use new kitchens, different lighting and visual backgrounds, and changed counter surfaces.
- Realistic scenarios: RT-1 is the most robust model across realistic kitchen generalization levels L1, L2, and L3.L1 changes layout and lighting; L2 adds unseen distractor objects; L3 adds substantially new settings, objects, or locations.
- Heterogeneous data: Mixing RT-1 and Kuka data yields 39% bin-picking performance versus 22% with RT-1 data alone, while Kuka-only transfer to the Everyday Robots morphology achieves 0%.The mixed model achieves this without explicit Everyday Robots bin-picking demonstrations, indicating benefits from multi-robot experience.
- Long-horizon scenarios: RT-1 shows no visible Kitchen1-to-Kitchen2 manipulation drop and supports ultra-long-horizon tasks of up to 50 steps.Kitchen2 is a more challenging generalization scene because training scenes resemble Kitchen1.
7 CONCLUSIONS, LIMITATIONS AND FUTURE WORK
RT-1 scales with data quantity and diversity, achieving broad task performance and generalization while supporting long-horizon execution. The authors identify limitations in imitation learning, compositional generalization, and task dexterity, and propose broader data and faster skill scaling as future directions.
- Conclusions: RT-1 scales with data quantity and diversity while absorbing heterogeneous simulation and robot-morphology data without sacrificing original-task performance.The model was trained on over 130k episodes collected over 17 months with 13 robots.
- Conclusions: 97% success rate covers over 700 instructions, while long-horizon SayCan tasks reach as many as 50 steps.These results combine broad instruction coverage with execution of extended task sequences.
- Limitations: RT-1 remains limited by imitation learning, which may prevent it from surpassing demonstrator performance.This is presented as a limitation inherited from the imitation-learning approach.
- Limitations: Generalization to new instructions is limited to combinations of previously seen concepts and does not yet extend to completely new motions.The limitation concerns compositional instruction generalization rather than all forms of task variation.
- Limitations and Future Work: The evaluated manipulation tasks are large in number but not very dexterous.The authors plan to extend the instruction set that RT-1 enables and generalizes to.
- Future Work: Future work targets faster skill scaling through directed data collection and model prompting, alongside improved environment diversity, reaction speed, and context retention.The authors specifically identify greater environment diversity as a route to improving robustness to backgrounds and environments.
C.2 DATA COLLECTION AT SCALE.
RT-1’s data-collection and evaluation framework combines teleoperation, randomized scenes, a model card, and large-scale real-world testing. The model is trained with natural-language tasks and images, evaluated across task performance and robustness, and remains bounded in deployment scope.
- Data Collection: Robots autonomously approach stations, communicate demonstration instructions, and use software to sample instructions and randomize background configurations.This process supports balanced data collection and randomized scenes.
- Data Collection: Demonstrations use two virtual-reality remotes, mapping arm, gripper, and mobile-base controls into the policy action space.The mapping preserves transition-dynamics consistency during teleoperation.
- Model Card: RT-1 is a Transformer-based imitation-learning model that takes natural-language tasks and images as inputs and outputs robot actions.Its architecture combines FiLM-conditioned EfficientNet, TokenLearner, and a Transformer.
- Metrics: Evaluation measures seen and unseen task performance, background and distractor robustness, and long-horizon success across randomized scenes.Each metric measures success on natural-language-specified tasks with varied objects, locations, and scenes.
- Training Data: The model is trained on 130k tele-operation demonstrations spanning 13 robots and 744 tasks.The model card identifies this as the training-data scale.
- Caveats: RT-1 shows high performance and robustness while learning from heterogeneous data, but its suitability outside the current research setting has not been evaluated.The model card also limits intended use to Everyday Robot manipulation tasks.
C.4 DATA COLLECTION PROCESS
As data collection grows, RT-1’s number of supported tasks and seen-task performance increase over time. The paper treats simultaneous growth in data and capability as an important future scaling target.
- Data Collection Process: The number of tasks and instructions supported by RT-1 grows as more data is collected.Figure 9 tracks data growth, task count, and policy performance over time.
- Data Collection Process: Seen-task performance also increases over time alongside data and task growth.The authors identify faster growth of data, robot performance, and general capabilities as future work.
D.1 EVALUATION DETAILS
The evaluation framework tests RT-1’s zero-shot generalization across increasingly difficult task, distractor, background, and realistic-instruction shifts. These procedures compare performance in familiar settings with novel kitchens, objects, lighting, and object locations.
- Evaluation Details: Zero-shot evaluations use standardized procedures with incrementally increasing difficulty to compare RT-1 ablations and baseline policies.The procedures target scenarios absent from the training dataset.
- Seen Tasks: Seen-task evaluation covers 744 training tasks in the classroom setting, using a skill-weighted average across individual skill evaluations.The metric aggregates performance across 12 skills.
- Unseen Tasks: Unseen-task evaluation covers 53 held-out tasks whose skill-object combinations differ from training examples while reusing the same environment and randomization procedure.Other combinations of the same skills and objects remain present in the training set.
- Distractor Robustness: Distractor robustness increases from 0, 2, or 5 distractors to 9 distractors, with the hardest setting partially occluding the target object.The evaluation uses three coke-can tasks across easy, medium, and hard settings.
- Background Robustness: Background robustness ranges from the training environment to a brand-new kitchen with altered countertop texture, drawer material, and color.The evaluation covers six tasks with progressively more challenging backgrounds and counter textures.
- Realistic Instructions: Realistic-instruction evaluations use a real office kitchen and add new lighting, distractors, objects, locations, and object-skill combinations across L1, L2, and L3 shifts.L1 introduces a new office kitchen and lighting; L2 adds unseen distractors; L3 adds new objects or locations.
D.2 HETEROGENEOUS DATA
RT-1 absorbs heterogeneous simulation and multi-robot data while preserving performance on original tasks. This supports transfer across domains and robot morphologies, including substantial gains on previously unseen settings.
- Experimental scope: The heterogeneous-data experiments test whether RT-1 can absorb real, simulated, and differently embodied robot experiences without sacrificing original-task performance.The study combines real and simulation data, then combines datasets collected by Kuka IIWA and Everyday Robots.
- Simulation data: RT-1 combines real and simulation data to learn skills involving objects and instructions absent from real-world training.Evaluations are performed in the real world, including seen and unseen skills with simulation objects.
- Simulation data: Simulation data leaves real-object performance unchanged while improving performance on simulation-only objects and unseen instructions.Performance on simulation-only objects rises from 23% to 87%, while unseen-instruction performance increases from 7% to 33%.
- Multi-robot data: Mixing Kuka and Everyday Robots data minimally affects Classroom performance while nearly doubling Bin-picking performance on Everyday Robots.Bin-picking performance reaches 39% with multi-robot data versus 22% using RT-1 data alone, a 17% difference.
- Multi-robot data: Kuka-only training transfers poorly to Everyday Robots, whereas mixed data enables bin-picking without explicit Everyday Robots demonstrations.Kuka-trained RT-1 achieves 0% on the Everyday Robots bin-picking evaluation; combining both datasets enables transfer across morphologies.
D.3 LONG-HORIZON EVALUATION DETAILS
The long-horizon evaluation tests RT-1 within SayCan by chaining manipulation and navigation skills across kitchen instructions. RT-1 achieves strong execution in the familiar kitchen and retains manipulation performance in a more challenging kitchen.
- Evaluation setup: Long-horizon success declines exponentially with task length, making robust low-level manipulation and base-position handling important.SayCan combines multiple low-level instructions, so longer plans create more opportunities for failure.
- Evaluation setup: The evaluation covers 15 long-horizon instructions averaging 9.6 steps and 2.4 manipulation skills each.SayCan decomposes natural-language tasks into sequences of low-level skills.
- Results: RT-1 achieves 67% execution success in Kitchen1 while matching the 87% planning success rate shared by the non-original methods.The comparison includes SayCan with BC-Z, SayCan with Gato, and the originally reported SayCan results.
- Results: Kitchen2 is a harder generalization scene, where SayCan with Gato finishes no long-horizon tasks and SayCan with BC-Z reaches 13%.Kitchen1 resembles the Robot Classroom training scenes, whereas Kitchen2 does not.
- Results: SayCan-RT1 operates unseen drawers in Kitchen2 and executes ultra-long-horizon tasks of up to 50 steps.These capabilities are reported in the supplementary video.
D.4 MODEL ABLATIONS
RT-1 ablations examine model capacity, action representation, pretraining, history, robustness, and inference speed. The results identify discretized actions and ImageNet pretraining as important design choices while highlighting a real-time efficiency trade-off.
- Ablation scope: Ablations compare seen-task performance, unseen-task generalization, inference speed, and robustness to distractors and backgrounds.The study removes or changes major architectural and representation choices in RT-1.
- Action representation: Per-dimension action discretization outperforms a continuous Gaussian representation because it can represent complex multimodal action distributions.The Gaussian alternative captures only a single mode and causes a significant performance decline.
- Pretraining: ImageNet pretraining is particularly important for generalization and robustness, with its removal decreasing unseen-task performance by 33%.The paper attributes this effect to the large and diverse visual data used for pretraining.
- Robustness: Adding history primarily improves generalization to distractors, while attention in RT-1 focuses on interaction-relevant scene regions.Different attention heads commonly focus on graspable objects or task-relevant structures such as drawers.
- Inference speed: Autoregressive actions cause an approximately 2x inference slowdown without significant performance gains, so the final RT-1 omits them.RT-1 is almost an order of magnitude faster than Gato with a similar parameter count but slower than ResNet-based BC-Z.
- Summary and analysis: The analysis concludes that data diversity can matter more than data size for expressive multitask models, including data from simulation and different robot embodiments.The paper also links ImageNet and language pretraining, continuous actions, and early language-vision fusion to RT-1’s observed behavior.