PaperlayerView exact revision on arXiv

Source-linked AI summary

RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation

Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, Jun Zhu

arXiv:2410.07864v2cs.ROcs.AIcs.CVcs.LG

TL;DR

Generalizable bimanual manipulation foundation models face multimodal action distributions, scarce dual-arm data, and heterogeneous robot embodiments. RDT combines diffusion modeling, a scalable Transformer, unified physical action representations, and cross-robot pretraining with bimanual fine-tuning. It achieves strong performance and generalization, including zero-shot transfer and few-shot skill learning on real robots.

  • Problem

    Bimanual foundation models must handle multimodal coordinated actions and scarce, heterogeneous robot data while generalizing across unseen task elements.

  • Method

    RDT uses a diffusion Transformer, a physically interpretable unified action space, multi-robot pretraining, and fine-tuning on a 6K+ trajectory bimanual dataset.

  • Results

    RDT achieves a 56% improvement in success rates across challenging tasks, with zero-shot generalization and 1 ∼5-shot learning of new skills.

  • Takeaways & Limitations

    RDT demonstrates broad bimanual generalization to unseen objects, scenes, instructions, and skills while handling fine-grained dexterous tasks.

  • Takeaways & Limitations

    The model is tested on the ALOHA dual-arm robot, and the authors emphasize responsible deployment on real-world robots.

Abstract

from arXiv · show

Bimanual manipulation is essential in robotics, yet developing foundation models is extremely challenging due to the inherent complexity of coordinating two robot arms (leading to multi-modal action distributions) and the scarcity of training data. In this paper, we present the Robotics Diffusion Transformer (RDT), a pioneering diffusion foundation model for bimanual manipulation. RDT builds on diffusion models to effectively represent multi-modality, with innovative designs of a scalable Transformer to deal with the heterogeneity of multi-modal inputs and to capture the nonlinearity and high frequency of robotic data. To address data scarcity, we further introduce a Physically Interpretable Unified Action Space, which can unify the action representations of various robots while preserving the physical meanings of original actions, facilitating learning transferrable physical knowledge. With these designs, we managed to pre-train RDT on the largest collection of multi-robot datasets to date and scaled it up to 1.2B parameters, which is the largest diffusion-based foundation model for robotic manipulation. We finally fine-tuned RDT on a self-created multi-task bimanual dataset with over 6K+ episodes to refine its manipulation capabilities. Experiments on real robots demonstrate that RDT significantly outperforms existing methods. It exhibits zero-shot generalization to unseen objects and scenes, understands and follows language instructions, learns new skills with just 1~5 demonstrations, and effectively handles complex, dexterous tasks. We refer to https://rdt-robotics.github.io/rdt-robotics/ for the code and videos.

1 INTRODUCTION

RDT addresses the complexity and data scarcity of generalizable bimanual manipulation with a scalable diffusion-based architecture and cross-robot pretraining. It is scaled to 1.2B parameters, fine-tuned on over 6K+ bimanual trajectories, and achieves broad generalization and strong performance.

  • Bimanual policies must generalize across unseen objects and scenes, but existing methods rely on task-specific primitives or small-scale models, data, and tasks.
  • Data scarcity and architectural limitations jointly constrain bimanual foundation models because dual-arm systems are costly and multimodal robotic data requires scalable training.
  • RDT uses a scalable diffusion Transformer to represent multimodal bimanual actions and encode heterogeneous language, vision, and action inputs.
  • RDT is pretrained on the largest collection of multi-robot datasets, scaled to 1.2B parameters, and fine-tuned on a multi-task bimanual dataset with over 6K+ trajectories.
  • RDT improves success rates by 56% across challenging tasks and generalizes zero-shot or with 1 ∼5 shots to unseen objects, scenes, instructions, and skills.

2 RELATED WORK

Prior bimanual manipulation methods reduce complexity through specialized interfaces, priors, primitives, or simplified representations, while robotics foundation models learn generalist behaviors from large datasets. These approaches still leave limitations in general scope, data coverage, and action modeling.

  • Learning-based Bimanual Manipulation: Bimanual learning is challenged by high-dimensional action spaces, data scarcity, and multimodal behavior.
  • Learning-based Bimanual Manipulation: Existing methods reduce data requirements using hardware-specific interfaces, arm-role priors, movement primitives, or voxel representations, but these simplifications reduce application scope.
  • Foundation Models for Robotics: Robotics foundation models train multi-task generalist models on large multi-task robot datasets to enable generalizable behaviors.
  • Foundation Models for Robotics: Most robotics foundation models adapt large vision-language models to predict actions and demonstrate generalization to new objects and tasks.

3 PROBLEM FORMULATION AND CHALLENGES

The paper formulates language-conditioned, vision-based bimanual manipulation as producing two-arm actions from visual history, proprioception, and control frequency. Its central challenges are multimodal action distributions, scalable multimodal processing, and heterogeneous robot action spaces under scarce target-robot data.

  • Problem Formulation: The policy receives a language instruction and observation containing RGB history, robot proprioception, and control frequency, then produces actions for two robot arms.
  • Problem Formulation: A practical bimanual policy must generalize across unseen skills, objects, scenes, and execution modalities.
  • Challenges: Target-robot data is particularly scarce at fewer than 10K trajectories, motivating multi-robot pretraining followed by target-robot fine-tuning.
  • Challenge 1: Architecture: Bimanual action distributions are highly multimodal because multiple coordinated motion modes can complete the same task, requiring expressive and scalable architectures.
  • Challenge 2: Heterogeneous Data: Multi-robot training is difficult because robot physical structures and action spaces vary, creating heterogeneous data and potential negative transfer.

4 ROBOTICS DIFFUSION TRANSFORMER

RDT combines diffusion modeling, a scalable Transformer architecture, and a physically interpretable unified action space to address multimodal actions, heterogeneous inputs, robotic dynamics, and cross-robot data scarcity. It uses multi-robot pre-training and target-robot bimanual fine-tuning to support foundation-model-scale learning.

  • Diffusion Modeling: RDT models continuous multimodal action distributions with diffusion instead of deterministic regression, reducing the risk of averaging distinct action modes.Diffusion sampling starts from Gaussian noise and performs K denoising steps to produce a clean action sample conditioned on language and observation inputs.
  • Diffusion Modeling: Action chunks are predicted in one shot to encourage temporal consistency and reduce error accumulation by decreasing the number of decisions during a task.The modeled distribution is p(a_t:t+T_a|ℓ, o_t), where T_a denotes the chunk size.
  • Architecture: RDT encodes heterogeneous modalities into a unified latent space, using MLPs with Fourier features for low-dimensional physical inputs and specialized representations for image inputs.The architecture addresses differences in modality format, dimensionality, and information density across exterior and wrist-camera views.
  • Architecture: A Transformer backbone provides scalability, while QKNorm and RMSNorm address numerical instability and token or attention shifts during large-scale robotic time-series training.The figure caption reports unstable loss without QKNorm and RMSNorm, but the ablation models were trained without pre-training because of resource constraints.
  • Unified Action Space: The physically interpretable unified action space combines robot proprioception and actions, preserving physical meanings while enabling shared representation across heterogeneous robots.The unified space is designed to encompass the main physical quantities of robots with gripper arms and supports pre-training across 46 datasets totaling 1M+ trajectories and 21TB.
  • Training Data: RDT is pre-trained on heterogeneous multi-robot data and fine-tuned on a target-robot bimanual dataset containing 6K+ trajectories across 300+ tasks, 100+ objects, and 15+ rooms.Fine-tuning addresses the embodiment gap between broadly pre-trained robots and the target dual-arm robot.

5 EXPERIMENTS

RDT is evaluated on seven challenging bimanual tasks spanning unseen scenarios, instruction following, few-shot learning, and dexterous operations. Across these evaluations, RDT consistently outperforms the considered baselines and supports zero-shot generalization and few-shot skill learning.

  • Experiment Setup: Seven challenging tasks evaluate RDT across generalizability, unseen elements, instruction following, few-shot learning, and dexterous manipulation.The tasks are designed to represent complex scenarios relevant to real-world manipulation.
  • Experiment Setup: The fine-tuning data include 133 Wash Cup demos, 350 Pour Water demos, 18–19 demos per water level, 5 Handover demos, 1 Fold Shorts demo, and 68 Robot Dog demos.Unseen cups and rooms have zero listed demonstrations, while water-level demonstrations cover little, half, and full amounts.
  • Experiment Setup: RDT is scaled to 1.2B parameters, pre-trained for 1M iterations on 48 H100 GPUs, and fine-tuned for 130K steps.The model is described as the largest diffusion-based robotic foundation model in the supplied passage.
  • Experiment Setup: The evaluation uses success rate, with task-specific trial counts ranging from 8 to 25 trials per condition.Wash Cup and Pour Water use 24 trials across specified conditions, while Handover, Fold Shorts, and Robot Dog use 25 trials each.
  • Results: RDT zero-shot generalizes to unseen objects, scenes, and modalities while learning Handover and Fold Shorts from few demonstrations.It also follows previously unseen water-level instructions and performs delicate Robot Dog joystick operations.
  • Results: RDT consistently outperforms ACT, OpenVLA, and RDT trained from scratch across the seven-task quantitative evaluation.Table 3 reports success rates for objects, instructions, and scenes; ACT is unavailable for instruction following because it is not language-conditioned.

6 CONCLUSION

RDT addresses data scarcity and manipulation complexity through diffusion-based language-conditioned visuomotor imitation learning, cross-robot pre-training, and a unified action space. It outperforms existing methods while demonstrating dexterous manipulation, instruction following, few-shot learning, and zero-shot generalization.

  • Conclusion: RDT is a diffusion-based foundation model for language-conditioned visuomotor imitation learning in generalizable bimanual manipulation.It is pre-trained on extensive multi-robot data and fine-tuned on a self-collected bimanual dataset.
  • Conclusion: The Physically Interpretable Unified Action Space unifies action representations across robots while preserving physical meanings, enhancing robustness and transferability.
  • Conclusion: RDT outperforms existing methods and demonstrates dexterous bimanual capability, instruction following, few-shot learning, and zero-shot generalization to unseen objects and scenes.

ETHICS STATEMENT

The paper states that its data sources follow applicable licensing and usage guidelines, while its fine-tuning data was collected by the authors and volunteers. It also emphasizes responsible deployment and provides materials for reproduction.

  • The research uses open-source, well-documented datasets and follows applicable licensing and usage guidelines.
  • The authors and volunteers collected the fine-tuning dataset.
  • The model is tested on the ALOHA dual-arm robot, with harmful use neither intended nor encouraged.
  • The paper fully open-sources its code, model weights, and fine-tuning datasets, alongside reproduction and deployment instructions.

B ARCHITECTURE DETAILS

RDT encodes heterogeneous low-dimensional, language, image, frequency, and diffusion-step inputs into tokens for a scalable diffusion Transformer. Its 128-dimensional unified action space maps robot actions by physical meaning while distinguishing padding from meaningful zero values.

  • Proprioception and noisy action chunks are first embedded into a shared unified action space, then encoded into token space with a shared MLP.Frequency and diffusion time step are encoded through separate MLPs before concatenation with the other tokens.
  • During training, each modality input is independently masked with probability 10%.
  • The main network adapts Diffusion Transformers with cross-attention to process encoded low-dimensional inputs alongside language and image tokens.QKNorm and RMSNorm are added for training stability, while language and image tokens are alternately injected in cross-attention layers.
  • The unified action space has 128 dimensions and includes the main physical quantities of different robots.Each robot’s raw action elements occupy positions selected by physical meaning, with unavailable positions padded.

D PRE-TRAINING DATASETS

RDT pre-trains on a large, heterogeneous collection of robotics datasets spanning multiple robots, tasks, scenes, and modalities. Dataset weighting, cleaning, and physically consistent preprocessing are used to diversify samples and preserve transferable action semantics.

  • 46 datasets provide 1M+ trajectories and 21TB of pre-training data, described as the largest pre-training collection of robotics datasets to date.Sampling weights are adjusted for dataset diversity, quality, size, and intermediate loss behavior.
  • The collection includes multi-task datasets such as RT-1 with 130K trajectories across 13 embodiments and Mobile ALOHA with 1K+ bimanual trajectories.The cited datasets differ in action spaces, state representations, cameras, and robot embodiments.
  • Data cleaning removes repetitive or failed episodes, blank images, erroneous velocities, and overly short trajectories, while downsampling overlength trajectories.
  • Language, RGB images, proprioception, and action chunks are standardized into consistent model inputs across datasets.The preprocessing fixes image-view ordering and cleans language while retaining variable text length.
  • Physical quantities are aligned by shared units such as meters, radians, and velocities rather than strict normalization to preserve cross-dataset physical meaning.The authors state that rescaling can impair transfer across robots.

E FINE-TUNING DATASET

The fine-tuning dataset targets multi-task bimanual manipulation with diverse objects, scenes, modalities, language instructions, and challenging dexterous tasks. Training and evaluation use Mobile ALOHA hardware in static bimanual settings, supported by augmentation and monitoring procedures.

  • 300+ tasks, 6K+ trajectories, and 3M+ frames form the Mobile ALOHA fine-tuning dataset.The dataset is described as one of the largest open-source multi-task bimanual robot datasets to date.
  • The dataset combines three RGB camera views, dual-arm joint positions and velocities, gripper angles, and manually annotated instructions.GPT-4-Turbo generates 100 expanded instructions and one simplified instruction for each task.
  • More than 300 tasks span 100+ rigid and non-rigid objects across 15+ scenes with randomized object and robot initial positions.
  • Tasks include dexterous manipulation, language and vision comprehension, integrated reasoning, and bimanual cable insertion.Examples include unscrewing a bottle cap, spelling “love,” solving equations, and inserting a charging cable into a phone.
  • Training progress is monitored by comparing diffusion-sampled actions with ground truth using MSE, which is reported to correlate positively with deployment performance.The authors state that training can generally stop when MSE converges.
  • Image corruption, color jittering, proprioceptive Gaussian noise at 40dB SNR, and instruction expansion are used for data augmentation during fine-tuning.
  • The model is deployed on Cobot Mobile ALOHA, but autonomous mobility is not used; training and inference remain static bimanual manipulation.

H EXPERIMENT DETAILS

This section documents evaluation procedures, baseline implementations, model configurations, and training hyper-parameters for RDT and comparison methods. It also describes how total performance and deployment-oriented fine-tuning choices were determined.

  • Evaluation procedure: Total performance is computed by averaging each task’s success rate and any additional requirement columns.For Pour Water-L-1/3, the calculation averages Total, Correct Hand, and Correct Amount.
  • Model and baseline configurations: Table 7 compares baselines and RDT variants by model size, data size, and modeling scheme.
  • Model and baseline configurations: RDT’s encoder and model configurations are specified in Tables 8 and 9.
  • Baseline training: ACT is trained on 90% of the 6K fine-tuning episodes for 8000 epochs, with the best validation checkpoint selected at epoch 5413.The remaining 10% is used for validation.
  • Baseline training: OpenVLA fine-tuning uses task-relevant demonstrations, while full-parameter and LoRA tuning on the entire dataset reached only approximately 60% action-token accuracy for deployment.Task-specific fine-tuning targets approximately 95% action-token accuracy.
  • Baseline training: OpenVLA and Octo fine-tuning exhibit fluctuating rather than convergent action-prediction metrics across training steps.Figure 8 reports fluctuating action-token accuracy for OpenVLA, while Figure 9 reports fluctuating test MSE for Octo.

I MORE RESULTS

The additional results examine RDT’s instruction-following water-pouring behavior across eight trials. The visualizations show close agreement with the 1/3 target and generally good agreement with the 2/3 target despite two mistakes.

  • Instruction-following evaluation: Eight Pour Water trials show RDT’s completed water levels are extremely close to the ground-truth 1/3 standard.Figure 10 visualizes the resulting water levels across Pour Water-L-1/3 trials.
  • Instruction-following evaluation: Eight Pour Water-R-2/3 trials contain one empty-cup mistake and one water-level mistake, while the remaining trials roughly agree with 2/3.The section provides a zoom-in view of water levels across the eight instruction-following trials.
Loading 2410.07864v2…