PaperlayerView exact revision on arXiv

Source-linked AI summary

RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalization

Songming Liu, Bangguo Li, Kai Ma, Lingxuan Wu, Hengkai Tan, Xiao Ouyang, Hang Su, Jun Zhu

arXiv:2602.03310v1cs.ROcs.AIcs.CVcs.LG

TL;DR

VLA models remain limited by scarce robotic data, inefficient action modeling, and poor transfer across embodiments. RDT2 combines an enhanced UMI dataset with a three-stage VLM-based training strategy, achieving zero-shot generalization across multiple unseen factors and strong downstream performance. The paper also identifies privacy and physical-safety considerations for large-scale real-world deployment.

  • Problem

    VLA models struggle to generalize reliably to novel scenes, objects, instructions, and embodiments, while robotic data collection and cross-platform adaptation remain costly.

  • Method

    RDT2 uses a 7B pretrained VLM, over 10,000 hours of UMI demonstrations, and three-stage RVQ, flow-matching, and distillation training.

  • Results

    RDT2 demonstrates zero-shot transfer to unseen objects, scenes, instructions, and robotic platforms, with state-of-the-art performance in dexterous, long-horizon, and dynamic tasks.

  • Takeaways & Limitations

    The results support combining embodiment-agnostic data collection with staged alignment of discrete semantic knowledge and continuous motor control for generalist robotics.

  • Takeaways & Limitations

    Deployment on novel embodiments and real-world data from over 100 private households introduces physical-safety and privacy requirements.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models hold promise for generalist robotics but currently struggle with data scarcity, architectural inefficiencies, and the inability to generalize across different hardware platforms. We introduce RDT2, a robotic foundation model built upon a 7B parameter VLM designed to enable zero-shot deployment on novel embodiments for open-vocabulary tasks. To achieve this, we collected one of the largest open-source robotic datasets--over 10,000 hours of demonstrations in diverse families--using an enhanced, embodiment-agnostic Universal Manipulation Interface (UMI). Our approach employs a novel three-stage training recipe that aligns discrete linguistic knowledge with continuous control via Residual Vector Quantization (RVQ), flow-matching, and distillation for real-time inference. Consequently, RDT2 becomes one of the first models that simultaneously zero-shot generalizes to unseen objects, scenes, instructions, and even robotic platforms. Besides, it outperforms state-of-the-art baselines in dexterous, long-horizon, and dynamic downstream tasks like playing table tennis. See https://rdt-robotics.github.io/rdt2/ for more information.

1. Introduction

RDT2 addresses VLA limitations in data scale, architecture, and cross-embodiment transfer with a 7B VLM, large-scale UMI data, and three-stage training. Experiments evaluate zero-shot generalization and downstream robotic performance.

  • Current VLA models often fail on novel scenes, objects, instructions, and embodiments, limiting real-world applications.
  • Large-scale robotic learning is constrained by expensive, low-variety teleoperation data and by architectural tradeoffs between discretized and diffusion-based action modeling.
  • RDT2 combines a 7B pretrained VLM with RVQ-based action tokens, flow matching, and distillation across three training stages.Stage 1 aligns discrete action tokens with pretrained VLM knowledge; Stage 2 models continuous action distributions; Stage 3 distills the action expert for fast inference.
  • The model is trained on over 10,000 hours of human demonstrations collected with approximately 100 enhanced devices across more than 100 household environments.
  • Experiments test zero-shot generalization across unseen objects, scenes, instructions, and embodiments, alongside ablations of the training strategy and design choices.

2. Related Work

Prior robot-learning approaches trade off task specialization, data-collection cost, and cross-embodiment generalization. The related work motivates scalable, generalist VLA models.

  • Teleoperation data offers high fidelity but is expensive to collect and typically restricted to structured laboratory settings.
  • Small-scale imitation models such as Diffusion Policy and ACT are commonly trained per task and lack broad generalization across tasks or embodiments.

3. Problem Formulation and Challenges

The paper formulates language-conditioned bimanual manipulation as learning action-chunk distributions from demonstrations. It emphasizes compositional generalization across objects, scenes, instructions, and embodiments, while identifying data and architecture as core challenges.

  • A VLA model predicts action chunks from a language instruction and RGB observation within a sequential decision-making process.The action chunk is sampled from p(A_t | ℓ, o_t), where actions are d-dimensional and the observation is assumed sufficient for decision making.
  • Manipulation tasks combine objects, scenes, instructions, and embodiments, creating a vast combinatorial space that finite datasets cover only sparsely.
  • Teleoperation is difficult to scale because robotic hardware is expensive, nonportable, and often produces platform-specific datasets.
  • Human demonstrations are multimodal, requiring models to learn action distributions while balancing discrete representations against quantization error and continuous-model inefficiency.

4. Hardware and Dataset

The hardware and dataset design adapts UMI for reliable, portable, large-scale data collection. Its embodiment-agnostic interface supports transfer from collected demonstrations to diverse robot arms.

  • UMI records 6-DoF end-effector pose and gripper width with a handheld vision-and-tracking device, enabling deployment across robot arms with consistent grippers.
  • The redesigned UMI system targets structural rigidity, drift-free infrared tracking, and improved dexterity in cluttered environments.
  • The redesigned hardware resolves pose inconsistencies and reachability limitations, yielding improved data fidelity.
  • The resulting open-source UMI dataset contains over 10,000 hours of manipulation data gathered across more than 100 households in the wild.

5. Model and Training Pipeline

RDT2 uses a three-stage pipeline that combines discretized action-token pretraining, continuous-action diffusion modeling, and distillation for rapid inference. The design preserves VLM knowledge while addressing multimodal action generation and latency.

  • Stage 1: RDT2 first discretizes continuous actions with RVQ and trains the VLM backbone using cross-entropy loss.This stage aligns action representations with the VLM’s token-based training objective.
  • Stage 1: Discretized pretraining significantly accelerates VLA convergence compared with training directly with diffusion loss from the outset.
  • Stage 1: RVQ encodes action chunks into latent vectors, iteratively quantizes them with learnable codebooks, and reconstructs them through a reverse CNN decoder.Training includes measures such as cosine similarity, exponential moving averages, and inactive-codebook restarts to mitigate codebook collapse.
  • Stage 3: Stage 3 distills the multi-step diffusion expert into a trainable single-step generator, reducing latency for highly dynamic tasks.The diffusion generation target is computed on the fly during training rather than pre-generated.

6. Experiments

Experiments evaluate RDT2’s 4U zero-shot generalization, scaling behavior, challenging fine-tuned tasks, and training-component contributions. Results show combinatorial generalization, consistent gains from scaling data and model size, strong downstream performance, and improved convergence and inference efficiency from the proposed components.

  • 6.1. Zero-Shot Experiments: RDT2 is evaluated on unseen embodiments, objects, scenes, and instructions under a zero-shot 4U setting.The evaluation includes open-vocabulary manipulation tasks and repeated trials intended to reduce variance.
  • 6.1. Zero-Shot Experiments: Both RDT2 variants accomplish basic open-vocabulary tasks across combinations of unseen objects, scenes, instructions, and embodiments.The authors report no significant standard-error difference between RDT2-VQ and RDT2-FM.
  • 6.1. Zero-Shot Experiments: 1,000 Pick Task trials show convergence of the success rate, supporting n = 256 trials for subsequent experiments as a reliability–labor-cost balance.The standard-error region contained the final value as trials increased.
  • 6.2. Scaling Laws of Data and Model Size: RDT2’s scaling curves show consistent performance gains as both model parameters and data scale increase.The scaling analysis evaluates training loss against consumed tokens and model size; reported fit parameters include E ∼2.1108, α ∼0.4402, and β ∼0.2251.
  • 6.3. Fine-Tuning Experiments: RDT2 demonstrates superior performance across deformable-manipulation, long-horizon, and dynamic task categories against π0-FAST and π0.5.Reported examples include 4-times-higher performance on unseen objects in cloth folding, doubled full-task success in table bussing, and improved table-tennis ball hitting.
  • 6.4. Ablation Studies: AR pretraining enables faster convergence and lower final loss than diffusion-only training.The hybrid AR+Diffusion framework preserves pretrained discrete VLM knowledge and provides a favorable initialization.
  • 6.4. Ablation Studies: Stage 2 improves inference efficiency without performance degradation, while Stage 3 distillation makes RDT2-UltraFast the fastest diffusion VLA in the comparison.RVQ also reduces the token budget needed for comparable discretization error, saving up to about two-thirds of tokens relative to FAST.

7. Conclusion

RDT2 combines over 10,000 hours of embodiment-agnostic data with a three-stage training strategy to address data scarcity, inference latency, and cross-embodiment generalization. It demonstrates zero-shot transfer across objects, scenes, instructions, and robotic platforms, alongside strong fine-tuning results on demanding manipulation tasks.

  • RDT2 combines a massive embodiment-agnostic dataset with a three-stage training strategy for robotic foundation modeling.The approach targets data scarcity, inference latency, and cross-embodiment generalization.
  • RDT2 bridges discrete semantic reasoning in large VLMs with the continuous precision required for motor control.The paper also uses distillation to support real-time performance.
  • RDT2 demonstrates zero-shot transfer to novel objects, scenes, instructions, and robotic platforms.
  • RDT2 achieves state-of-the-art fine-tuning performance on dexterous, long-horizon, and dynamic tasks such as table tennis.

Impact Statement

The work may support broader deployment of robotic assistants, but its real-world use raises privacy and physical-safety concerns. These concerns follow from household data collection and zero-shot deployment in unseen physical contexts.

  • Deployment in domestic and industrial settings could yield benefits for elderly care and labor efficiency.
  • Data collected from over 100 private households requires privacy standards and anonymization to protect contributor identities.
  • Zero-shot deployment on novel robotic embodiments introduces physical-safety risks in unseen physical contexts.
  • Future deployment should include safety guardrails and verification protocols for human-robot interaction.

A.1. Data Collection Hardware (Handheld UMI)

The handheld UMI integrates computing, high-frequency vision, infrared pose tracking, and a custom gripper interface for data collection. The dataset spans private homes and other real-world environments, while deployment uses two robot arms with matched grippers and cameras.

  • The handheld device integrates a computing unit, high-frequency vision, infrared tracking, and a custom gripper interface.
  • The vision system uses an industrial camera with a Sony IMX273 global-shutter sensor and 30 Hz collection configuration.
  • Four HTC VIVE Tracker 3.0 units track the end-effector’s 6-DoF pose.
  • A custom non-actuated replica preserves the configuration and geometry of the ZhiXing gripper used for robotic execution.
  • RDT2 is evaluated on Franka Research 3 and Universal Robots UR5e arms equipped with the same gripper and eye-in-hand camera setup.The robot starts from a home pose that visually replicates the handheld device’s average starting perspective.
  • The UMI dataset contains approximately 10,000 hours collected across more than 100 home environments, whose long-tail objects and arrangements support ecological diversity.The dataset also includes structured settings such as showrooms, mock-up apartments, restrooms, and nursing homes.

B.2. Data Collection in In-Home Environments

In-home collection covers diverse daily manipulation activities using varied objects, strategies, and interaction types, supplemented by controlled facility recordings. A two-stage annotation pipeline and language augmentation connect task behavior with grounded, varied instructions, while multimodal pretraining broadens visual and linguistic supervision.

  • Data Collection in In-Home Environments: In-home recordings cover over 50 daily manipulation tasks, including picking, pouring, wiping, stirring, and organizing.Collectors followed high-level instructions and used objects available in their homes.
  • Data Collection in In-Home Environments: Collectors varied objects, containers, and strategies, producing interactions with over 1,000 unique objects.
  • Data Collection in In-Home Environments: The dataset includes contact-rich, deformable-object, and long-horizon tasks requiring multi-step planning and continued execution.
  • Data Collection in In-Home Environments: A 50-workstation facility supplements home data with parallel collection under controlled layout and sensing conditions.Its object pool exceeds 3,000 items spanning varied shapes, sizes, weights, materials, and everyday categories.
  • Data Collection in In-Home Environments: A two-stage annotation pipeline segments recordings into task-level clips and fine-grained action segments with structured natural-language labels.The schema specifies the hand, object, and action primitive to ground perception, language, and motor behavior.
  • Data Collection in In-Home Environments: Language augmentation generates semantically equivalent paraphrases and simplified variants of fine-grained annotations.

C. Training Details

RDT2 uses a three-stage training pipeline: autoregressive alignment of the VLM to discretized actions, flow matching for continuous control, and distillation for single-step inference.

  • Stage 1: Autoregressive alignment: Stage 1 aligns the Qwen2.5-VL backbone with robotics by predicting discretized action tokens using cross-entropy.The infrastructure uses distributed PyTorch and DeepSpeed training with streamed heterogeneous datasets.
  • Data and visual processing: Image augmentations combine color jitter with randomized noise, blur, and JPEG corruption chains to improve visual robustness.The corruption suite includes Gaussian/Laplace noise injection, motion blur, and compression artifacts.
  • Optimization: Training uses cosine learning-rate scheduling with exponential decay during the final 8K iterations.
  • Stage 2: Flow matching: Stage 2 freezes the VQA backbone and trains the RDT action expert with Conditional Flow Matching.Training samples timesteps from a Logistic Normal distribution concentrated near t ≈0.5 and monitors several action-quality metrics.
  • Stage 3: Distillation: Stage 3 distills the multi-step Stage 2 policy into a single-step generator for high-frequency inference.A student regresses the teacher’s effective trajectory from t = 0 using velocity MSE.

D.1. Baseline Implementations

The experiments implement π0.5 and π0-FAST from official configurations for fair comparison, then evaluate RDT2 under zero-shot and fine-tuning protocols across diverse real-world manipulation tasks.

  • Baseline implementations: RDT2 is compared with π0.5 and π0-FAST using official OpenPI implementations without architecture changes.Only configuration files and checkpoint paths were modified to support the comparison.
  • Baseline implementations: Each baseline is trained until stable convergence for every task, with training curves reported for Table Bussing.The cited figures show loss for π0-FAST and π0.5 on the Table Bussing Task.
  • Zero-shot evaluation: Zero-shot evaluation tests unseen embodiment, scene, object, and instruction across 3 environments, 2 embodiments, and over 100 unseen objects.The protocol evaluates each task over 256 trials and uses language augmentation to create unseen instructions.
  • Zero-shot evaluation: The zero-shot task suite covers language-conditioned grasping, object transport, towel wiping, bottle shaking, and keyboard pressing.Success generally requires completing the specified manipulation action, such as grasping the correct object and placing it in a target container.
  • Fine-tuning evaluation: Fine-tuning targets dexterous, long-horizon, or dynamic tasks using 200 demonstrations per task.Tasks include three-subtask shirt folding, multi-object dining setup, and evaluations involving unseen shirts or scenes.
  • Fine-tuning evaluation: Dining setup performance is measured with a Progress Score awarding 0.2 points for each item correctly placed.The task requires sequential pick-and-place actions under clutter and variable operation order.
Loading 2602.03310v1…