RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalization

• Current VLA models often fail on novel scenes, objects, instructions, and embodiments, limiting real-world applications.Source: §1 · p. 1
• Large-scale robotic learning is constrained by expensive, low-variety teleoperation data and by architectural tradeoffs between discretized and diffusion-based action modeling.Source: §1 · p. 1
• RDT2 combines a 7B pretrained VLM with RVQ-based action tokens, flow matching, and distillation across three training stages.Stage 1 aligns discrete action tokens with pretrained VLM knowledge; Stage 2 models continuous action distributions; Stage 3 distills the action expert for fast inference.Source: §1 · p. 2
• The model is trained on over 10,000 hours of human demonstrations collected with approximately 100 enhanced devices across more than 100 household environments.Source: §1 · p. 2
• Experiments test zero-shot generalization across unseen objects, scenes, instructions, and embodiments, alongside ablations of the training strategy and design choices.Source: §1 · p. 2

• Teleoperation data offers high fidelity but is expensive to collect and typically restricted to structured laboratory settings.Source: §2 · p. 2
• Small-scale imitation models such as Diffusion Policy and ACT are commonly trained per task and lack broad generalization across tasks or embodiments.Source: §2 · p. 2

• A VLA model predicts action chunks from a language instruction and RGB observation within a sequential decision-making process.The action chunk is sampled from p(A_t | ℓ, o_t), where actions are d-dimensional and the observation is assumed sufficient for decision making.Source: §3 · p. 3
• Manipulation tasks combine objects, scenes, instructions, and embodiments, creating a vast combinatorial space that finite datasets cover only sparsely.Source: §3 · p. 3
• Teleoperation is difficult to scale because robotic hardware is expensive, nonportable, and often produces platform-specific datasets.Source: §3 · p. 3
• Human demonstrations are multimodal, requiring models to learn action distributions while balancing discrete representations against quantization error and continuous-model inefficiency.Source: §3 · p. 3

• UMI records 6-DoF end-effector pose and gripper width with a handheld vision-and-tracking device, enabling deployment across robot arms with consistent grippers.Source: §4 · p. 3
• The redesigned UMI system targets structural rigidity, drift-free infrared tracking, and improved dexterity in cluttered environments.Source: §4 · p. 3
• The redesigned hardware resolves pose inconsistencies and reachability limitations, yielding improved data fidelity.Source: §4 · p. 4
• The resulting open-source UMI dataset contains over 10,000 hours of manipulation data gathered across more than 100 households in the wild.Source: §4 · p. 4

Stage 1

• RDT2 first discretizes continuous actions with RVQ and trains the VLM backbone using cross-entropy loss.This stage aligns action representations with the VLM’s token-based training objective.Source: §5 · p. 4, §5.1 · p. 4
• Discretized pretraining significantly accelerates VLA convergence compared with training directly with diffusion loss from the outset.Source: §5.1 · p. 4
• RVQ encodes action chunks into latent vectors, iteratively quantizes them with learnable codebooks, and reconstructs them through a reverse CNN decoder.Training includes measures such as cosine similarity, exponential moving averages, and inactive-codebook restarts to mitigate codebook collapse.Source: §5.1 · p. 4

Stage 3

• Stage 3 distills the multi-step diffusion expert into a trainable single-step generator, reducing latency for highly dynamic tasks.The diffusion generation target is computed on the fly during training rather than pre-generated.Source: §5.3 · pp. 5–6

6.1. Zero-Shot Experiments

• RDT2 is evaluated on unseen embodiments, objects, scenes, and instructions under a zero-shot 4U setting.The evaluation includes open-vocabulary manipulation tasks and repeated trials intended to reduce variance.Source: §6 · p. 6, §6.1 · p. 6
• Both RDT2 variants accomplish basic open-vocabulary tasks across combinations of unseen objects, scenes, instructions, and embodiments.The authors report no significant standard-error difference between RDT2-VQ and RDT2-FM.Source: §6.1 · p. 6
• 1,000 Pick Task trials show convergence of the success rate, supporting n = 256 trials for subsequent experiments as a reliability–labor-cost balance.The standard-error region contained the final value as trials increased.Source: §6.1 · p. 6

6.2. Scaling Laws of Data and Model Size

• RDT2’s scaling curves show consistent performance gains as both model parameters and data scale increase.The scaling analysis evaluates training loss against consumed tokens and model size; reported fit parameters include E ∼2.1108, α ∼0.4402, and β ∼0.2251.Source: §6.2 · p. 7, Figure 5 · p. 7

6.3. Fine-Tuning Experiments

• RDT2 demonstrates superior performance across deformable-manipulation, long-horizon, and dynamic task categories against π0-FAST and π0.5.Reported examples include 4-times-higher performance on unseen objects in cloth folding, doubled full-task success in table bussing, and improved table-tennis ball hitting.Source: §6.3 · pp. 7–8, Table 2 · p. 8

6.4. Ablation Studies

• AR pretraining enables faster convergence and lower final loss than diffusion-only training.The hybrid AR+Diffusion framework preserves pretrained discrete VLM knowledge and provides a favorable initialization.Source: Figure 6 · p. 7, §6.4 · p. 8
• Stage 2 improves inference efficiency without performance degradation, while Stage 3 distillation makes RDT2-UltraFast the fastest diffusion VLA in the comparison.RVQ also reduces the token budget needed for comparable discretization error, saving up to about two-thirds of tokens relative to FAST.Source: §6.1 · p. 6, Figure 7 · p. 7, §6.4 · p. 8

• RDT2 combines a massive embodiment-agnostic dataset with a three-stage training strategy for robotic foundation modeling.The approach targets data scarcity, inference latency, and cross-embodiment generalization.Source: §7 · p. 8
• RDT2 bridges discrete semantic reasoning in large VLMs with the continuous precision required for motor control.The paper also uses distillation to support real-time performance.Source: §7 · p. 8
• RDT2 demonstrates zero-shot transfer to novel objects, scenes, instructions, and robotic platforms.Source: §7 · p. 8
• RDT2 achieves state-of-the-art fine-tuning performance on dexterous, long-horizon, and dynamic tasks such as table tennis.Source: §7 · p. 8

• Deployment in domestic and industrial settings could yield benefits for elderly care and labor efficiency.Source: Impact Statement · p. 9
• Data collected from over 100 private households requires privacy standards and anonymization to protect contributor identities.Source: Impact Statement · p. 9
• Zero-shot deployment on novel robotic embodiments introduces physical-safety risks in unseen physical contexts.Source: Impact Statement · p. 9
• Future deployment should include safety guardrails and verification protocols for human-robot interaction.Source: Impact Statement · p. 9

• The handheld device integrates a computing unit, high-frequency vision, infrared tracking, and a custom gripper interface.Source: §A.1 · p. 14
• The vision system uses an industrial camera with a Sony IMX273 global-shutter sensor and 30 Hz collection configuration.Source: §A.1 · p. 14
• Four HTC VIVE Tracker 3.0 units track the end-effector’s 6-DoF pose.Source: §A.1 · p. 14
• A custom non-actuated replica preserves the configuration and geometry of the ZhiXing gripper used for robotic execution.Source: §A.1 · p. 14
• RDT2 is evaluated on Franka Research 3 and Universal Robots UR5e arms equipped with the same gripper and eye-in-hand camera setup.The robot starts from a home pose that visually replicates the handheld device’s average starting perspective.Source: §A.2 · pp. 14–15
• The UMI dataset contains approximately 10,000 hours collected across more than 100 home environments, whose long-tail objects and arrangements support ecological diversity.The dataset also includes structured settings such as showrooms, mock-up apartments, restrooms, and nursing homes.Source: §B.1 · p. 15

Data Collection in In-Home Environments

• In-home recordings cover over 50 daily manipulation tasks, including picking, pouring, wiping, stirring, and organizing.Collectors followed high-level instructions and used objects available in their homes.Source: §B.2 · p. 15
• Collectors varied objects, containers, and strategies, producing interactions with over 1,000 unique objects.Source: §B.2 · p. 15
• The dataset includes contact-rich, deformable-object, and long-horizon tasks requiring multi-step planning and continued execution.Source: §B.2 · p. 15
• A 50-workstation facility supplements home data with parallel collection under controlled layout and sensing conditions.Its object pool exceeds 3,000 items spanning varied shapes, sizes, weights, materials, and everyday categories.Source: §B.3 · pp. 15–16
• A two-stage annotation pipeline segments recordings into task-level clips and fine-grained action segments with structured natural-language labels.The schema specifies the hand, object, and action primitive to ground perception, language, and motor behavior.Source: §B.4 · p. 16
• Language augmentation generates semantically equivalent paraphrases and simplified variants of fine-grained annotations.Source: §B.5 · p. 16

Stage 1: Autoregressive alignment

• Stage 1 aligns the Qwen2.5-VL backbone with robotics by predicting discretized action tokens using cross-entropy.The infrastructure uses distributed PyTorch and DeepSpeed training with streamed heterogeneous datasets.Source: §C.2 · p. 18, §C.1 · p. 18

Data and visual processing

• Image augmentations combine color jitter with randomized noise, blur, and JPEG corruption chains to improve visual robustness.The corruption suite includes Gaussian/Laplace noise injection, motion blur, and compression artifacts.Source: §C.2 · p. 18

Optimization

• Training uses cosine learning-rate scheduling with exponential decay during the final 8K iterations.Source: §C.2 · p. 18

Stage 2: Flow matching

• Stage 2 freezes the VQA backbone and trains the RDT action expert with Conditional Flow Matching.Training samples timesteps from a Logistic Normal distribution concentrated near t ≈0.5 and monitors several action-quality metrics.Source: §C.3 · p. 19

Stage 3: Distillation

• Stage 3 distills the multi-step Stage 2 policy into a single-step generator for high-frequency inference.A student regresses the teacher’s effective trajectory from t = 0 using velocity MSE.Source: §C.4 · p. 19

Baseline implementations

• RDT2 is compared with π0.5 and π0-FAST using official OpenPI implementations without architecture changes.Only configuration files and checkpoint paths were modified to support the comparison.Source: §D.1 · p. 19
• Each baseline is trained until stable convergence for every task, with training curves reported for Table Bussing.The cited figures show loss for π0-FAST and π0.5 on the Table Bussing Task.Source: §D.1 · p. 20, Figure 13 · p. 20, Figure 14 · p. 20

Zero-shot evaluation

• Zero-shot evaluation tests unseen embodiment, scene, object, and instruction across 3 environments, 2 embodiments, and over 100 unseen objects.The protocol evaluates each task over 256 trials and uses language augmentation to create unseen instructions.Source: §D.4.1 · p. 21
• The zero-shot task suite covers language-conditioned grasping, object transport, towel wiping, bottle shaking, and keyboard pressing.Success generally requires completing the specified manipulation action, such as grasping the correct object and placing it in a target container.Source: §1 · p. 21, §2 · p. 21, §3 · p. 22

Fine-tuning evaluation

• Fine-tuning targets dexterous, long-horizon, or dynamic tasks using 200 demonstrations per task.Tasks include three-subtask shirt folding, multi-object dining setup, and evaluations involving unseen shirts or scenes.Source: §D.4.2 · p. 22, §1 · p. 22, §2 · p. 22
• Dining setup performance is measured with a Progress Score awarding 0.2 points for each item correctly placed.The task requires sequential pick-and-place actions under clutter and variable operation order.Source: §2 · p. 22