Source-linked AI summary
Scaling Bimanual Household Manipulation from 1,500 hours of Demonstrations to On-Policy Corrections
Jiafeng Xu, Qi Li, Yan Shen, Yiyu Ren, Travis Davies, Shaowen He, Ze Wang, Yifan Yang, Ran Cheng, Hao Dong
TL;DR
Large-scale bimanual manipulation learning needs high-quality demonstrations, while offline experts and on-policy corrections capture complementary data. The paper releases a 1,500-hour household corpus and trains XR-2, then studies both scaling axes on clothes folding: expert-data gains level off, while three DAgger rounds raise success from 58% to 93%.
Problem
Generalist bimanual policies require large-scale human demonstrations, and the relative scaling of offline expert and corrective on-policy data remains a question.
Source: §I · p. 1Method
The authors release a 1,500-hour household-manipulation corpus and train XR-2, then post-train with human DAgger corrections collected during policy rollouts.
Source: §I · p. 1, A. Main Architecture · p. 4Results
On clothes folding, success improves with expert data before gains level off, while three DAgger rounds raise success from 58% to 93%.
Source: §I · p. 2, B. Scaling of Expert Data · p. 5, C. DAgger Experiment · p. 5Takeaways & Limitations
Abstract
from arXiv · showhide
Learning generalist policies for robust bimanual manipulation is bottlenecked by the scarcity of high quality large scale human demonstration data. In this work, we release 1,500 hours of diverse bimanual manipulation demonstrations covering everyday household tasks, and use this comprehensive corpus to train XR-2, a powerful vision-language-action (VLA) model. Enabled by a purpose built high throughput data pipeline and a carefully designed multi stage training paradigm, XR-2 attains strong manipulation performance in our systematic experiments while retaining favorable training efficiency and high data utilization. We further study two critical scaling axes: varying the amount of expert demonstration data, and post training on DAgger correction data from real time human interventions. In both settings, task success rate improves steadily over the data ranges we probe, exhibiting a clear consistent scaling trend at our current data scale. These results validate both the learning capacity of XR-2 and the promising scaling properties of the released dataset, which we open source to support reproducible research on bimanual robot manipulation learning.
I. INTRODUCTION
The paper studies how expert demonstrations and on-policy corrections scale bimanual policy learning, releasing a large household-manipulation corpus and training XR-2. On clothes folding, added expert data improves performance before gains level off, while DAgger corrections continue to raise success.
I. INTRODUCTION
- The motivation is to examine how offline expert demonstrations and corrective on-policy data scale bimanual learning as complementary data sources.Expert demonstrations provide successful behaviors, while corrections capture deployment-time states and failures.Source: §I · p. 1
- The 1,500-hour corpus combines real-robot teleoperation and Universal Manipulation Interface handheld-gripper demonstrations.Its real-robot subset contains 32,518 trajectories, 57.4 million frames, and 531.7 hours of interaction.Source: §I · p. 1
- Long-horizon demonstrations are aligned with sub-task language annotations and organized into 11 atomic skills.The covered primitives include garment folding, washer interaction, object transfer, and laundry-basket handling.Source: §I · p. 2
- DAgger collects human corrections during on-policy rollouts and combines them with expert data, allocating more correction data to lower-success sub-tasks.The procedure extends supervision to policy-visited states and deployment-time failures.Source: §I · p. 2
- DAgger raises clothes-folding success from 58% to 93% over three rounds, while gains from additional expert data diminish.The paper presents expert demonstrations and corrective on-policy data as complementary scaling sources.Source: §I · p. 2
A. Robot System
X2W is a mobile bimanual platform for whole-body manipulation, with articulated arms, a waist, head, and omnidirectional base. It combines multi-view vision with synchronized proprioceptive, action, and visual data.
A. Robot System
- X2W has two 7-DoF arms, 1-DoF grippers, a 4-DoF waist, and a 2-DoF head, for 22 controllable joints on a three-wheel mobile base.Its motion uses 22 target joint positions and three target wheel velocities.Source: A. Robot System · p. 2
- Quasi-direct-drive actuators support responsive, compliant contact-rich motion, while independently driven wheels provide omnidirectional base movement.Source: A. Robot System · p. 2
- The unified 89-D proprioceptive state concatenates joint positions, velocities and torques, end-effector poses, and drive-wheel positions.Source: A. Robot System · p. 2
- A head-mounted camera provides global context, while two wrist cameras capture close-range manipulation details.Proprioceptive states, action sequences, and visual observations are recorded synchronously at 30 Hz.Source: A. Robot System · p. 2
B. Data Collection Pipeline
The data pipeline standardizes offline expert demonstrations and collects online corrections during real-world deployment. Human interventions can be recorded within an ongoing long-horizon rollout and turned into sub-task-linked corrective data.
B. Data Collection Pipeline
- Offline demonstrations follow task-specific procedures for scene setup, object arrangement, robot initialization, and manipulation, with VR operators teleoperating the robot.Source: B. Data Collection Pipeline · p. 2
- Raw trajectories undergo automatic processing, cloud quality validation, VLM-powered task and language annotation, and final manual review.The pipeline aims to produce consistent demonstrations across robots, scenes, and operators.Source: B. Data Collection Pipeline · p. 2
- During deployment, a human temporarily takes control to correct or complete a failed sub-task, then returns control to the policy without restarting the rollout.A single long-horizon trajectory can contain alternating policy- and human-controlled segments.Source: B. Data Collection Pipeline · pp. 2–3
- Dataset statistics cover four household settings, episode durations with a long tail around 150 s, and the distribution of 11 atomic skills.Source: Figure 3 · p. 3
- The system records intervention segments as corrective demonstrations, links them to sub-task prompts, and distinguishes human- from policy-controlled segments without requiring cloud re-annotation.As more robots are deployed, encountered failures are continuously converted into corrective data.Source: B. Data Collection Pipeline · p. 3
C. Real-robot Dataset Overview
The real-robot dataset contains 32,518 trajectories across more than 12 everyday household tasks, with frame-level language annotations. It spans four settings, long-horizon and skill-level demonstrations, and 11 atomic skills.
C. Real-robot Dataset Overview
- The real-robot corpus contains 32,518 trajectories, about 57.4 million frames, and 531.7 hours of interaction data across more than 12 household tasks.All trajectories have frame-level language annotations.Source: C. Real-robot Dataset Overview · p. 3
- Trajectories span four household settings and range from short atomic primitives to sequences exceeding 100 s, with some longer than 150 s.Multi-stage tasks include language-aligned sub-step annotations for task- and skill-level supervision.Source: C. Real-robot Dataset Overview · p. 3
- The 11 atomic skills include frequent clothes folding and less frequent but task-critical washer-loading and plush-transfer primitives; skill counts are not mutually exclusive.A long-horizon trajectory may contain multiple skills.Source: C. Real-robot Dataset Overview · p. 3
D. UMI Dataset Overview
The UMI corpus adds approximately 1,000 hours of in-the-wild bimanual garment-manipulation demonstrations to the broader collection. Its capture setup pairs synchronized dual-view trajectories and fine-grained language labels with varied homes, objects, lighting, and folding styles.
D. UMI Dataset Overview
- Approximately 1,000 hours of in-the-wild UMI demonstrations cover household garment folding and organizing, matching the Folding Station task distribution.The demonstrations were collected across more than 200 households by over 100 collectors.Source: D. UMI Dataset Overview · p. 3
- Each demonstration pairs synchronized egocentric video and dual-gripper trajectories with item-level and fine-grained action-step language annotations.Metric 6-DoF end-effector trajectories are recovered by visual-inertial SLAM; video is recorded at 960 × 960 and 30 Hz.Source: D. UMI Dataset Overview · p. 3
- The dataset spans 5,000 garments, varied household surfaces and lighting, and diverse collector folding strategies; its self-contained grippers record bimanual motion without encoders.The lightweight grippers support reset-free sessions of up to 90 minutes of continuous recording.Source: D. UMI Dataset Overview · pp. 3–4
A. Main Architecture
XR-2 combines a pretrained vision-language backbone with a flow-matching action expert in a 5B-parameter Mixture-of-Transformer model. The paper also describes action-prefix conditioning for asynchronous execution and conditional flow matching for learning from expert demonstrations.
A. Main Architecture
- XR-2 is a 5B-parameter Mixture-of-Transformer VLA combining a pretrained Qwen3-VL-4B-Instruct backbone with a flow-matching action diffusion transformer.The policy conditions action chunks on visual observations, language, and proprioceptive state.Source: A. Main Architecture · p. 4
- The action expert uses 18 transformer blocks with depth-aligned cross-attention to corresponding backbone layers, progressively grounding predictions in increasingly abstract vision-language features.Each block applies bidirectional self-attention before cross-attending to backbone states.Source: A. Main Architecture · p. 4
- Action-prefix conditioning anchors the already committed leading segment and adds noise and supervision only to the suffix, supporting asynchronous inference.Inference delay is estimated from recent execution durations, and five Euler integration steps synthesize full action sequences.Source: A. Main Architecture · p. 4
B. Post-training on Expert Data
- Conditional flow matching trains one policy jointly across sub-tasks by predicting the target flow from noisy action chunks conditioned on vision, robot state, and language.The noisy action chunk interpolates between noise and the demonstration action.Source: B. Post-training on Expert Data · p. 4
C. Post-training on DAgger Data
DAgger post-training addresses failure recovery and skill retention through trajectory-level data mixing and a budget weighted toward weaker sub-tasks. The allocation also preserves a minimum share for every sub-task.
C. Post-training on DAgger Data
- DAgger data allocation targets two challenges: learning recovery from online corrections without forgetting existing skills, and balancing sub-tasks with different learning dynamics.Corrective trajectories are streamed to the cloud for policy optimization on test-environment failures.Source: C. Post-training on DAgger Data · p. 4
- Mixing DAgger and expert data at a 1:1 trajectory-count ratio, rather than by frames, preserves the expert skill distribution because differing sub-task trajectory lengths can distort skill coverage under frame-level sampling.Source: C. Post-training on DAgger Data · p. 5
- The DAgger budget weights sub-tasks by current success, reserving a minimum share so stronger sub-tasks retain data as the budget shifts toward weaker ones.The method sets w_min = 0.05, ϵ = 0.05, and α = 1; α = 0 gives uniform allocation.Source: C. Post-training on DAgger Data · p. 5
IV. EXPERIMENT
The experiment uses asynchronous policy inference and a real-time whole-body controller, and tests under slight generalizations in materials and initial robot position.
A. Experiment Settings
- Policy inference runs at 10 Hz, joint commands at 30 Hz, and the whole-body motion controller at 1000 Hz.Inference runs on an NVIDIA GeForce RTX 4090 GPU and generates action chunks before temporal ensembling.Source: A. Experiment Settings · p. 5
- An 85% success rate under generalized conditions corresponds to a success rate exceeding 95% under in-distribution conditions.The tested generalizations include material size, material color, and initial robot position.Source: A. Experiment Settings · p. 5
- The deployment policy generates action chunks asynchronously, while the robot receives synchronized whole-body joint commands at a fixed 30 Hz.The controller optimizes commands in real time at 1000 Hz.Source: A. Experiment Settings · p. 5
B. Scaling of Expert Data
Clothes-folding results show expert-data gains leveling off at higher data volumes, while three rounds of DAgger post-training raise success from 58% to 93%. The conclusion presents expert and corrective data as complementary scaling levers.
B. Scaling of Expert Data
- Clothes-folding success rises from 34% at 30 hours to 84% at 120 hours, then reaches 82% at 160 hours.The authors report little further gain beyond 120 hours, with the last 40 hours bringing no improvement.Source: B. Scaling of Expert Data · p. 5
C. DAgger Experiment
- Success rises from 58% to 74%, 82%, and 93% across three DAgger iterations, a 35-point gain over the expert-only baseline.The experiments start from a checkpoint trained on 18,695 expert trajectories spanning 10 sub-tasks.Source: C. DAgger Experiment · p. 5
V. CONCLUSION
- The conclusion presents scaling expert data and structuring corrective data around on-policy failures as complementary levers for bimanual policy learning.The released corpus comprises 1,500 hours of data and includes temporally aligned sub-task language annotations.Source: §V · p. 5
APPENDIX
The appendix presents representative real-robot and UMI trajectories, describes public availability of robot hardware files, and lists dated challenge announcements.
REPRESENTATIVE TRAJECTORIES
- Rows show single episodes with active subtask instructions; real-robot head-camera and UMI wrist-view examples span household and garment-manipulation tasks.The UMI examples span four households and five garment types; real-robot examples include washer interaction, basket handling, transfer, unfolding, and folding.Source: REPRESENTATIVE TRAJECTORIES · p. 7, Figure 6 · p. 8, Figure 7 · p. 9
ROBOT DESCRIPTION
- URDF files for the robot hardware are available in the challenge data repository for non-commercial teaching, experimentation, and academic research.Source: ROBOT DESCRIPTION · p. 7
CHANGE LOG
- The change log lists a July 2026 data release and challenge launch, an August workshop-article submission, and a September notice that the challenge is underway.Source: CHANGE LOG · p. 7