PaperlayerView exact revision on arXiv

Source-linked AI summary

Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Models

Haoqi Yuan, Zhixuan Liang, Anzhe Chen, Ye Wang, Haoyang Li, Pei Lin, Yiyang Huang, Zixing Lei, Tong Zhang, Jiazhao Zhang, Jie Zhang, Jingyang Fan, Gengze Zhou, Qihang Peng, Chenxu Lv, Xiaoyue Chen, An Yang, Fei Huang, Junyang Lin, Dayiheng Liu, Jingren Zhou, Chenfei Wu, Xiong-Hui Chen

arXiv:2606.17846v2cs.ROcs.CVcs.LG

TL;DR

Robotic manipulation lacks the diverse, aligned data needed to test whether foundation-model scaling yields genuine generalization. Qwen-RobotManip addresses this with unified alignment and large-scale open-source data, substantially outperforming prior models across out-of-distribution settings.

  • Problem

    Robotic manipulation data is heterogeneous, expensive to collect, and narrow in diversity, leaving it unclear whether foundation-model scaling can achieve genuine generalization.

  • Method

    Qwen-RobotManip unifies representation, motion, and behavioral alignment while scaling training with synthesized human-to-robot data and a ~38,100-hour open-source corpus.

  • Results

    Qwen-RobotManip substantially outperforms prior models across OOD settings, including a 21.5-point advantage over π0.5 on RoboTwin-C2R Hard (69.4 vs. 47.9).

  • Takeaways & Limitations

    The results support alignment and scale as jointly necessary for robotic manipulation models to generalize beyond their training distributions.

  • Takeaways & Limitations

    The approach remains limited by synthesis artifacts, predominantly simulation-based OOD evaluation, and inference constraints for reactive sub-second control.

Abstract

from arXiv · show

Foundation models in language and multimodality achieve strong generalization by aligning heterogeneous data under a unified formulation and training at scale. In this report, we investigate whether this scaling recipe can be applied to robotic manipulation to achieve genuine generalization. This is challenging because, unlike text, manipulation data is heterogeneous by nature, expensive to collect, and narrow in diversity, making alignment and scale simultaneously difficult. We present Qwen-RobotManip, a generalizable Vision-Language-Action foundation model built on Qwen-VL. Qwen-RobotManip introduces a unified alignment framework across the representation, motion, and behavioral dimensions of manipulation, making large-scale multi-source training coherent rather than conflicting. This alignment capability in turn enables Qwen-RobotManip to absorb manipulation data at a scale that prior training regimes could not sustain. A human-to-robot synthesis pipeline converts egocentric hand demonstrations into robot trajectories across 15 platforms, and a rigorous curation pipeline harmonizes heterogeneous datasets. Using only open-source datasets and human videos without proprietary data collection, Qwen-RobotManip constructs a ~38,100-hour pretraining corpus and exhibits emergent generalization capabilities, including zero-shot instruction following, robustness to perturbations, reactive error recovery, and cross-embodiment transfer. We find that standard benchmarks fail to capture pretraining quality and instead adopt OOD settings including RoboCasa365, LIBERO-Plus, EBench, RoboTwin-Clean2Rand, RoboTwin-IF, and RoboTwin-XE. Qwen-RobotManip substantially outperforms prior state-of-the-art models, including $π$0.5, across all OOD settings, ranks 1st in RoboChallenge with a 20% relative improvement, and is validated on real-robot platforms including AgileX ALOHA, Franka, UR, and ARX.

1 Introduction

QWEN-ROBOTMANIP applies an alignment-first scaling strategy to robotic manipulation, addressing heterogeneous embodiments and data through unified representations and large-scale multi-source training. It evaluates generalization with OOD benchmarks and validates performance across four physical robot platforms.

  • Motivation: The introduction argues that narrow demonstration corpora and insufficient alignment prevent existing pretrained priors from transferring across embodiments and tasks.Existing corpora are concentrated in limited teleoperation setups, while data diversity alone cannot resolve incoherent signals from different embodiments.
  • Scalable data corpus: The model scales training to ∼38,100 hours by combining open-source robot datasets with egocentric human demonstrations synthesized into trajectories across 15 robot platforms.A multi-stage curation pipeline harmonizes heterogeneous sources and ensures signal quality.
  • Evaluation: QWEN-ROBOTMANIP replaces reliance on standard in-domain benchmarks with OOD settings designed to distinguish genuine generalization from in-distribution memorization.The proposed evaluation suite includes LIBERO-Plus, RoboTwin-Clean2Rand, RoboCasa365, EBench, RoboTwin-IF, and RoboTwin-XE.
  • Alignment framework: QWEN-ROBOTMANIP introduces unified alignment across representation, motion, and behavioral dimensions to make multi-source training coherent rather than conflicting.Its mechanisms include canonical masked state-action representations, camera-frame delta poses, and in-context policy adaptation.
  • Real-robot validation: QWEN-ROBOTMANIP ranks 1st on the RoboChallenge Table30 v1 generalist track and is validated across AgileX ALOHA, Franka, UR, and ARX.Validation spans in-domain, out-of-domain, few-shot adaptation, and zero-shot cross-embodiment transfer settings.

2 Data Sources for Robotic Manipulation

The training corpus combines robotic demonstrations, egocentric human manipulation videos, and synthetic robot trajectories under unified curation. Human demonstrations are retargeted across 15 robot morphologies, producing approximately 24,808 hours of synthesized data.

  • Corpus Composition: The corpus integrates three complementary modalities: multi-platform robotic demonstrations, egocentric human manipulation videos, and synthetic robot data generated through human-to-robot conversion.The corpus is designed to support generalization across embodiments, tasks, and environments through large-scale heterogeneous data curation.
  • Robot Demonstrations: Robot demonstrations span single-arm, bimanual, dexterous, mobile, and humanoid loco-manipulation in simulation and the real world, totaling over 11,000 hours across nine open-source datasets.The datasets cover diverse hardware platforms and manipulation settings, including multimodal sensing and long-horizon simulated tasks.
  • Egocentric Human Data: Egocentric human hand data complements robot demonstrations because its camera perspective naturally aligns with robot-mounted views, and the collected sources provide hand-pose annotations.EgoDex contributes 732 training hours from 338K demonstrations across 194 tabletop tasks, while VITRA supplies approximately 1M trajectories from unstructured egocentric videos.
  • Human-to-Robot Synthesis: Human-to-robot conversion retargets hand trajectories into robot end-effector position, orientation, and gripper-width actions while smoothing the morphology gap.The gripper frame is constructed from hand geometry, and robot base placement is optimized separately for each of 15 morphologies using representative trajectory keyframes and reach constraints.
  • Human-to-Robot Synthesis: Approximately 24,808 hours of synthesized demonstrations result from rendering each human demonstration into 15 bimanual robot configurations.The configurations include Panda, UR5e, ARX-L5, xArm7, Sawyer, Kinova Gen3, IIWA, Jaco, FR3, UR10e, ViperX, WidowX, Piper, YAM, and AgileX ALOHA.

3 Qwen-RobotManip: The Generalizable Vision-Language-Action Model Design

Qwen-RobotManip couples a pretrained vision-language backbone with a flow-matching action expert and aligns heterogeneous embodiments through a canonical state-action representation and camera-frame action encoding. It further enables parameter-free behavioral adaptation by conditioning predictions on execution history, with stochastic context sampling preventing trivial action copying.

  • Model Architecture: The model decouples multimodal perception and semantic reasoning from continuous motor control, allowing the action expert to specialize in fine-grained actions while the backbone is jointly trained end-to-end.The architecture combines a vision-language backbone with a flow-matching action expert for high-frequency control.
  • Canonical State-Action Representation: An 80-dimensional canonical state-action vector unifies heterogeneous embodiments by assigning shared semantic slots to arm states and actions, with reserved dimensions supporting additional degrees of freedom.Different robots populate relevant subsets of the template, while zero-padded dimensions are excluded from the loss using a per-dimension binary mask.
  • Cross-Embodiment Alignment: Camera-frame delta poses align visually similar motions with numerically proximate actions, helping the model extract cross-embodiment synergies from heterogeneous data.The representation addresses coordinate-frame inconsistencies across datasets by expressing end-effector motion relative to the camera frame.
  • In-Context Policy Adaptation: The policy adapts to new robots or environments without parameter updates by conditioning current action prediction on a structured window of recent observation-action history from the same episode.This in-context mechanism is designed for rapid behavioral adaptation during deployment.
  • In-Context Policy Adaptation: Stochastic context sampling prevents the model from minimizing training loss through copying the most recent action chunk, improving robustness to missing, partial, or temporally displaced context.At deployment, the policy uses a rolling window of recent chunks, while training randomizes context subsets to encourage behavioral-dynamics reasoning.

4 Training

Qwen-RobotManip trains jointly on heterogeneous robot-action and vision-language data, using masked flow matching and auxiliary language supervision to preserve generalization. It then applies generalist supervised fine-tuning, optionally mixing target-domain data with nearby pretraining data to reduce benchmark-specific shortcut learning.

  • Pre-training: Joint training combines multi-source robot demonstrations, egocentric hand videos, and human-to-robot trajectories with vision-language supervision in a 9:1 robot-to-VL ratio.The VLM stream is co-trained to prevent perceptual and language capabilities from degrading during action-prediction optimization.
  • Pre-training: Masked flow matching restricts learning to valid embodiment-specific action dimensions and timesteps, ensuring samples contribute equally regardless of their number of active slots.The mask operates over the shared 80-dimensional canonical action space, while λ = 0.1 weights auxiliary VLM next-token supervision.
  • Supervised fine-tuning: Generalist SFT combines all demonstrations for each deployment domain into one model that executes every target task, rather than training separate specialist policies.SFT optimizes only flow matching on complete unfiltered demonstrations and applies color-jitter augmentation.
  • SFT limitations: Domain-specific SFT can exploit limited visual and instructional diversity, memorizing benchmark patterns instead of preserving genuine language grounding and sensitivity to novel contexts.Similar train-test visual patterns make high benchmark scores achievable through shortcut correlations between scenes and actions.
  • Post-training enhancement: Mixed post-training co-trains benchmark data with distributionally nearby pretraining samples, broadening adaptation while preserving robust execution without diluting domain-specific learning.RoboTwin-IF evaluates preserved instruction following by varying instructions across the same or similar visual scenes, rather than rewarding default actions selected from visual patterns.

5 Deployment

Qwen-RobotManip deploys with inference on a remote server, transmitting observations and actions over WiFi. Real-Time Chunking (RTC) (Black et al., 2026) hides network latency by generating the next action chunk while the robot executes the current one, enabling smooth real-time control.

  • 5 Deployment: Remote-server inference uses WiFi to transmit observations and actions, while RTC (Black et al., 2026) asynchronously generates the next action chunk during execution to mitigate latency and enable smooth real-time control.RTC effectively hides round-trip latency from cloud-based inference and network transmission.

6 Experiments · 6.1 Are Standard Benchmarks Enough?

QWEN-ROBOTMAN’s evaluation shows that standard in-distribution benchmarks can obscure the value of large-scale robotic pretraining, whereas OOD benchmarks reveal genuine transfer and robustness. This motivates using distribution-shifted evaluation to assess manipulation foundation models.

  • 6.1 Are Standard Benchmarks Enough?: The evaluation spans robotic manipulation benchmarks covering diverse embodiments and task types, including LIBERO’s four single-arm tabletop suites across 130 task combinations and RoboTwin.The comparison includes several recent VLA models and begins with these widely used standard benchmarks.
  • 6.1 Are Standard Benchmarks Enough?: On standard LIBERO and RoboTwin benchmarks, StarVLA and Ours-scratch match or exceed pretrained π0.5 and Abot-M0 despite lacking large-scale robotic pretraining.Figure 4 reports this comparison across two widely used VLA benchmarks spanning diverse embodiments and task types.
  • 6.1 Are Standard Benchmarks Enough?: Because training and evaluation share the same environment and task distributions, standard benchmarks can yield high success rates while the pretrained prior contributes negligible transferable value.This is presented as a structural property of the benchmarks rather than a coincidence.
  • 6.1 Are Standard Benchmarks Enough?: 85.7% to 10.6%: StarVLA collapses from RoboTwin Easy IID to RoboTwin-Clean2Rand OOD, while π0.5 substantially outperforms scratch-trained models under distribution shift.The gap widens as perturbation severity increases, supporting OOD evaluation as the correct measure of foundation-model quality.
  • 6.1 Are Standard Benchmarks Enough?: On LIBERO-Plus and RoboTwin-Clean2Rand, pretrained π0.5 separates clearly from StarVLA and Ours-scratch, demonstrating transferable value absent from in-distribution evaluation.These OOD benchmarks diverge from training conditions and expose generalization that training from scratch cannot replicate.
  • 6.1 Are Standard Benchmarks Enough?: For real deployment, models must generalize from a handful of demonstrations on new hardware, workspaces, and tasks rather than merely rank highly on a benchmark’s in-domain distribution.Thus, the relevant property is the amount of generalizable structure internalized by the foundation model.

6.2 Generalization Capabilities

QWEN-ROBOTMANIP is evaluated primarily on OOD benchmarks because standard in-distribution scores do not reliably measure genuine generalization. It consistently surpasses prior methods, including π0.5, across task, scene, instruction, perturbation, and cross-embodiment settings.

  • Overall OOD Results: QWEN-ROBOTMANIP consistently outperforms prior state-of-the-art models across OOD benchmarks, with its advantage over π0.5 widening as evaluation difficulty increases.It surpasses π0.5 by 7.0 points on LIBERO-Plus, 21.5 on RoboTwin-C2R Hard, 18.5 on EBench, and 19.0 on RoboCasa365.
  • Zero-Shot Cross-Embodiment: QWEN-ROBOTMANIP transfers zero-shot across embodiments more effectively with camera-frame EEF actions, reaching 23.9% versus π0.5’s 7.5% when trained only on AgileX ALOHA demonstrations.Camera-frame alignment makes physically similar motions numerically proximate across robots, whereas joint-space control transfers poorly because joint configurations are robot-specific.
  • Task and Scene Generalization: QWEN-ROBOTMANIP achieves strong generalization across task, scene, and perturbation shifts, including 35.9% on RoboCasa365, 45.6% overall success on EBench, and the highest Hard-setting success on RoboTwin-Clean2Rand.On EBench, it also reaches a composite score of 60, while on RoboTwin-Clean2Rand it achieves 62.6% with joint-space control and 60.8% with end-effector control.
  • Perturbation Robustness: QWEN-ROBOTMANIP remains robust under compounded perturbations, maintaining 44.5–46.8% success across EBench dimensions while π0.5 declines from 34.6% to 23.3% under mixed perturbations.On EBench Table Top, it reaches 50.0% success and a score of 70, nearly quadrupling π0.5’s 12.9% success rate.
  • Instruction Following: QWEN-ROBOTMANIP achieves 72.2% on RoboTwin-IF versus π0.5’s 49.6%, preserving language-conditioned control across five instruction-following suites.The gains are especially large for target-object grounding, spatial relations, and multi-way verb discrimination, which the dual-stream co-training strategy and diverse pretraining corpus support.

6.3 Real-World Evaluation

QWEN-ROBOTMANIP achieves strong real-world performance on both in-domain and shifted CobotMagic ALOHA tasks, substantially outperforming π0.5 and StarVLA. Across broader real-robot evaluations, it also demonstrates bimanual coordination, robust pick-and-place, retry-based recovery, and gains on challenging long-horizon tasks.

  • In-Domain Evaluation: 88.6% average ID success versus 42.9% for π0.5 and 20.0% for StarVLA on CobotMagic ALOHA, with five tasks solved in all 5 trials.Performance remains 4/5 on towel-folding but falls to 2/5 on yellow-disc-insertion, exposing precise insertion as the main weakness.
  • Out-of-Domain Evaluation: 87.5% average OOD success versus 37.5% for π0.5 and 0.0% for StarVLA on CobotMagic ALOHA under shifts in scenes, objects, and instructions.QWEN-ROBOTMANIP achieves perfect success on target-object-in-basket and left-right-bowl-stacking, while remaining effective on the harder tool-on-towel task.
  • Multi-Platform Evaluation: 45% success and 59.83 process score on Table30 v1, exceeding DM0_generalist by 8 percentage points in success and 11.4 process-score points.On 30 benchmark tasks, QWEN-ROBOTMANIP averages 40% on tight bimanual-coordination tasks versus 21.2% for π0.5 and is the only model succeeding on pour fries into plate.
  • Multi-Platform Evaluation: 63.3% average success on 12 cross-embodiment pick-and-place tasks, surpassing DM0 by 15.0 percentage points.The tasks range from single-object grasping to four- or five-object sequential manipulation across four platforms.
  • Emergent Retry Behavior: QWEN-ROBOTMANIP spontaneously retries failed actions across picking, placing, pouring, folding, wiping, and sweeping, improving fault tolerance through self-correction.This behavior enables recovery from intermediate failures that would otherwise cause task-level failures, as illustrated by repeated attempts after a dropped object.
  • Challenging Tasks: 36.7% success versus 5% for previous SOTA methods on challenging real-world tasks, while achieving 70% on arrange paper cups and 50% on sort electronic products.It is the only model reaching at least 30% on plug in network cable and pour fries into plate, and the only model with non-zero success on make vegetarian sandwich.

6.4 Ablation Study

The ablations show that aligned action representations enable predictable cross-embodiment scaling and downstream gains, while human-to-robot synthesis, VL co-training, structured prompting, and in-context adaptation each improve robustness or transfer under specific conditions.

  • Action space alignment: Unified representations produce approximately log-linear validation-MSE declines across 1%–100% data, whereas unaligned action fields do not consistently benefit from more data.The experiment uses nested data subsets and a fixed OOD set spanning 15 embodiment types and 154 unseen tasks; Figures 18 and 19 connect prediction scaling to downstream RoboTwin-C2R performance.
  • Embodiment prompts: On RoboTwin-Clean2Rand, natural-language embodiment prompts modestly improve over soft prompts, while structured prompts provide stronger conditioning than coarse identity alone.The prompt ablation uses an early checkpoint on an OOD joint-control evaluation with speed set to 500.
  • In-context policy adaptation: In-context adaptation adds dynamic behavioral information beyond static embodiment prompts, but zero-padded initial context can delay motion, motivating both context-conditioned and context-free variants.Without context, four denoising steps suffice for stable behavior without observed jitter; deployment configuration therefore trades rapid response against richer intra-episode adaptation.
  • Human-to-robot synthesis: 58.7% on RoboTwin-Clean2Rand Hard, +4.0 over Robot-only and +3.7 over +Ego, demonstrates that human-to-robot synthesis improves robustness; Easy also rises 72.9 → 73.4 → 74.2.On LIBERO-Plus, +H2R raises average success from 87.1% to 89.0%, with the largest Camera gain of +7.2, from 72.8 → 80.0.
  • VL data co-training: Removing VL data causes small LIBERO drops of 0.9 and 1.2 points but substantially larger RoboTwin-Clean2Rand easy and hard drops of 6.7 and 8.2 points.The ablation compares the full model with a variant pretrained without any VL-data mixture.
  • Architecture: Last-layer cross-attention achieves the highest LIBERO-Plus average success, 87.5%, while minimizing computational cost among the compared vision-language-action fusion architectures.It avoids per-layer feature fusion and storing all VLM vision-language tokens in the DiT; pure self-attention variants show no clear advantage.

6.5 New Features after Alignment

Post-training with VL data improves out-of-distribution and language-related generalization by preserving the VLM’s foundational capabilities, while mixed VL/VLA training mitigates domain overfitting only with UnifiedEEF. Camera-frame delta EEF further provides an embodiment-agnostic action interface that improves control, skill transfer, and zero-shot cross-embodiment generalization.

  • Setting 1: Post-training with VL data co-training: Adding VL data during post-training improves LIBERO-Plus from 90.1 to 91.4, RoboTwin-Clean2Rand (easy) from 73.2 to 74.0, and RoboTwin-IF from 71.6 to 73.1.RoboTwin-Clean2Rand (hard) remains nearly unchanged at 62.6 vs. 62.5, indicating stronger benefits for out-of-distribution generalization than visual-perturbation robustness.
  • Setting 1: Post-training with VL data co-training: Language-related gains are strongest: LIBERO-Plus language-perturbation success rises from 86.9% for QWEN-ROBOTMANIP to 93.9% with VL data, while Pick-Diverse-Object improves from 76% to 81%.The improvement is attributed to preserving the pretrained VLM’s language understanding and visual grounding instead of causing catastrophic forgetting through action-only fine-tuning.
  • Setting 2: Post-training with VL and auxiliary VLA data co-training: Fine-tuning only on domain-specific RoboTwin-Clean causes progressive RoboTwin-IF degradation as training continues, whereas mixed VL/VLA post-training is introduced to reduce this domain overfitting.The augmented data combines VL tasks such as detection and pointing with auxiliary-simulator trajectories and real-world demonstrations from morphologically similar embodiments.
  • Setting 2: Post-training with VL and auxiliary VLA data co-training: UnifiedEEF is required for mixed post-training: removing it lowers success from 71.6% to 35.0%, while mixed data without it causes complete failure at 0.0%.The optimal configuration using the base architecture and mixed pretraining reaches 75.8%.
  • Camera-frame delta EEF: Camera-frame delta EEF achieves 72.5% / 56.6% on RoboTwin-C2R Easy / Hard, versus 49.0% / 33.0% for the best alternative action-space design.It also yields 55.0% compositional transfer versus 12.5% without UnifiedEEF, and 23.9% average zero-shot cross-embodiment success versus 14.5% for joint control.
  • Camera-frame delta EEF: Camera-frame delta EEF enables 55.0% compositional transfer on novel ARX tasks, 4.4× the 12.5% achieved without UnifiedEEF, because shared visual-coordinate deltas decouple skills from robot kinematics.In zero-shot deployment from AgileX to ARX, UR5, and Franka, UR5 success reaches 22.8% in EEF mode versus 4.1% in joint mode.

7 Conclusion

QWEN-ROBOTMANIP supports applying the alignment-and-scale recipe to robotic manipulation, showing that unified cross-embodiment alignment is a prerequisite for scaling heterogeneous data. The report also identifies synthesis and evaluation limitations while outlining directions for broader data, improved fidelity, and longer-horizon manipulation.

  • Conclusion: QWEN-ROBOTMANIP presents an affirmative case that aligning heterogeneous manipulation data under a unified formulation enables genuine generalization through scaling.The conclusion frames alignment and scale as tightly coupled prerequisites rather than independent engineering challenges.
  • Conclusion: Unified alignment across canonical state-action representations, camera-frame delta poses, and in-context policy adaptation enables data scaling, whereas naïve representations do not show scaling behavior.The framework supports diverse embodiments while converting additional data volume into improved capability.
  • Limitations: The human-to-robot synthesis pipeline remains limited by retargeting distribution gaps and inpainting artifacts, while OOD evaluations remain predominantly simulation-based.Broader real-world evaluation is needed, and fixed action chunk length and inference latency constrain the current system.
  • Future Work: Future work should expand robot morphologies and task domains, improve hand-robot retargeting and physically grounded rendering, and support agentic systems for longer-horizon reasoning and manipulation.The report also advocates shifting VLA evaluation away from in-domain benchmarks.

8 Authors

The paper lists core and additional contributors, identifies equal-contribution, corresponding-author, and project-lead roles, and acknowledges experimental and hardware support.

  • Authors: The core contributor team includes Haoqi Yuan, Zhixuan Liang, Anzhe Chen, Ye Wang, Haoyang Li, Pei Lin, Yiyang Huang, Zixing Lei, Tong Zhang, and others.The full core-contributor list is provided in the paper.
  • Authors: Haoqi Yuan, Zhixuan Liang, Anzhe Chen, Ye Wang, Haoyang Li, Pei Lin, Yiyang Huang, Zixing Lei, and Tong Zhang are marked as equal contributors; Chenfei Wu is a corresponding author, and Xiong-Hui Chen is project lead.The role markers are defined as ∗ equal contribution, † corresponding author, and ‡ project lead.
  • Authors: Additional contributors include Jinhui Ye, Sicheng Xie, Hale Yin, Xudong Guo, Shuai Bai, Lulu Hu, Minying Zhang, Shurui Li, Wenhu Xiao, Yue Wang, Kun Yan, Xiao Xu, Jiahao Li, and Xuancheng Ren.These contributors are listed separately from the core contributors.
  • Acknowledgments: The authors acknowledge the National Pilot Base for Embodied Intelligence Applications, AgileX Robotics, Prof. Hao Dong, and Prof. Yao Mu for experimental, hardware, and other support.The National Pilot Base provided the real-robot experimental environment and equipment, while AgileX Robotics provided hardware support.
Loading 2606.17846v2…