Source-linked AI summary
DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Control
Junjie Wen, Yichen Zhu, Jinming Li, Zhibin Tang, Chaomin Shen, Feifei Feng
TL;DR
General robot control remains constrained by limited action representations, costly demonstration requirements, and the difficulty of learning across embodiments. DexVLA addresses these issues with a billion-parameter diffusion expert and a three-stage embodied curriculum, achieving strong results across robot types and complex tasks while remaining limited in some very long-horizon, contact-rich settings.
Problem
VLA robot models face action-representation, data-scarcity, and architectural-imbalance bottlenecks when learning general skills across diverse environments and embodiments.
Method
DexVLA combines a one-billion-parameter, multi-head diffusion action expert with curriculum stages for cross-embodiment pre-training, embodiment alignment, and task adaptation.
Results
DexVLA performs across single-arm, bimanual, dexterous-hand, and mobile-bimanual robots, including adaptation with fewer than 100 demonstrations and direct-prompt laundry folding.
Takeaways & Limitations
The framework supports generalization across embodiments and challenging long-horizon tasks without task-specific adaptation in several evaluated settings.
Takeaways & Limitations
DexVLA remains limited in very long-horizon, contact-rich scenarios such as folding crumpled shirts and continuous bin-picking.
Abstract
from arXiv · showhide
Enabling robots to perform diverse tasks across varied environments is a central challenge in robot learning. While vision-language-action (VLA) models have shown promise for generalizable robot skills, realizing their full potential requires addressing limitations in action representation and efficient training. Current VLA models often focus on scaling the vision-language model (VLM) component, while the action space representation remains a critical bottleneck. This paper introduces DexVLA, a novel framework designed to enhance the efficiency and generalization capabilities of VLAs for complex, long-horizon tasks across diverse robot embodiments. DexVLA features a novel diffusion-based action expert, scaled to one billion parameters, designed for cross-embodiment learning. A novel embodiment curriculum learning strategy facilitates efficient training: (1) pre-training the diffusion expert that is separable from the VLA on cross-embodiment data, (2) aligning the VLA model to specific embodiments, and (3) post-training for rapid adaptation to new tasks. We conduct comprehensive experiments across multiple embodiments, including single-arm, bimanual, and dexterous hand, demonstrating DexVLA's adaptability to challenging tasks without task-specific adaptation, its ability to learn dexterous skills on novel embodiments with limited data, and its capacity to complete complex, long-horizon tasks using only direct language prompting, such as laundry folding. In all settings, our method demonstrates superior performance compared to state-of-the-art models like Octo, OpenVLA, and Diffusion Policy.
1 Introduction
DexVLA targets data scarcity and architectural bottlenecks in general-purpose robot control by combining a billion-parameter diffusion action expert with embodied curriculum learning. The framework is evaluated across embodiments and supports complex tasks, while remaining limited in some very long-horizon, contact-rich scenarios.
- Motivation: Data scarcity and architectural imbalance remain key bottlenecks for general-purpose VLA robot models.Existing models rely on thousands of hours of demonstrations, while their large VLM components remain disconnected from embodied sensorimotor control.
- Method: DexVLA introduces a diffusion-based action expert with one billion parameters and multiple embodiment-specific heads for cross-embodiment learning.The larger expert is intended to learn intricate motor skills and control policies from diverse data.
- Method: Its embodied curriculum progresses from cross-embodiment motor-skill pre-training to embodiment-specific alignment and task-specific adaptation.Stage 1 trains only the diffusion expert; Stage 2 bridges vision-language representations to a robot’s physical constraints; Stage 3 targets long-horizon tasks and novel objects.
- Limitations: The method remains limited in very long-horizon, contact-rich settings such as folding crumpled shirts and continuous bin-picking.These cases are identified as difficult despite the model’s learned dexterous skills.
- Method: Substep reasoning decomposes long-horizon demonstrations into language-guided subtasks that map instructions to precise motor primitives.Examples include breaking shirt folding into smoothing wrinkles, aligning sleeves, and securing folds.
- Results: DexVLA performs effectively across single-arm, bimanual, dexterous-hand, and mobile-bimanual robots, including tasks without task-specific adaptation.It learns novel dexterous skills with fewer than 100 demonstrations, outperforms π0 on laundry folding, and runs at 60Hz after 100 hours of pre-training.
2 Related Work
Related work frames VLAs as generalist robot policies built from vision-language models and diffusion policies as expressive action-distribution models. DexVLA extends these directions with a diffusion expert and embodied curriculum learning.
- Vision-Language-Action Models: Vision-language-action models adapt internet-pretrained vision-language models for generalist robotic control.Their large backbones provide visual and linguistic representations for robot policies.
- Diffusion Models: Diffusion Policy established diffusion models as a way to represent multimodal action distributions in robot learning.Later work extends diffusion-based policies to 3D environments, larger scales, improved efficiency, and new architectures.
- DexVLA: DexVLA combines a diffusion expert with a VLM-based architecture and trains it through embodied curriculum learning.The architecture and curriculum are presented together as the framework’s central design.
3 Method
DexVLA combines a vision-language model with a scalable, multi-head diffusion action expert and trains them through an embodied curriculum. The design supports cross-embodiment learning, embodiment alignment, and structured reasoning for long-horizon manipulation.
- Model Architecture: DexVLA combines a transformer-based VLM with a diffusion action expert that processes projected visual and language representations.Qwen2-VL provides the VLM backbone; image tokens from multiple camera views are concatenated, and the VLM produces reasoning and action tokens.
- Model Architecture: The diffusion expert uses a multi-head architecture for embodiment-specific outputs and scales to one billion parameters for cross-embodiment action learning.The architecture extends ScaleDP with multi-head outputs because naive ScaleDP is not designed for cross-embodiment pre-training.
- Experiments: The evaluation spans four robot configurations and ten tasks, including bimanual, single-arm, and dexterous-hand systems.The experiments include bimanual UR5e, Franka, bimanual AgileX, and Franka with dexterous hands.
- Embodied Curriculum Learning: Stage 2 alone enables tasks such as shirt folding and bin picking on in-domain objects, while later adaptation targets complex tasks and novel objects.Embodiment-specific training connects abstract VLM representations to the target robot’s physical constraints, camera views, and language instructions.
- Substep Reasoning: Substep annotations structure long-horizon demonstrations so the model learns decomposed behaviors instead of skipping critical steps.Annotations are typically provided every five seconds; the paper reports that models without this pre-training frequently skip critical steps in very long tasks.
4 Real World Experiments
Real-world experiments evaluate DexVLA across embodiments, adaptation settings, and task horizons. The results show strong performance without task-specific adaptation, rapid learning on novel embodiments, and improved handling of complex long-horizon tasks, while ablations clarify the roles of staged training and diffusion-expert scale.
- Without Task-Specific Adaptation: DexVLA achieves 0.92 on shirt folding without task-specific adaptation and substantially outperforms baselines on bin picking and table bussing.Scores are normalized averages over 10 episodes per task and method.
- New Embodiments: DexVLA achieves an average score of 0.90 across two novel-embodiment tasks using only 100 demonstrations.The novel systems include a 12-DoF dexterous hand and a kinematically distinct bimanual UR5e.
- Complex Long-Horizon Tasks: DexVLA scores 0.4 on laundry folding and surpasses π0 by 0.08 points on table bussing under direct prompting.These results concern complex long-horizon tasks without an external high-level planner.
- Ablation Study: Stage 1 or Stage 2 alone yields a 0% success rate on folding tasks, while omitting Stage 1 prevents learning meaningful actions.Stage 1 pretrains the diffusion expert, whereas Stage 2 uses embodiment-specific data; the authors interpret Stage 1 as parameter warm-up.
- Ablation Study: Without Stage 3 task-specific adaptation, laundry-folding performance drops from 0.4 to 0, indicating Stage 3 is essential for challenging long-horizon tasks.The ablation directly evaluates laundry folding without Stage 3 training.
- Ablation Study: A 93M UNet-based diffusion policy barely completes shirt folding, achieving an average score of 0.17 and exhibiting oscillatory robot movements.The comparison tests whether the 1B diffusion expert provides advantages over smaller action experts.
5 Conclusion
DexVLA combines a vision-language model for semantic information with a billion-parameter diffusion expert for visuomotor policies. Its three-stage embodied curriculum and sub-step reasoning support progression from embodiment-agnostic skills to complex dexterous and very long-horizon tasks.
- DexVLA uses vision-language models for semantic information and a billion-parameter diffusion expert for robust, generalizable visuomotor policies.
- Embodied curriculum learning progresses through embodiment-agnostic motor skills, embodiment-specific dexterous skills, and complex tasks across three training stages.
- Sub-step reasoning enables very long-horizon tasks without relying on a high-level policy model.
A.1 Visual generalization.
Visual generalization tests evaluate DexVLA on novel objects and scenes for shirt folding and drink pouring. The experiments report effective generalization to novel visual environments, including challenging low-contrast settings.
- Visual generalization is tested on shirt folding with a bimanual AgileX and drink pouring with a dexterous-hand Franka robot.Shirt folding is evaluated without task-specific adaptation, while drink pouring uses 100 demonstrations of the new embodiment.
- The tests vary shirt colors, backgrounds, scenes, cups, and bottles to assess generalization across novel objects and visual environments.A highlighted example is folding white shirts on a white table.
- DexVLA is compared with DP and OpenVLA on the LIBERO benchmark, with the evaluation reported in Table 5.
- Training cost is compared between training only the diffusion expert and training the entire VLA using epochs completed per hour.
A.2 LIBERO simulation experimental results.
On the LIBERO benchmark, DexVLA outperforms the listed baseline methods, including π0 and π0-FAST. The result is presented as evidence of strong benchmark performance.
- DexVLA outperforms all baselines on LIBERO, including π0 and π0-FAST.DexVLA uses Stage 1 pre-trained weights in this comparison.
A.3 Training cost of stage 1.
Stage 1 trains only the diffusion expert, making training faster than training the full VLA. DexVLA also performs strongly on complex tasks and transfers to a dexterous hand without further training.
- Training cost: 2.78 times faster training results from training only the diffusion expert rather than the entire VLA model.The comparison reports training epochs completed per hour with the same batch size; the full VLA is three times larger.
- Long-horizon task performance: 0.8 points on dryer unloading compared with 0 points for both Octo and OpenVLA demonstrates stronger performance on a complex task.Across Figure 11’s tasks, DexVLA significantly outperformed both baselines over 10-trial averages.
- Long-horizon task performance: Nearly 3 times higher scores on sorting and bin picking hard show DexVLA’s advantage over the other baselines on challenging tasks.Sorting was not included in pre-training data.
- Cross-embodiment transfer: 60% average success across 30 novel bin-picking objects was achieved after transferring from a gripper to a dexterous hand.The original gripper achieved 67%, while the dexterous-hand evaluation used objects unseen during Stages 1 and 2.
B.1 Does training with substep reasoning help?
Sub-step reasoning substantially improves DexVLA’s performance on long-horizon tasks. Implicit sub-step reasoning also outperforms replacing it with SayCan-based planning.
- Ablation study: 0.92 versus 0.07 average shirt-folding scores show that direct prompting during diffusion-expert training severely reduces performance.The direct-prompting setup gives each task only one language instruction, whereas the comparison setup uses sub-step reasoning.
- Ablation study: 0 score results when both Stage 1 and Stage 2 use direct-prompting data, indicating failure without sub-step reasoning in those stages.The paper associates sub-step reasoning with a more disentangled action space for long-horizon learning.
- SayCan comparison: Implicit sub-step reasoning significantly outperforms SayCan-based reasoning in the DexVLA comparison.The paper describes implicit reasoning as disentangling the action space across different substeps.
C.1 Task suite.
The task suite spans mobile, bimanual, single-arm, and dexterous-hand settings, covering familiar, novel-object, and long-horizon manipulation tasks across multiple embodiments.
- Novel-embodiment tasks: Drink pouring and packing test dexterous-hand and bimanual UR5 capabilities using 100 demonstrations per task.Drink pouring uses a Franka with dexterous hand; packing uses a bimanual UR5.
- Long-horizon task: Laundry folding requires removing randomized crumpled clothing from a bin, folding it, and stacking it on previously folded items.The randomized initial configuration is identified as a major generalization challenge.
- Core task suite: Dryer unloading, sorting, bin picking, bussing table, and shirt folding form the core evaluation tasks across mobile bimanual and Franka gripper settings.These tasks include both pre-training tasks and sorting, which is explicitly excluded from pre-training data.
- Generalization settings: Easy and hard variants distinguish seen from unseen objects in bin picking and bussing-table tasks.The hard bin-picking and bussing-table settings use objects unseen during evaluation-related training conditions.
C.2 Evaluation protocol.
Evaluation covers four robot configurations and ten tasks, using repeated trials with task-specific scoring criteria. The protocol spans gripper, dexterous-hand, bimanual, and mobile robot setups.
- Scoring protocol: 10 trials per task provide averaged scores for the evaluation suite.The task descriptions specify scoring criteria for each evaluation.
- Task-specific criteria: Laundry folding is scored out of 4 across five shirts, with points for retrieval, flattening, folding, and placement or stacking.Two trials are performed for each item, including medium and small shirts beginning randomly crumpled in a laundry bin.
- Task-specific criteria: Shirt folding, bussing table, dryer unloading, sorting, drink pouring, bin picking, and packing use task-specific scores ranging from 2 to 8.Points are assigned for successful sub-actions such as folds, object sorting, retrieval, pouring, and placement.
- Robot configurations: Four robot configurations include Franka with gripper, Franka with dexterous hand, bimanual UR5e, and bimanual AgileX.The configurations differ in action dimensionality, cameras, and data-collection rates.
- Data and annotation: Sub-step reasoning data supports long-horizon task completion without a high-level policy model.The protocol describes object detection and video segmentation procedures for constructing corresponding annotations.
D.4 Training data details.
DexVLA’s pretraining data spans approximately 100 hours and 91 tasks across four embodiments, but most tasks are short-horizon while evaluation targets long-horizon tasks.
- Approximately 100 hours of data cover 91 distinct tasks across four embodiments.Most data came from the Agilex ARX arm (42.7%) and single Franka Emika robot (34.7%).
- The dataset includes two distinct 6-DoF arm configurations, ARX and PIPER, which differ in kinematic structure and operational characteristics.
- Most pretraining tasks are short-horizon, whereas evaluated tasks are long-horizon, creating a notable distributional difference.