Source-linked AI summary
HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Model
Jiaming Liu, Hao Chen, Pengju An, Zhuoyang Liu, Renrui Zhang, Chenyang Gu, Xiaoqi Li, Ziyu Guo, Sixiang Chen, Mengzhen Liu, Chengkai Hou, Mengdi Zhao, KC alex Zhou, Pheng-Ann Heng, Shanghang Zhang
TL;DR
Autoregressive VLAs provide pretrained reasoning but disrupt precise control through action quantization, while diffusion-based VLAs preserve continuity without fully using token-level reasoning. HybridVLA unifies both paradigms in one LLM through collaborative training and adaptive ensembling, achieving higher manipulation success across simulation and real-world tasks.
Problem
Autoregressive VLAs quantize continuous actions into discrete bins, whereas diffusion-based VLAs use separate heads that do not fully leverage VLM token-level reasoning.
Method
HybridVLA integrates diffusion denoising into next-token prediction within a single LLM and adaptively ensembles diffusion- and autoregressive-based actions.
Results
HybridVLA(7B) achieves a 74% average success rate across 10 distinct tasks, outperforming OpenVLA by 33% and CogACT by 14%.
Takeaways & Limitations
The shared model combines continuous diffusion actions with pretrained semantic reasoning and supports strong generalization to unseen configurations.
Takeaways & Limitations
Inference speed is constrained by slower autoregressive generation, similar to prior autoregressive VLA methods.
Abstract
from arXiv · showhide
A fundamental objective of manipulation policy design is to endow robots to comprehend human instructions, reason about scene cues, and execute generalized actions in dynamic environments. Recent autoregressive vision-language-action (VLA) methods inherit common-sense reasoning capabilities from vision-language models (VLMs) for next action-token prediction. However, these methods quantize actions into discrete bins, which disrupts the continuity required for precise control. In contrast, existing diffusion-based VLA methods incorporate an additional diffusion head to predict continuous actions solely conditioned on feature representations extracted by the VLM, without fully leveraging the VLM's pretrained reasoning capabilities through token-level generation. To address these limitations, we introduce HybridVLA, a unified framework that absorbs the continuous nature of diffusion-based actions and the contextual reasoning of autoregression within a single large language model. To mitigate interference between the two generation paradigms, we propose a collaborative training recipe that seamlessly incorporates diffusion denoising into the next-token prediction process. With this recipe, we find these two action prediction methods not only reinforce each other but also exhibit varying strength across different tasks. Therefore, we design a collaborative action ensemble mechanism that adaptively fuses both predictions, leading to more robust control. HybridVLA outperforms previous state-of-the-art VLA methods by 14\% and 19\% in mean success rate on simulation and real-world tasks, respectively, while demonstrating stable manipulation in unseen configurations.
1 Introduction
HybridVLA addresses complementary limitations of autoregressive and diffusion-based VLA methods by integrating both action-generation paradigms within one LLM. Collaborative training and adaptive action ensembling support robust manipulation across diverse tasks and unseen configurations.
- Autoregressive VLA methods leverage pretrained reasoning but quantize continuous actions into discrete bins, disrupting action-pose continuity and hindering precise control.
- Diffusion-based VLA methods enable precise manipulation but typically use an independent diffusion head conditioned on VLM features, without fully leveraging token-level pretrained reasoning.
- HybridVLA integrates diffusion and autoregressive action prediction within a single LLM to combine continuous action representations with semantic reasoning.
- Collaborative training integrates diffusion denoising into next-token prediction, allowing the two generation paradigms to mutually reinforce one another.
- HybridVLA achieves state-of-the-art performance across diverse tasks and generalizes strongly to unseen configurations.
2 Related Work
Related VLA research extends vision-language reasoning to robot control, while diffusion policies provide flexible continuous action generation. Existing diffusion-based VLA systems commonly separate reasoning from diffusion-based action prediction.
- VLA models use language and visual observations to generate task plans and predict low-level SE(3) robot poses.
- RT2 quantizes 7-DoF actions into discrete bins for autoregressive pose prediction, while ManipLLM adds affordance priors through chain-of-thought reasoning.
- Diffusion models have been applied across robotic reinforcement learning, imitation learning, grasping, and motion planning.
- Diffusion-based VLA models commonly append diffusion heads or experts after VLMs, separating semantic reasoning from action prediction.
3 HybridVLA Method
HybridVLA organizes multimodal, diffusion, and autoregressive tokens within one LLM, trains both generation modes collaboratively, and ensembles their actions for control. Its design combines continuous diffusion actions with autoregressive contextual reasoning while adapting inference to task-dependent strengths.
- Collaborative training recipe: The collaborative training recipe combines token-sequence formulation, hybrid objectives, and structured training stages to coordinate the two generation paradigms.Training includes large-scale robotic-data pretraining followed by fine-tuning on self-collected data.
- Token sequence formulation design: A unified token sequence organizes robot state, diffusion noise, and autoregressive tokens in the LLM embedding space.Specialized marker tokens bridge the two generation paradigms.
- Ensembled actions: During inference, diffusion and autoregressive actions are generated concurrently and ensembled for execution.Diffusion predictions tend to excel at precise manipulation, while autoregressive predictions perform better on tasks requiring scene semantic reasoning.
- Diffusion and autoregressive actions: HybridVLA predicts diffusion actions through iterative denoising and autoregressive actions through next-token generation within the shared LLM.DDIM inference uses four denoising steps, and autoregressive generation is additionally conditioned on continuous diffusion-token representations.
- Ensembled actions: Autoregressive confidence guides the ensemble: sufficiently confident predictions are averaged with diffusion actions, otherwise diffusion actions alone control the robot.The reported confidence threshold is θ = 0.96.
4 Experiment
The experiments evaluate HybridVLA in simulation, real-world manipulation, ablations, and unseen configurations. Across these settings, the results support its performance, generalization, and collaborative training design.
- Experiment Design: The evaluation covers RLBench’s 10 tabletop tasks, real-world single- and dual-arm tasks, and unseen objects, backgrounds, spatial positions, and lighting conditions.Real-world evaluation uses human-determined success, and dual-arm comparison is conducted solely with π0 because CogAct lacks multi-view support.
- Simulation Experiment: 74% average success rate across 10 RLBench tasks lets HybridVLA(7B) outperform OpenVLA by 33% and CogACT by 14%.HybridVLA-dif(7B) also improves over CogACT and π0 by 6% and 11%, respectively.
- Ablation Study: The ablations show that collaborative training improves both diffusion-only and autoregressive-only manipulation performance relative to separately trained alternatives.The authors associate these gains with avoiding negative interference while capturing continuous actions and pretrained reasoning capabilities.
- Real-World Experiment: HybridVLA achieves 90% and 95% success on real-world Pick and place and Unplug charger, respectively.On Pour water, HybridVLA and HybridVLA-dif outperform the previous SOTA by 35% and 30%, respectively.
- Real-World Experiment: In dual-arm tasks, HybridVLA consistently outperforms previous VLA approaches across five tasks while using 14-DOF action representations.The 14 dimensions represent 7-DOF end-effector poses for each arm.
- Generalization Experiment: HybridVLA has the smallest accuracy drop with unseen manipulated objects and maintains satisfactory results with unseen cluttered backgrounds.The generalization experiments compare HybridVLA with CogAct on single-arm Pick and place and with π0 on dual-arm Lift ball and place.
5 Conclusion and Limitation
HybridVLA combines diffusion and autoregressive action generation in one LLM through collaborative training, targeting robust manipulation and generalization. Its main limitation is slower autoregressive inference, although diffusion-only inference reaches 9.4 Hz.
- Conclusion: HybridVLA equips a single LLM with both diffusion-based and autoregressive action generation capabilities.The framework is intended to combine continuous diffusion actions with autoregressive semantic reasoning.
- Conclusion: Collaborative training integrates diffusion denoising into next-token prediction, enabling mutual reinforcement and improving manipulation robustness.The framework achieves strong generalization across simulation and real-world tasks.
- Limitation: Inference speed is constrained by slower autoregressive generation, but HybridVLA-dif enables diffusion-only inference at 9.4 Hz.The authors identify this as a limitation shared with prior autoregressive VLA methods.
Appendix B. Additional simulation experiments and ablation studies are presented.
The appendix adds manipulation visualizations and describes the large-scale pretraining dataset used for HybridVLA.
- Additional Visualizations: Additional visualizations cover both single-arm and dual-arm manipulation processes.
- Pretraining Data: The pretraining collection contains 35 datasets, 760k trajectories, and 33m frames with adjustable sampling weights.Datasets are reformulated to emphasize end-effector sequence control.
A.2 Self-collected Real-world Dataset
The self-collected real-world benchmark covers diverse single-arm and dual-arm tabletop manipulation tasks using multi-view camera setups. Tasks require language-conditioned object handling, sequential execution, and bilateral coordination.
- Experimental Assets: Single-arm tasks use Franka robot assets with front and wrist views, while dual-arm tasks use bilateral wrist views and an overhead view.The setups use RealSense cameras for single-arm capture and Orbbec plus RealSense cameras for dual-arm capture.
- Single-Arm Tasks: Single-arm Pick and place requires language-conditioned selection of a colored block and placement into a matching colored bowl.
- Single-Arm Tasks: Single-arm tasks also test charger manipulation, pouring with precise bottle alignment, wiping an unfixed region, and four-step drawer manipulation.The drawer sequence is open drawer, pick object, place object, and close drawer.
- Dual-Arm Tasks: Dual-arm Pick and place requires both arms to place two language-specified objects in a container.
- Dual-Arm Tasks: Dual-arm Lift ball and place tests simultaneous contact, transport, and slip prevention through coordinated bilateral motion.
- Dual-Arm Tasks: Additional dual-arm tasks require bottle placement at a rack, board holding during wiping, and sequential shorts folding with both arms.
B.1 Additional Simulation Experiments
Additional simulation experiments show that autoregressive and diffusion action predictions have complementary strengths across tasks, supporting adaptive combination within HybridVLA.
- B.1 Additional Simulation Experiments: HybridVLA-ar outperforms HybridVLA-dif on 4 of 10 RLBench tasks, while HybridVLA-dif leads on the remaining 6.All modes use joint training with the collaborative recipe, but inference uses exclusively autoregressive or diffusion actions in the two specialized variants.
- B.1 Additional Simulation Experiments: Diffusion predictions excel at precise manipulation tasks, including Phone on base, Toilet seat down, and Close laptop lid.
- B.1 Additional Simulation Experiments: Autoregressive predictions perform better on scene-level semantic reasoning tasks, including Sweep to dustpan, Water plants, and Frame off hanger.
- B.1 Additional Simulation Experiments: Table 7 reports success rates for HybridVLA, HybridVLA-ar, and HybridVLA-dif under the collaborative training recipe.
B.2 Additional Ablation Study
The ablations examine confidence thresholds, inference caching, and denoising steps, identifying settings that preserve accuracy while improving or stabilizing execution speed and ensemble quality.
- B.2 Additional Ablation Study: Figure 4 plots manipulation success rate against the number of denoising steps.
- B.2 Additional Ablation Study: A confidence threshold below 0.94 makes autoregressive predictions unreliable and slightly degrades ensemble performance.At 0.98, valid autoregressive actions become too limited, so ensemble performance approaches diffusion-only performance.
- B.2 Additional Ablation Study: KV caching increases HybridVLA-dif inference speed from 5.0 Hz to 9.4 Hz without materially changing its average success rate of 66% across 10 simulation tasks.
- B.2 Additional Ablation Study: Reducing DDIM denoising steps from 30 to 4 does not significantly degrade manipulation performance, motivating the final setting of 4 steps.
C Additional Visualizations
The real-world visualizations document single-arm and dual-arm execution, highlighting precise pose, gripper-timing, and coordinated-arm capabilities across the demonstrated tasks.
- C Additional Visualizations: The visualizations use an FR3 execution setup with specified controller, libfranka, Franka ROS, Ubuntu, and FCI settings.
- C Additional Visualizations: HybridVLA accurately predicts position, rotation, and gripper-state timing during the illustrated real-world tasks.
- C Additional Visualizations: The dual-arm demonstrations show coordination of both arms for tasks beyond single-arm capability, such as transporting a ball to a container.
D Failure Case Analysis.
Real-world experiments identify rotational, kinematic, and dual-arm coordination failures, while the paper frames its contribution as a robotics research effort without direct societal impact.
- D Failure Case Analysis.: The three primary failure categories are rotational prediction deviations, poses exceeding robot limits, and failures in dual-arm coordination.
- D Failure Case Analysis.: Rotational errors arise in precise tasks through accumulated multi-step rotation errors or incorrect interaction angles.
- D Failure Case Analysis.: Predicted poses can exceed mechanical limits, workspace boundaries, or kinematic feasibility during complex transitions.
- D Failure Case Analysis.: Dual-arm failures can occur when one arm changes the object state and invalidates the other arm’s previously predicted action.
- D Failure Case Analysis.: The authors state that the work focuses on VLA structure innovation and has no direct societal impact.