Source-linked AI summary
Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbone
Jiacheng Ye, Shansan Gong, Jiahui Gao, Junming Fan, Shuang Wu, Wei Bi, Haoli Bai, Lifeng Shang, Lingpeng Kong
TL;DR
Autoregressive generation can limit complex visual planning and dynamic robotic control through sequential decoding and error accumulation. This paper introduces Dream-VL and pretrained Dream-VLA using a bidirectional diffusion-language backbone, reporting competitive open-data VLM performance and top-tier robotic benchmark results while identifying improved planning and parallel action generation as key advantages.
Problem
Autoregressive VLM and VLA backbones are limited by sequential generation and inference error accumulation in complex visual planning and robotic control.
Method
The paper builds Dream-VL and Dream-VLA on a diffusion language model, using multimodal pretraining and robotic pretraining with bidirectional diffusion generation.
Results
Dream-VL is competitive with top open-data autoregressive VLMs and surpasses prior diffusion VLMs, while Dream-VLA achieves top-tier performance on LIBERO and SimplerEnv and surpasses leading VLA models.
Takeaways & Limitations
Bidirectional diffusion backbones provide a promising foundation for visual planning and VLA tasks through global planning, action chunking, and parallel generation.
Takeaways & Limitations
The study does not systematically investigate training data, and its current real-robot experiments remain preliminary.
Abstract
from arXiv · showhide
While autoregressive Large Vision-Language Models (VLMs) have achieved remarkable success, their sequential generation often limits their efficacy in complex visual planning and dynamic robotic control. In this work, we investigate the potential of constructing Vision-Language Models upon diffusion-based large language models (dLLMs) to overcome these limitations. We introduce Dream-VL, an open diffusion-based VLM (dVLM) that achieves state-of-the-art performance among previous dVLMs. Dream-VL is comparable to top-tier AR-based VLMs trained on open data on various benchmarks but exhibits superior potential when applied to visual planning tasks. Building upon Dream-VL, we introduce Dream-VLA, a dLLM-based Vision-Language-Action model (dVLA) developed through continuous pre-training on open robotic datasets. We demonstrate that the natively bidirectional nature of this diffusion backbone serves as a superior foundation for VLA tasks, inherently suited for action chunking and parallel generation, leading to significantly faster convergence in downstream fine-tuning. Dream-VLA achieves top-tier performance of 97.2% average success rate on LIBERO, 71.4% overall average on SimplerEnv-Bridge, and 60.5% overall average on SimplerEnv-Fractal, surpassing leading models such as $π_0$ and GR00T-N1. We also validate that dVLMs surpass AR baselines on downstream tasks across different training objectives. We release both Dream-VL and Dream-VLA to facilitate further research in the community.
1 Introduction
The paper addresses limitations of autoregressive VLMs for long-horizon visual planning and robotic control by introducing diffusion-based Dream-VL and Dream-VLA models. It reports competitive multimodal performance, stronger visual-planning results, and top-tier VLA performance while attributing these advantages to bidirectional attention, global planning, and parallel action generation.
- Motivation: Autoregressive generation can accumulate inference errors and hinder global reasoning as visual tasks require longer, more coherent action sequences.The introduction motivates visual planning across scientific, surgical, delivery, and household-robot settings.
- Motivation: Diffusion language models iteratively refine noisy sequences, supporting global coherence and parallel decoding for planning-intensive applications.These properties motivate using dLLM backbones for both visual understanding and action generation.
- Contributions: Dream-VL and Dream-VLA are introduced as diffusion-based models for general visual understanding and robotic manipulation, evaluated across visual, planning, and embodied-action benchmarks.The Dream family is built on the Dream-7B diffusion language model.
- Dream-VL: Dream-VL matches top-tier open-data autoregressive VLMs overall, surpasses prior diffusion VLMs, and outperforms strong AR baselines on long-horizon visual-planning tasks.The cited results include planning benchmarks such as LIBERO and ViPlan.
- Dream-VLA: Dream-VLA achieves top-tier performance on SimplerEnv and LIBERO, surpassing leading VLA models including π0 and GR00T-N1.It is built upon Dream-VL and uses diffusion-based modeling for vision-language-action tasks.
- Contributions: The authors attribute diffusion-model advantages to bidirectional visual-text fusion, goal-aligned global plans, and native action chunking with parallel generation.They state that these properties enable faster convergence during downstream VLA fine-tuning without architectural modification.
2 Related Work
Prior work established autoregressive VLMs and VLAs as strong foundations for multimodal understanding and robotic control, while recent studies began exploring diffusion alternatives. Existing diffusion VLA efforts mainly fine-tune models directly on specific robotic tasks without robotic pretraining.
- Vision-Language Models: Autoregressive VLMs have achieved strong visual understanding, reasoning, and instruction-following capabilities by aligning pretrained vision encoders with language models.
- Vision-Language-Action Models: VLA models extend vision-language capabilities to action generation, supporting agents that interact with physical and virtual environments.
- Diffusion VLAs: Recent work studies discrete diffusion for robotic control and autonomous driving as an alternative to standard autoregressive VLA modeling.
- Diffusion VLAs: DiscreteDiffusionVLA and LLaDA-VLA fine-tune diffusion backbones for specific robotic tasks without robotic pretraining.
3 Dream-VL: Diffusion-based VLM Built on Dream 7B
Dream-VL is a diffusion-based VLM built on Dream 7B and trained for broad visual understanding, with particular emphasis on visual planning. It matches strong open-data autoregressive VLMs overall while showing advantages in high- and low-level planning.
- 3.1 Setup: Dream-VL is a well-aligned diffusion-based VLM built upon Dream 7B and trained with 12M open-source multimodal examples.Visual inputs are encoded with Qwen2ViT and concatenated with text features; training uses the Dream 7B discrete diffusion loss and a multi-stage paradigm.
- 3.2 Experimental Results: Dream-VL significantly outperforms previous diffusion-based VLMs, including LLaDA-V, while using comparable training data.Against other Dream-based VLMs, the gains are primarily attributed to the larger training dataset, approximately 12M versus 2M samples.
- 3.2 Experimental Results: Dream-VL achieves competitive performance with open-data autoregressive VLMs, with notable advantages on MMMU, DocVQA, and ChartQA, but trails leading closed-data models.The reported gap relative to Qwen2.5-VL indicates remaining opportunities for data scaling and model optimization.
- 3.3.1 Dream-VL for High-level Action Planning: In ViPlan, Dream-VL outperforms the controlled autoregressive baseline MAmmoTH-VL-7B on most grounding and planning scenarios.It also consistently outperforms LLaDA-V across evaluation modes and difficulty tiers, while narrowing the gap with top autoregressive baselines.
- 3.3.2 Dream-VL for Low-level Action Planning: On LIBERO-Long, Dream-VL reaches 59.0% success versus 34.0% for Qwen2.5-VL after supervised fine-tuning.The result is reported as evidence that diffusion-based VLMs have stronger potential for long-horizon planning than autoregressive backbones.
- 3.3.2 Dream-VL for Low-level Action Planning: Generating 12 low-level actions at once yields a 27× speedup, while Dream-VL remains robust at longer action-chunk sizes.Only one diffusion step is needed for competitive low-level action prediction, unlike the error accumulation observed with autoregressive generation.
4 Dream-VLA: Dream-VL with Large-scale Robotic Pretraining
Dream-VLA extends Dream-VL through large-scale robotic pretraining on diverse manipulation trajectories. It achieves strong results across LIBERO and SimplerEnv, while diffusion-native action chunking supports architectural consistency and broad downstream adaptation.
- 4.1 Robotic Pretraining: Dream-VLA is obtained by continuing Dream-VL training on 970k diverse Open-X Embodiment robot manipulation trajectories.The data spans multiple robot embodiments, tasks, and scenes; training uses discrete diffusion loss with action chunks of size 8.
- 4.2 Main Results: Dream-VLA achieves 97.2% average success on LIBERO, surpassing OpenVLA-OFT’s 97.1% average.Its suite-level success rates are 97.6% on LIBERO-Spatial, 98.8% on LIBERO-Object, 97.2% on LIBERO-Goal, and 95.0% on LIBERO-Long.
- 4.2 Main Results: Dream-VLA reaches 71.4% overall average on WidowX tasks, exceeding π0 by 31.3 percentage points and OpenVLA-OFT’s 33.3%.It also outperforms DiscreteDiffusionVLA by 17.2 percentage points and shows strong task-level performance on fine-grained manipulation.
- 4.2 Main Results: On Google Robot tasks, Dream-VLA achieves 60.5% overall average, outperforming π0, OpenVLA-OFT, and GR00T-N1 while matching π0+FAST.It reaches 80.3% on Pick Coke and 78.3% on Move Near in Visual Matching, and 54.6% in Variant Aggregation.
- 4.3 Analysis: Robotic pretraining consistently improves fine-tuning outcomes across most evaluated tasks, although gains vary and are more modest on some tasks.LIBERO-Goal and Stack Green Block On Yellow are cited as examples with comparatively modest improvements.
- 4.3 Analysis: Unlike autoregressive VLA backbones, Dream-VLA supports action chunking without architectural changes, preserving a consistent model architecture.This supports downstream fine-tuning with different objectives without requiring structural modification.
OpenVLA-OFT
Dream-VLA converges faster and achieves strong robotic-control performance against OpenVLA-OFT, with diffusion-based generation supporting action chunking without architectural changes.
- Accelerated Convergence in Fine-tuning: Dream-VLA converges faster and reaches lower fine-tuning loss than OpenVLA-OFT, especially under discrete diffusion fine-tuning.The shared objective with language, vision-language, and vision-language-action pretraining is identified as a reason for the pronounced difference.
- Dream-VLA results: 97.2% success on LIBERO and 71.4% overall average on WidowX establish Dream-VLA’s top-tier robotic-control performance.The reported WidowX score substantially exceeds the previous best of 54.2%.
- Dream-VLA results: Dream-VLA consistently outperforms OpenVLA-OFT across diverse fine-tuning objectives.The comparison is reported across training objectives rather than a single fine-tuning setup.
- Action chunking: Diffusion-based backbones enable action chunking without architectural modifications, while providing structural-consistency and training-convergence advantages.The model uses action chunks in the reported PiPER sim-to-sim evaluation.
- PiPER Sim2Real evaluation: PiPER sim-to-real evaluation finds reliable object pickup under moderate appearance shifts, but camera placement and residual simulation-to-real gaps affect performance.The evaluation is qualitative rather than a strictly controlled quantitative comparison.
5 Conclusion and Future Work
The paper presents Dream-VL and Dream-VLA as preliminary diffusion-based models for visual planning and robotic action, while identifying data, action-representation, and real-world-evaluation gaps for future work.
- 5 Conclusion and Future Work: Dream-VL is competitive with open-data autoregressive VLMs, surpasses previous diffusion VLMs, and shows advantages on global-planning tasks.Its general capabilities still lag behind top closed-data autoregressive models.
- 5 Conclusion and Future Work: Dream-VLA achieves strong performance as a pioneering pretrained vision-language-action model based on diffusion language models.The authors describe both releases as preliminary attempts at diffusion-based models for visual planning.
- 5 Conclusion and Future Work: The study does not systematically analyze training data, whose design largely follows prior setups and may leave performance gains unexplored.The authors identify data improvement as a substantial remaining opportunity.
- 5 Conclusion and Future Work: Future work should investigate joint high-level planning and low-level control, improved discrete action representations, and larger, more diverse real-world datasets.The authors specifically mention mixture training, FAST-like representations, and systematic realistic-environment evaluation.