PaperlayerView exact revision on arXiv

Source-linked AI summary

DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge

Wenyao Zhang, Hongsi Liu, Zekun Qi, Yunnan Wang, Xinqiang Yu, Jiazhao Zhang, Runpei Dong, Jiawei He, Fan Lu, He Wang, Zhizheng Zhang, Li Yi, Wenjun Zeng, Xin Jin

arXiv:2507.04447v3cs.CVcs.RO

TL;DR

Existing VLA methods struggle with redundant image forecasting and incomplete dynamic, spatial, and semantic world knowledge. DreamVLA forecasts compact world knowledge within a perception-prediction-action loop, achieving 76.7% real-world success and 4.44 average length on CALVIN ABC-D. Its scope remains centered on parallel-gripper, RGB-centric manipulation with limited scene diversity.

  • Problem

    Existing VLA methods lack efficient image-based forecasting and comprehensive dynamic, spatial, and semantic world knowledge for future reasoning.

  • Method

    DreamVLA forecasts dynamic regions, depth, and semantic features through a world embedding, using structured attention and diffusion-transformer action modeling.

  • Results

    76.7% real-world success and 4.44 average length on CALVIN ABC-D are reported for DreamVLA.

  • Takeaways & Limitations

    Compact dynamic, spatial, and semantic forecasts provide look-ahead cues for action planning within a perception-prediction-action model.

  • Takeaways & Limitations

    DreamVLA mainly addresses parallel-gripper manipulation using RGB-centric data from scenes with limited geometric and material diversity.

Abstract

from arXiv · show

Recent advances in vision-language-action (VLA) models have shown promise in integrating image generation with action prediction to improve generalization and reasoning in robot manipulation. However, existing methods are limited to challenging image-based forecasting, which suffers from redundant information and lacks comprehensive and critical world knowledge, including dynamic, spatial and semantic information. To address these limitations, we propose DreamVLA, a novel VLA framework that integrates comprehensive world knowledge forecasting to enable inverse dynamics modeling, thereby establishing a perception-prediction-action loop for manipulation tasks. Specifically, DreamVLA introduces a dynamic-region-guided world knowledge prediction, integrated with the spatial and semantic cues, which provide compact yet comprehensive representations for action planning. This design aligns with how humans interact with the world by first forming abstract multimodal reasoning chains before acting. To mitigate interference among the dynamic, spatial and semantic information during training, we adopt a block-wise structured attention mechanism that masks their mutual attention, preventing information leakage and keeping each representation clean and disentangled. Moreover, to model the conditional distribution over future actions, we employ a diffusion-based transformer that disentangles action representations from shared latent features. Extensive experiments on both real-world and simulation environments demonstrate that DreamVLA achieves 76.7% success rate on real robot tasks and 4.44 average length on the CALVIN ABC-D benchmarks.

1 Introduction

Existing VLA methods map observations to actions but lack closed-loop future reasoning and comprehensive world knowledge. DreamVLA forecasts compact dynamic, spatial, and semantic representations for planning, achieving strong simulation and real-world results.

  • Existing VLA models directly map language instructions and visual observations to actions, without the closed-loop forecasting capability humans use for future environmental reasoning.
  • Image-based forecasting methods suffer from redundant pixel information, missing explicit spatial knowledge, and insufficient high-level semantic forecasting.
  • DreamVLA establishes a perception-prediction-action loop by forecasting dynamic regions, depth, and high-level semantic features instead of entire future frames.
  • DreamVLA uses block-wise structured attention and a diffusion-transformer decoder to reduce cross-type knowledge leakage and support coherent multi-step action reasoning.
  • 4.44 average task length is achieved on the CALVIN ABC-D benchmark, while real-world success reaches 76.7%.The contribution summary also reports up to 3.5% improvement on the simulation platform.

2 Related Works

VLA research combines pretrained vision-language representations with task-conditioned policies, while newer approaches add future-world prediction to support inverse dynamics. Existing two-stage generation-based strategies remain limited by video-generation performance and latency.

  • Many future-knowledge methods use copilot image or video generation models to produce future frames or goal states before action prediction.
  • Two-stage training with off-the-shelf video generation is easy to implement but limited by video-generation performance and latency.

3 Methodology

DreamVLA formulates robot control as a perception–prediction–action loop that forecasts compact dynamic, spatial, and semantic world knowledge before generating actions. Its unified architecture disentangles these knowledge types and uses predicted representations to condition multi-step action planning.

  • Problem Definition and Notation: DreamVLA maps language, visual observations, and proprioceptive states into a compact world embedding for inverse-dynamics reasoning.Learnable <dream> queries produce the shared representation used for future world-knowledge prediction and control.
  • Comprehensive World Knowledge Prediction: Three prediction branches forecast future dynamic regions, monocular depth, and high-level semantic features instead of reconstructing entire frames.The branches provide motion cues, spatial structure, and semantic context relevant to manipulation.
  • Comprehensive World Knowledge Prediction: Dynamic-region reconstruction uses CoTracker-derived motion areas to focus prediction on pixels moved by the end-effector or movable objects.The method reconstructs interaction-relevant regions rather than dense optical flow or complete future images.
  • Comprehensive World Knowledge Prediction: Depth forecasting uses ground-truth maps when available or Depth-Anything teacher predictions, with scale normalization preserving ordinal depth relationships.The normalized objective addresses monocular depth’s global scale ambiguity and supports future geometry prediction.
  • Comprehensive World Knowledge Prediction: Semantic forecasting predicts future DINOv2 and SAM features with InfoNCE, using ground-truth features as positives and spatially shifted features as negatives.This trains discriminative anticipation of task-relevant object identity and affordances.
  • Inverse Dynamics via Denoising Diffusion Transformer: Block-wise structured attention prevents dynamic, depth, and semantic subqueries from attending to one another, while diffusion generates an n-step action sequence from latent action features.The separated queries attend to shared visual, language, and state tokens; reverse diffusion produces diverse physically plausible trajectories.

4 Experiments

DreamVLA is evaluated on simulated and real-world manipulation benchmarks, alongside ablations of its world-knowledge components and architectural choices. The experiments show strong benchmark performance and identify dynamic-region forecasting, structured attention, and separate modality queries as important design choices.

  • Simulation Benchmark Experiments: 4.44 average task length is achieved on the CALVIN ABC-D benchmark, with DreamVLA outperforming prior methods across the reported tasks.On LIBERO, DreamVLA shows better or comparable performance across all tracks.
  • Ablation Studies: Dynamic-region prediction provides the greatest ablation gains, while depth and semantic cues provide smaller, roughly equal benefits.When used alone, depth and DINO or SAM feature supervision can fail to help or degrade performance.
  • Ablation Studies: Future world-knowledge prediction consistently outperforms auxiliary reconstruction of current RGB, depth, semantic, or DINOv2 features.Forecasting preserves multi-step accuracy longer, whereas reconstruction can lose coherence after the first few actions.
  • Ablation Studies: Dynamic-region prediction performs better than direct optical-flow forecasting because predicting the full future flow field increases training complexity and lowers multi-step success.DreamVLA instead uses a pretrained flow model to obtain a binary dynamic-region mask.
  • Ablation Studies: Structured attention maintains long-horizon control better than a vanilla causal mask by blocking intra-step cross-modal information leakage.The action query reads past language, state, and multimodal predictions without reading sibling queries from the same step.
  • Ablation Studies: K = 9 queries per modality yields the best success rate and longest uninterrupted task execution among the tested query counts.K = 4 lacks capacity for fine-grained motion, geometry, and semantics, while larger K can overload the backbone.

5 Limitation & Future Works

DreamVLA’s demonstrated scope remains limited to relatively narrow manipulation and data settings. Future work targets richer embodiment, spatial sensing, tactile information, and broader training for robustness.

  • Limitations: DreamVLA mainly addresses parallel-gripper manipulation rather than dexterous-hand interaction.The authors propose adding dexterous-hand demonstrations with rich contact annotations.
  • Limitations: The method relies on RGB-centric data and training scenes with limited geometric and material diversity.These constraints bound the current evaluation scope.
  • Future Works: Future work proposes integrating 3D point clouds, spatial information, and tactile signals into volumetric world states, while expanding data collection and on-policy fine-tuning.The stated goals are stronger generalization and long-horizon robustness.

6 Conclusion

DreamVLA uses comprehensive world-knowledge prediction to support inverse dynamics and a perception-prediction-action loop for manipulation. Its dynamic, spatial, and semantic forecasting, structured attention, and diffusion-transformer decoder are evaluated successfully in real and simulated environments.

  • Conclusion: DreamVLA enables inverse dynamics modeling through comprehensive world-knowledge prediction for manipulation tasks.The framework supports a perception-prediction-action loop.
  • Conclusion: Dynamic-region-guided forecasting combines spatial and semantic cues into compact representations for action planning.Structured attention and a diffusion-transformer decoder suppress cross-type representation noise during multi-step reasoning.
  • Conclusion: 76.7% success rate is achieved on real-world robot tasks, alongside performance that outperforms prior methods on CALVIN ABC-D.These results span real and simulated environments.

A.1 DreamVLA Architecture

DreamVLA encodes language, visual observations, and robot state into a shared transformer representation, then predicts structured world knowledge and future actions. Its architecture emphasizes dynamic regions, depth, semantics, and diffusion-based action generation.

  • Input Encoding: Language, images, and proprioceptive state are separately encoded before multimodal fusion.CLIP processes instructions, an MAE-pretrained ViT processes visual inputs, and an MLP tokenizes robot state.
  • World Knowledge Queries: Learnable <dream> queries extract structured dynamic, depth, and semantic knowledge from multimodal inputs.The <dream> queries provide structured supervision for comprehensive knowledge prediction.
  • World Knowledge Prediction: Specialized lightweight ViT decoders reconstruct dynamic regions, depth maps, and high-level semantic features from modality-specific latent and mask tokens.Semantic outputs include DINOv2 and SAM-style segmentation features.
  • Action Prediction: The <action> query conditions a DiT-B decoder that progressively denoises latent action tokens to model multimodal future-action distributions.Each predicted action is a 7-dimensional end-effector pose and gripper vector, with K = 2 future actions in the stated configuration.
  • Dynamic Region Extraction: Dynamic-region masks are derived from tracked interframe motion, thresholded by speed, dilated spatially, and applied to predicted and ground-truth patch embeddings.The mask focuses loss computation on motion-relevant regions.
  • Auxiliary Supervision: Depth supervision uses dataset annotations when available and monocular pseudo-labels otherwise, while DINOv2 and SAM provide semantic supervision.These modalities supply complementary spatial and semantic representation targets.

A.3 Training Detail

DreamVLA combines dynamic, depth, semantic, and diffusion action losses into one weighted objective and uses precomputed foundation-model features to reduce training cost.

  • Objective: The total training objective sums dynamic, depth, semantic, and DiT action losses.The loss is expressed as L = λdynLdyn + λdepthLdepth + λsemLsem + λDiTLDiT.
  • Objective: The loss weights are λdyn = 0.1, λdepth = 0.001, λsem = 0.1, and λDiT = 1.The action loss receives the largest listed coefficient.
  • Training Configuration: Training uses eight NVIDIA A800 GPUs and precomputes large spatial features to save training time and GPU memory.This approach requires additional storage for extracted foundation-model features.

B.1 Simulation Benchmark and Settings

The CALVIN evaluation tests language-conditioned, long-horizon manipulation across visually distinct environments using onboard observations and robot state. DreamVLA additionally predicts future world knowledge alongside actions.

  • Benchmark: CALVIN evaluates agents on long-horizon manipulation tasks driven by free-form language and onboard sensor inputs.The benchmark excludes privileged information and external supervision.
  • Benchmark: Four environments, A through D, vary textures, object placements, and scene layouts to test generalization.Each environment uses a Franka Emika Panda arm with a parallel gripper and tabletop objects.
  • Inputs and Actions: Observations combine fixed and gripper-mounted RGB cameras with low-dimensional end-effector and gripper state inputs.The agent outputs a 7-dimensional continuous action vector.
  • Dataset Split: The dataset contains approximately 2.4 million interaction steps and 40 million short-horizon action windows.Environments A, B, and C support pretraining, while D is held out for zero-shot evaluation.
  • Evaluation Protocol: DreamVLA evaluates 34 diverse tasks while jointly inferring future depth, dynamic saliency, DINOv2 features, and SAM-based segmentation maps.The tasks include pushing, placing, rotating, and other dexterous operations.

B.2 Simulation Results

On CALVIN ABC-D, DreamVLA outperforms prior methods and reaches an average episode length of 4.44. Visualizations show its predicted dynamic regions and depth maps, while real-world deployment uses a low-latency configuration.

  • Simulation Results: DreamVLA achieves superior performance across all CALVIN ABC-D tasks and substantially outperforms previous state-of-the-art methods.Training uses environments A, B, and C, while testing occurs exclusively in unseen Environment D.
  • Simulation Results: DreamVLA outperforms two-stage inverse-dynamics, visual-foresight, and video-generation-based methods.The authors associate this result with unified multimodal prediction and action generation.
  • Simulation Results: 4.44 average episode length on the ABC-D split establishes a new state-of-the-art on CALVIN.The result is reported as validating the benefits of predicting future knowledge.
  • Visualization: DreamVLA reconstructs semantically meaningful whole-scene representations despite applying dynamic supervision only to selected regions.The cited visualization passage attributes this to task-relevant regions receiving frequent interaction in long-horizon sequences.
  • Visualization: Predicted depth maps are relatively coarse because of patch-level reconstruction, but still provide downstream guidance and spatial awareness.The limitation concerns representation resolution rather than the absence of useful depth guidance.
  • Real-World Setup: Real-world training uses seven-frame histories, predicts three future visual representations and action steps, and adapts from 100 task-specific demonstrations per task.Pretraining uses approximately 76,000 successful DROID trajectories.
  • Efficiency: Inference runs at 11 Hz, with dream queries adding 3 ms, or 3.4%, to processing latency on an RTX 4090.The reported times are 91 ms with dream queries versus 88 ms without them.

D Additional Discussions and Future Work

DreamVLA’s future work focuses on testing scalability, baseline compatibility, view contributions, and transfer to more complex tasks and robot embodiments. These directions aim to clarify how the framework extends beyond its current CALVIN manipulation setting.

  • Scaling Laws: Scaling-law studies will test how larger visual encoders and language models affect DreamVLA’s multimodal reasoning and action generation.Planned variants include replacing the current text encoder with larger models such as LLaMA-2 or GPT.
  • Integration with Additional Baselines: Evaluating DreamVLA with newer, diverse baselines such as RoboVLMs could standardize comparisons and expose architectural synergies.
  • Contribution of Multi-View Observations: Ablating fixed and egocentric camera views will quantify their separate contributions to spatial reasoning and robustness under occlusion or ambiguity.
  • Extension to More Complex and Long-Horizon Tasks: Future evaluations will target complex long-horizon tasks with extended temporal dependencies, delayed rewards, and multi-stage subgoals.The proposed extensions include sustained interaction, sequential tool use, and high-level planning, requiring stronger temporal modeling, memory, goal abstraction, and hierarchical reasoning.
  • Application to Robotic Navigation and Humanoid: DreamVLA could extend beyond tabletop manipulation to instruction-driven navigation by predicting dynamic regions, obstacles, and semantic scene components.
  • Application to Robotic Navigation and Humanoid: The framework could also support humanoid control by incorporating proprioception and richer action spaces for whole-body motion, balance, and grounded interaction.

E Broader Impacts

DreamVLA uses predicted world knowledge as an intermediate representation before action generation, improving action grounding and generalization. Its lightweight design and use of unlabeled perceptual signals support efficient, extensible deployment across robot systems.

  • DreamVLA predicts depth, dynamic motion, segmentation, and semantic features before generating actions, improving action grounding and generalization.
  • A lightweight decoder and learnable queries improve existing VLA backbones with minimal parameter overhead and compatibility with VLM architectures.
  • Using unlabeled perceptual signals during training can reduce reliance on expensive, domain-specific language-instruction datasets.
  • DreamVLA is presented as a practical, extensible, and training-efficient framework for improving VLA systems and encouraging multimodal abstraction research.
Loading 2507.04447v3…