Source-linked AI summary
DriveWorld-VLA: Unified Latent-Space World Modeling with Vision-Language-Action for Autonomous Driving
Feiyang jia, Lin Liu, Ziying Song, Caiyan Jia, Hangjun Ye, Xiaoshuai Hao, Long Chen
TL;DR
Existing VLA and world-model approaches do not sufficiently share latent states to unify future scene evolution with action planning. DriveWorld-VLA tightly integrates them in latent space, achieving strong performance across open- and closed-loop evaluations while its training and inference remain constrained by feedback mechanisms.
Problem
Existing methods inadequately share latent states between world modeling and VLA planning, limiting their integration of future scene evolution with action decisions.
Method
DriveWorld-VLA shares LLM hidden-state representations, uses action-conditioned candidate rollouts, and progressively trains imagination, controllability, and reward-guided refinement.
Results
DriveWorld-VLA achieves state-of-the-art performance across open-loop and closed-loop benchmarks, including 91.3 PDMS on NAVSIMv1.
Takeaways & Limitations
The framework uses latent world-model states as decision-making states so the planner can evaluate candidate actions against future scene evolution.
Takeaways & Limitations
The integration of generative and planning tasks is constrained by the feedback mechanism during inference.
Abstract
from arXiv · showhide
End-to-end (E2E) autonomous driving has recently attracted increasing interest in unifying Vision-Language-Action (VLA) with World Models to enhance decision-making and forward-looking imagination. However, existing methods fail to effectively unify future scene evolution and action planning within a single architecture due to inadequate sharing of latent states, limiting the impact of visual imagination on action decisions. To address this limitation, we propose DriveWorld-VLA, a novel framework that unifies world modeling and planning within a latent space by tightly integrating VLA and world models at the representation level, which enables the VLA planner to benefit directly from holistic scene-evolution modeling and reducing reliance on dense annotated supervision. Additionally, DriveWorld-VLA incorporates the latent states of the world model as core decision-making states for the VLA planner, facilitating the planner to assess how candidate actions impact future scene evolution. By conducting world modeling entirely in the latent space, DriveWorld-VLA supports controllable, action-conditioned imagination at the feature level, avoiding expensive pixel-level rollouts. Extensive open-loop and closed-loop evaluations demonstrate the effectiveness of DriveWorld-VLA, which achieves state-of-the-art performance with 91.3 PDMS on NAVSIMv1, 86.8 EPDMS on NAVSIMv2, and 0.16 3-second average collision rate on nuScenes. Code and models will be released in https://github.com/liulin815/DriveWorld-VLA.git.
1. Introduction
DriveWorld-VLA addresses weak coupling between VLA planning and world modeling by sharing latent representations and using action-conditioned imagination. Its progressive training and closed-loop refinement support proactive decision-making, with strong open- and closed-loop evaluation results.
- Motivation: Existing approaches either treat world models as external simulators or share features without leveraging explicit causal reasoning and prospective imagination.These designs limit VLA internalization of environment dynamics and action-outcome relationships.
- Method: DriveWorld-VLA uses LLM hidden states as a shared latent space for imagination and action prediction.This feature-level sharing is intended to internalize physical laws and environmental dynamics within the reasoning engine.
- Method: Action-conditioned “what-if” reasoning evaluates multiple candidate trajectories to support proactive, consequence-aware planning.The framework uses a diffusion transformer for prospective rollouts of candidate actions.
- Training: DriveWorld-VLA uses three-stage progressive training to align multimodal perception, enforce action-conditioned controllability, and refine decisions through reward feedback.The stages jointly train imagination and action prediction, add controllability, and close the loop with reward-guided refinement.
- Results: Extensive open-loop and closed-loop evaluations report state-of-the-art performance and improvements over specialized baselines in reasoning accuracy and safety-critical planning.The evaluations cover NAVSIM and nuScenes benchmarks.
2. Related Works
The related-work context spans world models for environmental evolution and VLA systems for semantic reasoning and direct control. DriveWorld-VLA is presented alongside a pipeline that progressively connects shared-latent imagination, action conditioning, and reward-guided refinement.
- World Models: World-model research for autonomous driving models environmental evolution through video sequences, point-cloud synthesis, and related scene representations.The cited examples include DriveWorld, Vista, LidarDM, and Copilot4D.
- VLAs: VLA research has progressed from high-level semantic interpretation toward end-to-end mappings from multimodal inputs and instructions to control trajectories.Recent work also explores integrating VLAs with world models.
- DriveWorld-VLA Pipeline: DriveWorld-VLA’s pipeline jointly learns future BEV imagination and action prediction from a shared latent representation in Stage 1.This stage establishes the shared representation used by the later controllability and refinement stages.
- DriveWorld-VLA Pipeline: Stage 2 conditions the generative branch on future actions, mapping action sequences to corresponding futures for controllable imagination.This action conditioning provides the pipeline’s feature-level what-if capability.
- DriveWorld-VLA Pipeline: Stage 3 predicts actions, imagines their resulting future, and uses reward feedback to refine action prediction.The training process is organized as guided evaluation and refinement after joint training and controllability fine-tuning.
3. Method
DriveWorld-VLA unifies VLA planning and world modeling through shared latent representations, then adds action-conditioned imagination and reward-guided refinement. Its three-stage training progressively supports future prediction, controllable scene evolution, and consequence-aware action decisions.
- 3.1. VLA & WM Joint Training: DriveWorld-VLA jointly feeds multimodal inputs into a VLM and uses final-layer hidden states as a shared latent representation for future imagination and action prediction.Inputs include multi-view images, textual prompts, historical actions, and BEV representations.
- 3.1. VLA & WM Joint Training: Future imagination predicts future BEV states from the shared representation and current BEV state, while a lightweight action decoder forecasts future actions from latent, BEV, and historical-action inputs.The denoiser’s history-conditioned branch provides dense future supervision, while the action decoder is trained with expert-action imitation supervision.
- 3.2. Action Controllability Fine-Tuning: Because joint training is not conditioned on future actions, it cannot close the reasoning loop between actions and future scene generation.The limitation motivates explicit action-conditioned future imagination rather than relying only on historical-observation extrapolation.
- 3.2. Action Controllability Fine-Tuning: Action controllability fine-tuning uses a DiT-based flow-matching denoiser conditioned on the BEV state and ground-truth future actions to model controllable future evolution.The denoiser processes BEV features through normalization, scaling, embedding, and DiT blocks; flow-matching loss is the sole loss in this stage.
- 3.3. Future-Guided Evaluation & Refinement: Future-guided refinement predicts actions, generates action-conditioned future BEV imagination, and evaluates trajectories using a learned reward function tied to simulator-based online rewards.The denoising branches use Euler-based sampling with 25 sampling steps, and the reward-driven process prioritizes higher-reward trajectories for action refinement.
- 3.3. Future-Guided Evaluation & Refinement: During refinement, the denoiser and VLM remain frozen while reward and action components are trained using action, semantic-BEV, and reward supervision.Future BEV latents from both denoising branches are fused before semantic BEV decoding.
4. Experiments
DriveWorld-VLA is evaluated on open-loop and closed-loop autonomous-driving benchmarks, with experiments covering main results, training-process ablations, strategy choices, and qualitative trajectory planning.
- Main Results: 91.3 PDMS on NAVSIMv1, outperforming top methods across different paradigms.The comparison includes DiffusionDrive, WoTE, and DriveVLA-W0.
- Main Results: 86.8 EPDMS on NAVSIMv2, again outperforming all compared methods.DriveWorld-VLA also achieves 99.1 DAC, 99.6 DDC, and 97.0 LK.
- Main Results: 0.16% collision rate and 0.61m average L2 on the nuScenes validation set.The evaluation uses 3-second planning results and disables ego-state information.
- Ablation Study: Stage 2 and Stage 3 increase NAVSIMv1 PDMS by +1.9 and +1.8, respectively, across the training process.The ablation compares training stages across NAVSIMv1 and nuScenes.
- Ablation Study: -7.7 PDMS under non-progressive training, indicating that the progressive strategy is effective.The non-progressive strategy updates the future-imagination and action-prediction branches simultaneously after Stage 1.
- Ablation Study: The model must first learn from ground-truth actions and enhance the latent feature space before benefiting from action prediction.The ablation further reports that generative and planning tasks need asynchronous unification even within the same feature space.
- Ablation Study: Omitting VLM pre-training or fully freezing VLM parameters limits performance, while the optimal strategy includes initial VLM optimization.The VLM strategy ablation compares freezing and pre-training choices during Stage 1.
- Ablation Study: Future imagination in Stage 3 produces more robust trajectory predictions and significantly reduces collision risk relative to Stage 2.The qualitative visualization attributes the changes across stages to world-modeling accuracy and physical-dynamics understanding.
5. Conclusion
DriveWorld-VLA tightly integrates VLA and world models by sharing latent scene representations and using world-model states for decision-making. Evaluations across closed-loop and open-loop benchmarks show substantial performance gains over existing methods.
- DriveWorld-VLA shares scene representations in latent space so VLA decisions can use information about future scene evolution.
- World-model latent states serve as decision-making states for evaluating the long-term impact of candidate actions.
- DriveWorld-VLA was evaluated on multiple closed-loop and open-loop planning benchmarks.
- The reported results significantly outperform current state-of-the-art methods in decision-making.
Impact Statement
The paper aims to advance end-to-end autonomous driving by integrating Vision-Language-Action models with World Models for improved decision-making and forward-looking imagination. It also frames safer autonomous vehicle technology as a positive societal goal.
- The work integrates Vision-Language-Action models and World Models to improve autonomous-driving decision-making and forward-looking imagination.
- The stated societal goal is to contribute to safer transport through autonomous vehicle technology.
A.1. Prompt
The InternVL system prompt specifies the driving task, evaluation metrics, inputs, and outputs. Figure S1 illustrates the prompt setup for the two benchmarks.
- The InternVL system prompt describes the task, metrics, inputs, and outputs.
- Figure S1 provides an example of the system prompt for the two benchmarks.
A.2. Image Tokens
Visual information is inserted into the model input as a text-domain visual placeholder token sequence. Images are adaptively tiled into 448 × 448 patches, with an additional thumbnail providing a global view when multiple patches are used.
- Visual inputs are represented as a text-domain visual placeholder token sequence within the input.
- Images are adaptively partitioned into 448 × 448 feature patches to accommodate varying aspect ratios.
- When multiple patches are used, an additional 448 × 448 thumbnail supplies a global view.
A.3. Vision-language Representation
The vision-language representation fuses visual and language information into hidden states, then compresses them into a fixed-length latent representation using learnable queries and cross-attention.
- Vision-language fusion: Fused sequence hidden states H combine textual context with visual information at image-context positions.The hidden states have shape H ∈ R^(B×L×D), where L is the maximum tokenized sequence length and D is the language-backbone dimension.
- Latent compression: The model projects hidden states from D = 1536 to d = 256 before latent aggregation.
- Latent compression: 700 learnable latent query vectors summarize the variable-length sequence into a fixed-length representation through cross-attention.The queries serve as keys-and-values aggregators over the projected hidden states, producing Z ∈ R^(B×N_L×d).
B. More Visualization.
The supplementary visualizations show trajectory examples for NAVSIM and nuScenes, while the InternVL prompt specifies how historical trajectories and navigation commands are represented.
- Dataset coverage: The supplementary examples cover both NAVSIM and nuScenes validation data.
- Prompt representation: InternVL prompts represent ground-truth historical trajectories as 2D-coordinate sequences and navigation commands as text.Examples of commands include “turn left,” “go straight,” and “turn right.”
- NAVSIM: NAVSIM visualization figures label sample tokens on the left and trajectory sources at the top.
- nuScenes: nuScenes validation visualizations identify the source of each trajectory at the top of the figures.