DriveWorld-VLA: Unified Latent-Space World Modeling with Vision-Language-Action for Autonomous Driving

Motivation

• Existing approaches either treat world models as external simulators or share features without leveraging explicit causal reasoning and prospective imagination.These designs limit VLA internalization of environment dynamics and action-outcome relationships.Source: §1 · p. 2

Method

• DriveWorld-VLA uses LLM hidden states as a shared latent space for imagination and action prediction.This feature-level sharing is intended to internalize physical laws and environmental dynamics within the reasoning engine.Source: §1 · p. 2
• Action-conditioned “what-if” reasoning evaluates multiple candidate trajectories to support proactive, consequence-aware planning.The framework uses a diffusion transformer for prospective rollouts of candidate actions.Source: §1 · p. 2

Training

• DriveWorld-VLA uses three-stage progressive training to align multimodal perception, enforce action-conditioned controllability, and refine decisions through reward feedback.The stages jointly train imagination and action prediction, add controllability, and close the loop with reward-guided refinement.Source: §1 · p. 2

Results

• Extensive open-loop and closed-loop evaluations report state-of-the-art performance and improvements over specialized baselines in reasoning accuracy and safety-critical planning.The evaluations cover NAVSIM and nuScenes benchmarks.Source: §1 · p. 2

World Models

• World-model research for autonomous driving models environmental evolution through video sequences, point-cloud synthesis, and related scene representations.The cited examples include DriveWorld, Vista, LidarDM, and Copilot4D.Source: §2 · p. 2

VLAs

• VLA research has progressed from high-level semantic interpretation toward end-to-end mappings from multimodal inputs and instructions to control trajectories.Recent work also explores integrating VLAs with world models.Source: §2 · pp. 2–3

DriveWorld-VLA Pipeline

• DriveWorld-VLA’s pipeline jointly learns future BEV imagination and action prediction from a shared latent representation in Stage 1.This stage establishes the shared representation used by the later controllability and refinement stages.Source: Figure 2 · p. 3
• Stage 2 conditions the generative branch on future actions, mapping action sequences to corresponding futures for controllable imagination.This action conditioning provides the pipeline’s feature-level what-if capability.Source: Figure 2 · p. 3
• Stage 3 predicts actions, imagines their resulting future, and uses reward feedback to refine action prediction.The training process is organized as guided evaluation and refinement after joint training and controllability fine-tuning.Source: Figure 2 · p. 3

3.1. VLA & WM Joint Training

• DriveWorld-VLA jointly feeds multimodal inputs into a VLM and uses final-layer hidden states as a shared latent representation for future imagination and action prediction.Inputs include multi-view images, textual prompts, historical actions, and BEV representations.Source: §3.1 · pp. 3–4
• Future imagination predicts future BEV states from the shared representation and current BEV state, while a lightweight action decoder forecasts future actions from latent, BEV, and historical-action inputs.The denoiser’s history-conditioned branch provides dense future supervision, while the action decoder is trained with expert-action imitation supervision.Source: §3.1 · p. 4

3.2. Action Controllability Fine-Tuning

• Because joint training is not conditioned on future actions, it cannot close the reasoning loop between actions and future scene generation.The limitation motivates explicit action-conditioned future imagination rather than relying only on historical-observation extrapolation.Source: §3.2 · p. 4
• Action controllability fine-tuning uses a DiT-based flow-matching denoiser conditioned on the BEV state and ground-truth future actions to model controllable future evolution.The denoiser processes BEV features through normalization, scaling, embedding, and DiT blocks; flow-matching loss is the sole loss in this stage.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3. Future-Guided Evaluation & Refinement

• Future-guided refinement predicts actions, generates action-conditioned future BEV imagination, and evaluates trajectories using a learned reward function tied to simulator-based online rewards.The denoising branches use Euler-based sampling with 25 sampling steps, and the reward-driven process prioritizes higher-reward trajectories for action refinement.Source: §3.3 · p. 5
• During refinement, the denoiser and VLM remain frozen while reward and action components are trained using action, semantic-BEV, and reward supervision.Future BEV latents from both denoising branches are fused before semantic BEV decoding.Source: §3.3 · p. 5

Main Results

• 91.3 PDMS on NAVSIMv1, outperforming top methods across different paradigms.The comparison includes DiffusionDrive, WoTE, and DriveVLA-W0.Source: §4.2 · pp. 5–6
• 86.8 EPDMS on NAVSIMv2, again outperforming all compared methods.DriveWorld-VLA also achieves 99.1 DAC, 99.6 DDC, and 97.0 LK.Source: §4.2 · p. 6
• 0.16% collision rate and 0.61m average L2 on the nuScenes validation set.The evaluation uses 3-second planning results and disables ego-state information.Source: §4.2 · p. 6

Ablation Study

• Stage 2 and Stage 3 increase NAVSIMv1 PDMS by +1.9 and +1.8, respectively, across the training process.The ablation compares training stages across NAVSIMv1 and nuScenes.Source: §4.3 · p. 6, Table 4 · p. 7
• -7.7 PDMS under non-progressive training, indicating that the progressive strategy is effective.The non-progressive strategy updates the future-imagination and action-prediction branches simultaneously after Stage 1.Source: §4.3 · p. 7, Table 5 · p. 7
• The model must first learn from ground-truth actions and enhance the latent feature space before benefiting from action prediction.The ablation further reports that generative and planning tasks need asynchronous unification even within the same feature space.Source: §4.3 · p. 7
• Omitting VLM pre-training or fully freezing VLM parameters limits performance, while the optimal strategy includes initial VLM optimization.The VLM strategy ablation compares freezing and pre-training choices during Stage 1.Source: §4.3 · p. 7, Table 6 · p. 7
• Future imagination in Stage 3 produces more robust trajectory predictions and significantly reduces collision risk relative to Stage 2.The qualitative visualization attributes the changes across stages to world-modeling accuracy and physical-dynamics understanding.Source: Figure 4 · p. 8

• DriveWorld-VLA shares scene representations in latent space so VLA decisions can use information about future scene evolution.Source: §5 · p. 8
• World-model latent states serve as decision-making states for evaluating the long-term impact of candidate actions.Source: §5 · p. 8
• DriveWorld-VLA was evaluated on multiple closed-loop and open-loop planning benchmarks.Source: §5 · p. 8
• The reported results significantly outperform current state-of-the-art methods in decision-making.Source: §5 · p. 8

• The work integrates Vision-Language-Action models and World Models to improve autonomous-driving decision-making and forward-looking imagination.Source: Impact Statement · p. 9
• The stated societal goal is to contribute to safer transport through autonomous vehicle technology.Source: Impact Statement · p. 9

• The InternVL system prompt describes the task, metrics, inputs, and outputs.Source: §A.1 · p. 12
• Figure S1 provides an example of the system prompt for the two benchmarks.Source: §A.1 · p. 12

• Visual inputs are represented as a text-domain visual placeholder token sequence within the input.Source: §A.2 · p. 12
• Images are adaptively partitioned into 448 × 448 feature patches to accommodate varying aspect ratios.Source: §A.2 · p. 12
• When multiple patches are used, an additional 448 × 448 thumbnail supplies a global view.Source: §A.2 · p. 12

Vision-language fusion

• Fused sequence hidden states H combine textual context with visual information at image-context positions.The hidden states have shape H ∈ R^(B×L×D), where L is the maximum tokenized sequence length and D is the language-backbone dimension.Source: §A.3 · p. 12

Latent compression

• The model projects hidden states from D = 1536 to d = 256 before latent aggregation.Source: §A.3 · p. 12
• 700 learnable latent query vectors summarize the variable-length sequence into a fixed-length representation through cross-attention.The queries serve as keys-and-values aggregators over the projected hidden states, producing Z ∈ R^(B×N_L×d).Source: §A.3 · p. 12

Dataset coverage

• The supplementary examples cover both NAVSIM and nuScenes validation data.Source: B. More Visualization · p. 12

Prompt representation

• InternVL prompts represent ground-truth historical trajectories as 2D-coordinate sequences and navigation commands as text.Examples of commands include “turn left,” “go straight,” and “turn right.”Source: Figure S1 · p. 13

NAVSIM

• NAVSIM visualization figures label sample tokens on the left and trajectory sources at the top.Source: Figure S2 · p. 14, Figure S3 · p. 15, Figure S4 · p. 16

nuScenes

• nuScenes validation visualizations identify the source of each trajectory at the top of the figures.Source: Figure S7 · p. 19, Figure S8 · p. 20