PaperlayerView exact revision on arXiv

Source-linked AI summary

DriveWLA: Unifying Future Dynamics and Scene Semantics through World-Language-Action Modeling for Autonomous Driving

Shuzheng Wang, Haoxuan Li, Shengzhen Li, Chenxi Wu, Yajing Sun, Lu Hou, Wenyu Liu, Xinggang Wang

arXiv:2610.17903v1cs.CVcs.AIcs.RO

TL;DR

Autonomous driving needs semantic understanding and future scene dynamics together for trajectory planning, but integrating them effectively remains challenging. DriveWLA combines World, Language, and Action Experts with controlled cross-expert attention and reinforcement-learning refinement. It reports state-of-the-art NAVSIM performance and zero-shot generalization across open- and closed-loop benchmarks.

  • Problem

    VLMs lack explicit mechanisms for future scene evolution, while integrated systems do not explicitly model interaction between semantic and dynamic representations for trajectory planning.

    Source: §1 · pp. 1–2

  • Method

    DriveWLA jointly models semantic understanding, future dynamics, and trajectory generation with three experts, Cross-Expert Attention, and Flow-GRPO refinement of the Action Expert.

    Source: §1 · p. 2

  • Results

    DriveWLA achieves state-of-the-art NAVSIM performance and strong zero-shot generalization across open- and closed-loop benchmarks, including 92.3 PDMS on NAVSIM v1.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The results support World-Language-Action modeling as a promising paradigm and DriveWLA as a potential strong baseline for future autonomous driving research.

    Source: §6 · p. 9

    Prior integrated systems are limited by not explicitly modeling interaction toward trajectory planning, leaving semantic and dynamic representations insufficiently refined and aligned.

    Source: §1 · p. 2

Abstract

from arXiv · show

Vision-Language Models (VLMs) and world models have demonstrated strong capabilities in semantic understanding and future scene evolution for autonomous driving. However, effectively integrating both capabilities for trajectory generation remains challenging. In this work, we propose DriveWLA, a unified World-Language-Action model based on Mixture-of-Transformers (MoT) that jointly models semantic understanding, future scene dynamics, and trajectory generation. DriveWLA consists of three experts: a World Expert for future prediction, a Language Expert for semantic understanding, and an Action Expert for trajectory generation. We further introduce Cross-Expert Attention, which enables controlled information flow across experts while aligning semantic and dynamic representations. To improve trajectory generation beyond imitation learning, we apply Flow-GRPO reinforcement learning post-training to refine the Action Expert. Extensive experiments show that DriveWLA achieves superior performance in both open-loop and closed-loop evaluation. DriveWLA achieves 92.3 PDMS on NAVSIM v1 and 90.1 EPDMS on NAVSIM v2, demonstrating strong performance among VLM-based and world-model-based planners. Without specific fine-tuning, DriveWLA further demonstrates strong zero-shot generalization to HUGSIM, nuScenes, and Bench2Drive. These results validate the effectiveness of our unified World-Language-Action modeling, establishing DriveWLA as a generalizable framework for autonomous driving.

1 INTRODUCTION

Autonomous driving must combine scene understanding, future prediction, and trajectory planning, but integrating semantic and dynamic representations for planning remains challenging. DriveWLA unifies these capabilities through three experts and controlled cross-expert information flow, and reports strong benchmark performance and zero-shot generalization.

1 INTRODUCTION

  • Reliable autonomous driving requires scene understanding, future prediction, and trajectory planning, motivating their joint use for appropriate driving decisions.The paper connects this challenge to how human drivers combine current scene understanding with future prediction.Source: §1 · p. 1
  • VLMs support semantic understanding but lack explicit mechanisms for future scene evolution, while future prediction alone cannot determine which changes matter for driving.The paper frames semantic understanding and future modeling as complementary capabilities for trajectory planning.Source: §1 · p. 1
  • DriveWLA combines a World Expert for future prediction, a Language Expert for semantic understanding, and an Action Expert for trajectory generation through Cross-Expert Attention.All experts are jointly trained with trajectory supervision during imitation learning, followed by Flow-GRPO refinement of the driving policy beyond expert demonstrations.Source: §1 · p. 2
  • 92.3 PDMS on NAVSIM v1 establishes DriveWLA as state of the art among recent VLM-based and world-model-based planners.It also achieves 41.47 HD-Score on HUGSIM and maintains low collision rates on nuScenes and Bench2Drive without fine-tuning.Source: §1 · p. 2

2 RELATED WORK

Prior work studies VLM-based semantic reasoning, world-model future prediction, and reinforcement learning for driving, with methods combining semantic and dynamic capabilities in several ways. DriveWLA distinguishes itself by jointly modeling three experts with explicit cross-expert information exchange.

2 RELATED WORK

  • VLM-based systems use textual reasoning or latent representations for semantic understanding, while world models learn predictive representations of future scene evolution from driving videos.World-model approaches include cascaded prediction followed by planning and joint formulations that model future dynamics with action generation.Source: §2 · p. 2
  • Figure 1 depicts the three-expert architecture and training sequence: joint trajectory imitation learning followed by reinforcement learning refinement of the Action Expert.The caption identifies Cross-Expert Attention as the mechanism unifying the experts.Source: Figure 1 · p. 3
  • DriveWLA jointly models World, Language, and Action Experts through Cross-Expert Attention for continuous information exchange and representation alignment.Prior integration strategies include auxiliary future-image prediction or conditioning one paradigm's representations into another architecture.Source: §2 · pp. 2–3
  • Driving reinforcement learning improves behavior-cloned policies with task-level rewards, and DriveWLA uses factorized advantages and variance-based shrinkage during policy optimization.The cited approaches include reinforcement learning for end-to-end models and generative trajectory planners.Source: §2 · p. 3

3 PRELIMINARIES

The problem definition formulates autonomous driving as generating a safe and smooth future trajectory from sensor, history, navigation, and ego-status inputs.

3 PRELIMINARIES

  • The driving system generates a safe and smooth future trajectory from camera input, historical trajectories, navigation commands, and ego status.The formulation describes the end-to-end system trajectory-generation process.Source: §3 · p. 3
  • The predicted ego trajectory covers the next T time steps, with position and heading expressed in the current ego-centric coordinate frame.Each time step is represented by x_t, y_t, and θ_t.Source: §3 · p. 3

4 METHOD

DriveWLA combines specialized World, Language, and Action Experts, using Cross-Expert Attention for controlled information exchange and Flow-GRPO with factorized advantage estimation to refine trajectory generation.

4.1 MODEL ARCHITECTURE

  • Three experts encode future-scene latents, transform frozen VLM hidden states into planning features, and generate trajectories with a flow-matching policy.During imitation learning, the World Expert, VLM Adapter, and Action Expert are jointly optimized with a flow-matching behavior-cloning objective.Source: §4.1 · p. 4

4.2 CROSS-EXPERT ATTENTION

  • Cross-Expert Attention enables controlled exchange and progressive alignment, with bidirectional World–Language visibility while Action reads both without feeding information back.The experts retain their own transformer blocks while exchanging information through masked self-attention.Source: §4.2 · pp. 4–5

4.3 FLOW MATCHING GROUP RELATIVE POLICY OPTIMIZATION

  • Flow-GRPO post-training optimizes Action Expert LoRA with behavior-cloning regularization, using normalized per-metric rewards and variance shrinkage to stabilize updates.The advantage combines weighted normalized metric scores and reduces unreliable updates from low-variance trajectory groups.Source: §4.3 · p. 5

5 EXPERIMENTS

DriveWLA is evaluated on NAVSIM and in zero-shot transfer benchmarks, with ablations testing expert composition, training, information flow, and sampling. It achieves the highest reported NAVSIM scores and shows strong zero-shot results, while the ablations identify complementary experts and effective training and inference choices.

5.1 EXPERIMENTAL SETUP

  • DriveWLA trains on 103,288 NAVSIM navtrain scenes and evaluates on 12,146 navtest scenes; its Action Expert predicts eight waypoints over four seconds.The evaluation also includes zero-shot closed-loop HUGSIM and open-loop nuScenes and Bench2Drive transfer.Source: §5.1 · p. 6

5.2 MAIN RESULTS

  • 92.3 PDMS on NAVSIM v1 and 90.1 EPDMS on NAVSIM v2 are the best overall scores reported among compared planners.On NAVSIM v1, DriveWLA exceeds the best VLM-based and world-model-based planners by 1.2 and 0.8 points, respectively; on NAVSIM v2, it leads across multiple evaluation dimensions.Source: §5.2 · p. 7
  • Zero-shot, DriveWLA leads 8 of 11 HUGSIM metrics, beats listed nuScenes methods at 0.94 m average L2 error and 0.09% collisions, and records Bench2Drive’s lowest collision rate, 1.06%.On HUGSIM, DriveWLA scores higher than DrivoR on Medium, Hard, and Extreme scenarios; on nuScenes, it maintains a comparable L2 error but lower collision rate than fine-tuned planners.Source: §5.2 · pp. 7–8, Table 3 · p. 7, Table 4 · p. 7

5.3 ABLATION STUDIES

  • 90.8→86.2 PDMS without the World Expert, versus 89.1 without the Language Expert, indicates the larger degradation from removing the World Expert in this setting.The results also show that using both experts gives complementary benefits over either representation alone.Source: §5.3 · p. 8, Table 5 · p. 8
  • 92.3 PDMS with Flow-GRPO† follows 90.8 with joint World Expert optimization and 92.1 with original Flow-GRPO; freezing the World Expert yields 88.0.The authors suggest that video-prediction dynamics are not fully aligned with driving requirements before joint optimization.Source: §5.3 · p. 8, Table 6 · p. 8
  • 90.8 PDMS is highest with bidirectional World–Language exchange and Action receiving both; allowing Action-to-World or Action-to-Language flow degrades performance.Removing either direction of World–Language exchange also lowers performance, and removing both causes a further drop.Source: §5.3 · p. 8, Table 7 · p. 8
  • 10 sampling steps achieve the best measured PDMS, 90.8, at 970 ms; more steps add latency without improving performance.One step yields 49.2 PDMS, while five steps reach 90.6.Source: Table 8 · p. 8, §5.3 · p. 9

5.4 QUALITATIVE ANALYSIS

  • RL refinement raises DAC from 0 to 1 and yields a valid trajectory distinct from the expert demonstration; expert removals fail in different left-turn scenes.The qualitative comparisons use NAVSIM navtest bird’s-eye views.Source: §5.4 · p. 9, Figure 3 · p. 9

6 CONCLUSION

The conclusion presents DriveWLA as a unified framework in which semantic understanding and future dynamics provide complementary information for trajectory planning. It reports state-of-the-art NAVSIM performance and zero-shot generalization, and identifies World-Language-Action modeling as a promising research direction.

6 CONCLUSION

  • DriveWLA combines semantic understanding and future dynamics through controlled cross-expert interaction for trajectory planning.The conclusion describes the two information sources as complementary.Source: §6 · p. 9
  • The authors report state-of-the-art NAVSIM performance and strong zero-shot generalization across open-loop and closed-loop benchmarks.Source: §6 · p. 9
  • The authors report using large language models to edit the manuscript for clarity and identify relevant prior work.Source: AI USE STATEMENT · p. 10

A IMPLEMENTATION DETAILS

The implementation configures separate World, Language, and Action experts, trains their transformer components jointly with trajectory supervision, and uses a branch-wise attention mask and deterministic Euler sampling.

A.1 EXPERT CONFIGURATION

  • The World Expert uses a driving-video-pretrained model, while its video VAE and text encoder stay frozen and its DiT is frozen during Flow-GRPO refinement.The World DiT is optimized during imitation learning.Source: §A.1 · p. 14
  • The frozen driving-pretrained VLM feeds final-layer hidden states to a VLM Adapter, which is optimized during imitation learning and frozen during Flow-GRPO refinement.The VLM processes the current front-view image using dynamic 448 × 448 tiling.Source: §A.1 · p. 14
  • The Action Expert uses four historical poses, current ego state, and navigation command to generate eight waypoints over four seconds; Flow-GRPO updates only its rank-16 LoRA parameters.Inference uses ten deterministic Euler steps.Source: §A.1 · p. 14
  • The World DiT, VLM Adapter, and Action DiT train jointly for 30 epochs with flow matching, and trajectory supervision also updates the first two through Cross-Expert Attention.Imitation learning uses no auxiliary video-prediction or text-generation loss.Source: §A.1 · p. 14

A.2 CROSS-EXPERT ATTENTION MASK

  • The Cross-Expert Attention mask indexes query-token rows and key-token columns; zero permits attention, while −∞ blocks it across branch token pairs.The arrows indicate information flow from key/value branches to query branches.Source: §A.2 · p. 14

A.3 TRAJECTORY SAMPLING

  • At inference, World hidden states condition the Action Expert without decoding future RGB frames, and ten deterministic Euler steps integrate its velocity field from τ = 1 to τ = 0.With fixed future latents, driving inputs, and initial Action noise, the Euler trajectory is deterministic.Source: §A.3 · p. 14

B EVALUATION PROTOCOLS

The protocols define NAVSIM v1 and v2 scores through progress, safety, comfort, and compliance components, with v2 adding further compliance terms and human-reference filtering.

B.1 NAVSIM V1

  • NAVSIM v1 PDMS combines ego progress, time-to-collision compliance, and comfort, with multiplicative penalties for no-at-fault collisions and drivable-area compliance.The protocol defines EP relative to a reference planner and TTC and comfort through collision risk and kinematic comfort.Source: §B.1 · pp. 14–15
  • The ResNet-34 entry for DriveWorld-VLA denotes its BEV encoder for latent world modeling, not a video-generation backbone.Source: §B.1 · p. 15

B.2 NAVSIM V2

  • NAVSIM v2 EPDMS adds driving-direction and traffic-light compliance penalties, retains EP and TTC, replaces comfort with history comfort, and adds lane keeping and extended comfort.The corrected evaluator applies human-reference filtering to NC, DAC, DDC, TLC, EP, TTC, LK, and HC, but not separately to EC.Source: §B.2 · p. 15

B.3 HUGSIM

The HUGSIM protocol evaluates DriveWLA without HUGSIM-specific training across scenarios from four datasets, using rollout scores and a difficulty-weighted aggregate for comparison.

B.3 HUGSIM

  • DriveWLA is evaluated without HUGSIM-specific training or fine-tuning on 436 scenarios spanning nuScenes, KITTI-360, Waymo, and PandaSet.The protocol uses the corrected HUGSIM evaluator and averages reported metrics over all scenarios.Source: §B.3 · p. 15
  • HUGSIM PDMS summarizes rollout safety and comfort, while HD-Score additionally accounts for route completion.HD-Score groups scenarios into four difficulty levels and weights their scores by scenario counts.Source: §B.3 · p. 15
  • The Tab. 3 comparison uses the same 436 scenarios, commands, corrected controller, metric implementation, and aggregation, but camera counts differ across methods.DriveWLA uses one front-view camera, LTF uses three, and VAD, UniAD, and DrivoR use four.Source: §B.3 · p. 15

B.4 NUSCENES

The nuScenes and Bench2Drive protocols assess recorded-observation trajectory prediction without further planner training on those datasets, using L2 error and collision rate at multiple horizons.

B.4 NUSCENES

  • On nuScenes, DriveWLA is evaluated on recorded observations without further planner training, using L2 error and collision rate averaged across 1-, 2-, and 3-second horizons.Collision rate follows the UniAD protocol.Source: §B.4 · p. 15
  • The nuScenes comparison uses the full 150-scene validation split and UniAD L2 and collision definitions, but the methods differ in training data and ego-state inputs.DriveWLA, DriveVLA-W0, and PWM do not use nuScenes training data; UniAD, OpenDriveVLA, and UniDriveVLA-L do.Source: §B.4 · pp. 15–16

B.5 BENCH2DRIVE

  • Bench2Drive evaluation uses recorded observations rather than interactive rollouts, with L2 error and collision rate averaged across 1-, 2-, and 3-second horizons and no further planner training.All listed methods are trained on NAVSIM and evaluated without Bench2Drive fine-tuning; these are open-loop metrics, not the interactive closed-loop Driving Score.Source: §B.5 · p. 16

C FLOW-GRPO TRAINING DETAILS

The appendix specifies a grouped, scene-level Flow-GRPO refinement procedure and reports qualitative evidence that World Expert latents organize observed and future context over time.

C FLOW-GRPO TRAINING DETAILS

  • Flow-GRPO samples eight trajectories per scene where the imitation policy scores below 90 PDMS, then combines standardized rewards using a factorized advantage.The rewards are NC, DAC, EP, TTC, and C, with weights 2, 2, 5, 5, and 2; τ = 0.1 shrinks variance, computed across each scene’s eight samples.Source: C FLOW-GRPO TRAINING DETAILS · p. 16
  • Training uses eight epochs and ten stochastic flow steps, updating only Action-branch LoRA parameters while freezing the World and Language Experts and base Action weights.Inference removes exploration noise and uses a deterministic Euler sampler.Source: C FLOW-GRPO TRAINING DETAILS · p. 16
  • The expert-composition, attention-mask, and sampling-step ablations use the pre-refinement imitation checkpoint; Flow-GRPO rows report subsequent refinement.The “Trainable World Expert” row also reports the 90.8-PDMS checkpoint.Source: C FLOW-GRPO TRAINING DETAILS · p. 16

D WORLD-EXPERT FEATURE VISUALIZATION

  • Across three scenes, PCA-projected World Expert latents show structured, scene-specific patterns evolving across observed history and a full 4-second future horizon.The visualization includes history latents 1 and 4 and future latents 1 and 2; each future latent summarizes four frames spanning two seconds.Source: Figure 4 · p. 16, D WORLD-EXPERT FEATURE VISUALIZATION · p. 17

Related papers

Loading 2610.17903v1…