Source-linked AI summary
SimWAM: A Simple World Action Model for End-to-End Autonomous Driving
Zongchuang Zhao, Xin Zhou, Tianyang Xu, Zhengyang Sun, Kaixuan Zhou, Honglin Li, Dingkang Liang, Xiang Bai
TL;DR
Existing world-action driving methods often require costly future-frame imagination during inference, motivating efficient transfer of video dynamics priors to action prediction. SimWAM co-trains video and action experts with joint flow matching and an isolated attention mask, then directly predicts trajectories; it reaches 91.5 PDMS on NAVSIM, lowers latency versus WAM planners, and transfers zero-shot to nuScenes.
Problem
Existing driving World-Action Models commonly place future-video synthesis in the real-time planning loop, increasing inference latency.
Method
SimWAM jointly trains a pretrained video expert and lightweight action expert with flow matching, using an isolated attention mask to keep action prediction independent of future frames.
Results
91.5 PDMS on NAVSIM, substantially lower latency than state-of-the-art world-model planners, and zero-shot transfer to nuScenes without fine-tuning.
Takeaways & Limitations
Training-time future-video generation can provide effective traffic-dynamics supervision while deployment retains direct trajectory inference without costly future imagination.
Abstract
from arXiv · showhide
World-Action Models (WAMs) improve end-to-end autonomous driving by transferring video dynamics priors to action prediction, but existing methods incur costly test-time future imagination. We present SimWAM, a simple yet effective WAM that leverages future-video prediction as a training-time supervision signal. It co-trains a pretrained video expert and a lightweight action expert with joint flow matching. An isolated attention mask keeps action prediction independent of future frames, allowing trajectory prediction without explicit future-frame generation at inference. Since the two experts share no parameters and interact only through a unified attention interface, the video backbone could be replaced and the action expert scaled independently without modifying the learning objective or inference pipeline. We further apply reinforcement learning to optimize a compositional driving reward beyond trajectory imitation. Our SimWAM achieves 91.5 PDMS on NAVSIM, surpasses state-of-the-art WAM-based planners with substantially lower latency, and transfers zero-shot to nuScenes. These results position SimWAM as a simple yet solid baseline that could readily benefit from advances in video generation for efficient autonomous driving. The code and model weights are available at https://github.com/H-EmbodVis/SimWAM/.
1 Introduction
Existing end-to-end and world-action planners model driving dynamics imperfectly or incur costly future-frame synthesis at inference. SimWAM uses future-video supervision during training while directly predicting trajectories, achieving strong NAVSIM performance with lower latency and cross-domain transfer.
- Motivation: World-action models transfer explicit environment-evolution priors to driving, but imagine-then-act planners increase real-time inference latency through future-frame synthesis.DriveLaW and DriveWAM jointly train video predictors and planners, yet condition trajectory outputs on generated future frames.
- Approach: SimWAM jointly trains a pretrained video expert and lightweight action expert with flow matching, while an isolated mask blocks action access to future frames.The action expert retains learned traffic-dynamics priors without auxiliary motion modules or explicit future-frame generation at deployment.
- Results: 91.5 PDMS on NAVSIM demonstrates SimWAM’s effectiveness with substantially lower inference latency than state-of-the-art world-model planners.The result is reported using the single-front-camera setting in the main results and conclusion.
- Results: SimWAM supports different pretrained video generators and transfers zero-shot to nuScenes without fine-tuning.The authors present these results as evidence of architectural scalability and cross-domain generalization.
2 Related Work
Related work progresses from unified end-to-end planning and semantic VLA reasoning toward world-action models that jointly learn future observations and actions. Reinforcement learning complements imitation by optimizing task-level driving rewards.
- End-to-end and VLA planning: End-to-end planners reduce modular interfaces and error propagation but remain largely dependent on expert trajectory supervision.This supervision provides limited support for explicit semantic reasoning about route intent and complex traffic interactions.
- End-to-end and VLA planning: Driving VLAs add semantic knowledge, temporal context, reasoning, and multimodal trajectory generation to improve scene understanding and connect planning with user intent.Representative systems include AutoVLA, ORION, FutureSightDrive, and ExploreVLA.
- World-Action Models: World-Action Models jointly learn action prediction and image generation to capture object motion, physical interactions, task progress, and future scene evolution.DreamZero and LingBot-VA extend pretrained video generation toward generalizable robotic control and closed-loop policy execution.
- Reinforcement learning: Reinforcement learning complements imitation learning by directly optimizing driving policies with task-level rewards.Prior work applies expert-guided rewards, privileged world models, and reinforcement learning to trajectory planning and VLA-based driving.
3 Preliminary
The preliminary method models future frames and trajectories with rectified flow, sampling by integrating a probability-flow ODE from noise to data. SimWAM converts this ODE into an SDE to enable stochastic exploration and tractable transition likelihoods for policy optimization.
- Flow matching: Rectified flow linearly interpolates clean targets and Gaussian noise, training a network to predict the constant interpolation velocity under conditioning.The interpolation is x_τ = (1−τ)x + τϵ, with velocity ϵ−x.
- Flow matching: Sampling integrates the probability-flow ODE from noise at τ=1 to data at τ=0.The learned vector field v_θ determines the trajectory through state space during sampling.
- From ODE to SDE: The deterministic ODE generates one trajectory and lacks a tractable transition density, limiting alternative-trajectory exploration and policy-gradient optimization.SimWAM follows Flow-GRPO by transforming the ODE into an SDE that preserves the same marginal distributions.
- From ODE to SDE: The resulting Euler–Maruyama transitions are isotropic Gaussian distributions with tractable log-likelihoods for importance sampling.The Wiener increment supplies stochasticity, while a controls the noise scale.
4 Method
SimWAM co-trains a pretrained video expert and lightweight action expert so future-video prediction supervises trajectory planning without future-frame rollout at inference. Its isolated attention design decouples the experts, enabling flexible video-backbone replacement and independent capacity scaling, while reinforcement learning further optimizes driving quality.
- Direct policy interface: SimWAM predicts trajectories directly from current observations, ego state, and navigation commands instead of conditioning on synthesized future-scene latents.The traffic-dynamics prior is learned through future-video supervision during training, avoiding future-scene generation and auxiliary motion modules at inference.
- Video expert: The video expert supplies a traffic-aware motion prior through future-video flow matching without a driving-specific prediction module.It uses a pretrained video-generation model whose future frames are noised and reconstructed during training.
- Action expert: The lightweight action expert uses flow matching to predict a trajectory velocity field and integrates an ODE from noise to generate the planned trajectory.At inference, explicit future-frame prediction is omitted and trajectories are generated directly.
- Co-training: Joint flow matching lets future-scene prediction shape the observation representation used for planning while the two experts retain their original architectures.The experts interact through shared attention rather than shared weights.
- Efficient inference and refinement: An isolated attention mask decouples action prediction from explicit future-scene generation, substantially reducing inference latency and supporting reinforcement-learning optimization of trajectories.The future-frame decoder can be discarded after training, while reinforcement learning refines the imitation-trained action policy toward driving quality.
- Architectural flexibility: The unified attention interface allows different pretrained video generators and independently sized action experts without changing the action objective or inference pipeline.Video and action capacity can be adjusted separately to balance representation quality, planning capacity, and computation budgets.
5 Experiments
Experiments show that SimWAM improves NAVSIM planning through joint video-action training and reinforcement learning, while avoiding future-frame generation at inference. The design remains flexible across video backbones and action-expert scales, and transfers zero-shot to nuScenes.
- Component analysis: 90.3 PDMS after joint video-action training improves the action-only DiT baseline from 86.6 PDMS, while reinforcement learning raises performance to 91.5.Joint training improves all reported metrics, and RL directly optimizes driving quality beyond imitation.
- Action expert scalability: 89.9 to 90.3 PDMS results from scaling the action DiT from 0.21B to 1.02B parameters.The parameter-independent architecture allows video and action experts to be adjusted separately.
- Cross-dataset generalization: 0.04% average collision rate is achieved zero-shot on nuScenes without fine-tuning or auxiliary annotations, with 0.96 m average L2 error.The safety result remains strong under domain shift from NAVSIM to nuScenes.
- Input resolution: 384×672 offers the strongest reported accuracy-efficiency balance, improving PDMS by 1.4 points over 192×352 for only 9 ms additional latency.Increasing resolution further to 768×1344 yields only a 0.3-point gain with considerably more computation.
6 Conclusion
SimWAM transfers traffic-dynamics priors from a pretrained video expert to direct trajectory planning, using isolated attention to avoid future-frame generation at inference. It further supports efficient deployment and broader driving optimization.
- SimWAM transfers traffic-dynamics priors from a pretrained video expert to a lightweight action expert through joint flow matching.
- An isolated attention mask decouples action prediction from future frames, enabling direct trajectory planning without explicit future-frame prediction at inference.
- The decoupled design makes the video backbone replaceable and lets the two experts scale independently without redesigning the planner or inference pipeline.
- 91.5 PDMS on NAVSIM was achieved using a single front camera, with zero-shot transfer to nuScenes.
- Reinforcement learning aligns trajectory generation with driving quality beyond imitation.