PaperlayerView exact revision on arXiv

Source-linked AI summary

The DAWN of World-Action Interactive Models

Hongbo Lu, Liang Yao, Chenghao He, Haoyu Wang, Xiang Gu, Xianfei Li, Wenlong Liao, Tao He, Pai Peng

arXiv:2605.11550v1cs.CV

TL;DR

Existing WAMs inadequately model the action-contingent reciprocity between future scene evolution and maneuver selection. The paper introduces WAIMs and DAWN, which recursively couple latent world prediction with action denoising through a short rollout; DAWN reports strong planning and safety results across autonomous-driving benchmarks.

  • Problem

    Existing WAMs decouple world and action generation, although decision-relevant driving futures depend on the maneuver being considered.

  • Method

    DAWN instantiates WAIM with a compact latent World Predictor and World-Conditioned Action Denoiser that recursively update one another through a short explicit rollout.

  • Results

    DAWN achieves strong overall planning and safety-oriented performance across benchmarks, including the best NAVSIM v1 PDMS of 89.1 and best Time-to-Collision score.

  • Takeaways & Limitations

    Interactive world-action generation provides a practical path toward safer, more actionable autonomous-driving models.

  • Takeaways & Limitations

    DAWN's short latent rollout may be insufficient for long-range anticipation or extended multi-agent interaction, and benchmark gains do not establish deployment readiness.

Abstract

from arXiv · show

A plausible scene evolution depends on the maneuver being considered, while a good maneuver depends on how the scene may evolve. Existing World Action Models (WAMs) largely miss this reciprocity, treating world prediction and action generation as either isolated parallel branches or rigid predict-then-plan pipelines. We formalize this perspective as World-Action Interactive Models (WAIMs), and instantiate it in autonomous driving with \textbf{DAWN} (\textbf{D}enoising \textbf{A}ctions and \textbf{W}orld i\textbf{N}teractive model), a simple yet strong latent generative baseline. DAWN operates in a compact semantic latent space and couples a \emph{World Predictor} with a \emph{World-Conditioned Action Denoiser}: the predicted world hypothesis conditions action denoising, while the denoised action hypothesis is fed back to update the world prediction, so that both are recursively refined during inference. Rather than eliminating test-time world evolution altogether or rolling out the full future in pixel space, DAWN performs a short explicit latent rollout that is sufficient to support long-horizon trajectory generation in complex interactive scenes. Experiments show that DAWN achieves strong planning performance and favorable safety-related results across multiple autonomous driving benchmarks. More broadly, our results suggest that interactive world-action generation is a principled path toward truly actionable world models.

1 Introduction

Existing WAMs decouple world prediction from action generation, despite decision-relevant futures depending on candidate actions. WAIMs address this reciprocity through recursive world-action inference, instantiated by DAWN with a short latent rollout for autonomous driving.

  • Decision-relevant driving futures depend on the ego maneuver, including gap feasibility, yielding, and safety-critical interactions.
  • WAIMs retain a short latent rollout rather than requiring either full pixel-space evolution or no explicit test-time rollout.
  • Existing WAMs use parallel branches or sequential predict-then-plan pipelines that do not let world and action hypotheses iteratively reshape one another.
  • WAIMs infer future world states and actions together, recursively refining each hypothesis to form a mutually aligned future-action pair.
  • DAWN couples a World Predictor with a World-Conditioned Action Denoiser, using predicted worlds to condition actions and denoised actions to update world predictions.
  • 89.1 PDMS on NAVSIM v1 and the best Time-to-Collision score demonstrate DAWN's reported planning and safety performance on that benchmark.

2 Methodology

DAWN formulates world and action inference as recursive interaction in a compact latent space. Its staged architecture combines latent compression, world prediction, action denoising, and trajectory refinement for planning.

  • Problem formulation: DAWN defines a WAIM as jointly inferring future world and action through interaction rather than independent generation or a fixed one-way order.This extends WAMs, which model future world and action together but do not necessarily iteratively update one from the other.
  • Latent representation: The model compresses dense visual encoder features into compact latent world tokens with an Auto-Encoder Resampler.The resampler is trained as a token-space bottleneck intended to preserve information needed for future prediction and action generation.
  • Interactive architecture: A causal World Predictor forecasts future latent tokens from latent context and an action hypothesis, while a World-Conditioned Action Denoiser uses the predicted world to denoise actions.The two modules form the core recursive interaction, with denoiser weights shared across proposal and refinement roles.
  • Training: DAWN is trained in stages spanning vision pretraining, resampler training, World Predictor training, and joint world-action training.The final stage trains the predictor and action denoiser together, with the denoiser first proposing and then refining actions conditioned on predictor rollouts.
  • Inference: At inference, the denoiser initializes an action from the latent context, alternates short latent rollouts with action denoising, and decodes the final action state into a trajectory.The same architecture supports planning from scratch and interactive refinement of an input trajectory.

3 Experiments

DAWN is evaluated on NAVSIM v1 and nuScenes, with ablations testing its components, interactive refinement, latent capacity, coupling directions, rollout horizon, and qualitative behavior.

  • Evaluation Setup: DAWN is evaluated on NAVSIM and nuScenes using planning, collision, progress, comfort, and time-to-collision metrics.NAVSIM reports aggregate PDMS and rule-based metrics; nuScenes reports trajectory L2 error and collision rate across multiple horizons.
  • Main Results: 89.1 PDMS is DAWN’s best perception-free NAVSIM v1 score, alongside the best NC, Ego Progress, and Time-to-Collision scores.The full-resolution model improves over DAWN* from 87.9 to 89.1 PDMS.
  • Main Results: 0.33 m average L2 error is DAWN’s nuScenes result, compared with 0.47 m for WorldRFT, while collision-related metrics remain leading or tied-leading.DAWN also reaches 0.31 m at 2 s and 0.52 m at 3 s.
  • Ablations: 82.8 to 83.2 PDMS results from increasing Resampler output tokens from 16 to 64, although larger latent representations increase rollout and denoising cost.The analysis links aggressive compression to possible loss of planning-relevant scene structure.
  • Ablations: 81.6 and 84.9 PDMS result when World→Action and Action→World interaction directions are removed, respectively, versus full DAWN.The results support bidirectional coupling during world rollout and action denoising.
  • Ablations: A 2–3 s latent rollout approaches the full 4 s result, whereas zero rollout performs clearly worse and longer horizons increase latency.The study therefore treats short explicit rollout as a useful efficiency–performance compromise.
  • Qualitative Results: DAWN generates trajectories consistent with human motion and local road topology in intersections, narrow streets, constrained roads, and curved junctions.Qualitative comparisons include front-view observations and corresponding BEV visualizations.

4 Related Work

Related work spans general world models, end-to-end autonomous driving, and driving world models, with increasing attention to latent prediction and planning-aligned representations.

  • World and Action Models: World models learn environmental evolution to support prediction, planning, and decision making, while WAMs jointly model future world states and actions.V-JEPA 2 provides predictive representations, but the cited discussion distinguishes passive prediction from action-coupled modeling.
  • End-to-End Autonomous Driving: End-to-end driving maps raw sensor inputs directly to actions, while related systems unify planning tasks, model uncertainty, or learn planning-aligned representations.Examples include UniAD, VADv2, SparseDrive, Drive-JEPA, Orion, and UniDriveVLA.
  • Driving World Models: Driving world models generally learn forward scene dynamics as internal simulators from visual histories, with Drive-JEPA additionally combining video pretraining and trajectory distillation.The cited examples include GAIA-1, Drive-WM, and Drive-JEPA.

5 Conclusion

The paper introduces WAIMs as a coupled alternative to decoupled pipelines and instantiates the idea in DAWN through short latent world-action interaction.

  • Conclusion: WAIMs infer future world states and actions as coupled variables rather than producing them through decoupled pipelines.DAWN couples a World Predictor with a World-Conditioned Action Denoiser through short explicit latent rollout.
  • Conclusion: DAWN’s design improves planning quality, interactive safety, and trajectory smoothness while remaining efficient at inference time.The authors position interactive world-action generation as a direction for more actionable autonomous systems.

A Limitations

The paper identifies limitations in both the WAIM formulation and DAWN’s short latent-rollout instantiation, including missing guarantees, limited interpretability, and dataset dependence.

  • Formulation Limitations: WAIM may be unnecessary in simpler settings, and its recursive interaction currently lacks formal convergence or safety guarantees.The formulation is presented as suitable for action-contingent interactive decision-making rather than universally optimal.
  • Instantiation Limitations: DAWN’s short latent rollout may be insufficient for long-range anticipation or extended multi-agent interaction.Compact latent interaction also makes future representations less interpretable and complicates diagnosis of rare safety-critical cues.
  • Scope Boundary: Benchmark gains should not be interpreted as deployment readiness because WAIM and DAWN depend on pretraining and downstream driving-data coverage.The paper also notes risks from over-trust and uneven generalization across regions, environments, or traffic conditions.

C.1 Datasets and Metrics

The evaluation covers four autonomous-driving benchmarks across open-loop and closed-loop settings, using benchmark-specific planning, comfort, compliance, accuracy, and safety metrics.

  • Four autonomous-driving benchmarks are used to evaluate DAWN in both open-loop and closed-loop settings.
  • NAVSIM v1: NAVSIM v1 reports NC, DAC, EP, C, TTC, and aggregate PDMS under its simulator-based rule-metric protocol.
  • NAVSIM v2: NAVSIM v2 extends PDMS to EPDMS and adds DDC, TL, LK, HC, and EC alongside NC, DAC, EP, and TTC.
  • nuScenes: nuScenes reports trajectory L2 error and Collision Rate at 1 s, 2 s, and 3 s, plus their horizon averages.
  • Higher values are better for NAVSIM metrics, whereas lower values are better for nuScenes L2 error and collision rate.

C.2 Detail Experimental Settings

DAWN uses a video-pretrained latent architecture with compact token compression, a causal world predictor, and a diffusion-based action denoiser, trained with large-scale optimization resources.

  • Inputs are sampled at 2 Hz, with four observed frames used to predict 12 future latent states.The crop size is 512 × 256.
  • The ViT-Large V-JEPA 2 backbone uses patch size 16 and tubelet size 2 for visual representation learning.
  • The Auto-Encoder Resampler compresses dense encoder tokens into 16 latent tokens and uses an auxiliary diffusion planner head to preserve action-relevant information.
  • The World Predictor is a 12-layer causal Transformer with embedding dimension 384, 12 attention heads, RoPE, and activation checkpointing.
  • The World-Conditioned Action Denoiser is a DiT-style diffusion planner whose timestep/status conditioning modulates self-attention, MLP, and latent-world cross-attention branches.
  • Training runs for 150 epochs with peak learning rate 1 × 10^-4, weight decay 0.04, eight warmup epochs, EMA, and large-scale A100 training.The full-scale experiments use 80 NVIDIA A100 GPUs.

D.1 Comparison with existing SOTA Methods on other Datasets

On NAVSIM v2, DAWN is evaluated against representative perception-based baselines under the official protocol, achieving leading extended-comfort performance while remaining competitive on several related measures.

  • Table 7 reports perception-based baselines and DAWN under the same official NAVSIM v2 evaluation protocol, with EPDMS as the aggregate score.
  • DAWN achieves the best extended-comfort score on NAVSIM v2 and competitive lane-keeping performance.
  • DAWN maintains strong traffic-light compliance and history-comfort scores, indicating smooth and stable trajectories.
  • DAWN's aggregate EPDMS is lower than the strongest baselines, mainly because of weaker drivable-area compliance and collision-related performance.

D.2 Detailed Ablation Studies

Ablations show that explicit latent prediction and interactive refinement drive performance gains, while compact short rollouts balance planning quality against inference efficiency.

  • Components: Adding the World Predictor improves PDMS from 82.8 to 85.2, while enabling interactive refinement further raises PDMS to 87.9.
  • Components: Interactive coupling consistently improves safety-related metrics including TTC and DAC, beyond the benefit of better world modeling alone.
  • Interactive rounds: Increasing interactive rounds from 1 to 4 raises PDMS from 85.2 to 87.9 while also improving NC, DAC, EP, and TTC.
  • Interactive rounds: Beyond four rounds, PDMS drops to 87.2 with five rounds and 86.9 with six rounds, so four rounds are used by default.
  • Latent tokens: Expanding latent tokens from 16 to 64 raises PDMS from 82.8 to 83.2 but increases inference latency from 331.3 ms to 963.6 ms.
  • Latent rollout: The rollout design space spans zero-rollout, short-rollout, and full predict-then-plan regimes, with DAWN positioned in the short-rollout middle.
  • Latent rollout: Most performance gains appear with a short latent future, while longer rollout provides smaller additional benefits and full scene reconstruction is unnecessary.

E.1 Planning Results

The supplied material presents DAWN’s training and inference procedures alongside qualitative results for planning, prediction, and feature representations. Inference uses compact latent context, short future rollout, action refinement, and final trajectory decoding.

  • Qualitative Results: Qualitative results cover planning, prediction, and feature representations across Figures 6–11.The supplied captions identify Figure 6 as planning, Figures 7–9 as prediction, and Figures 10–11 as feature results.
  • Training: The training setup uses pretraining and task datasets, student and teacher encoders, resamplers, a world predictor, an action denoiser, and an action head.The algorithm specifies task examples containing current observation, instruction, next observation, and target trajectory.
  • Training: Training proceeds through vision pretraining, token-space resampler training, world-predictor training, and joint world-action training.The joint stage initializes the world predictor from Stage 3 and attaches the action denoiser and action head before task-data training.
  • Inference: DAWN inference extracts compact latent context, encodes non-visual conditions, and initializes an action hypothesis from the resampler latent.The inference procedure then uses the initialized hypothesis as the basis for subsequent interaction rounds.
  • Inference: DAWN rolls out a short latent future conditioned on the current action, refines the action with that future, and decodes the final action state into a predicted trajectory.These steps explicitly couple world prediction and action refinement during inference.
Loading 2605.11550v1…