PaperlayerView exact revision on arXiv

Source-linked AI summary

DriveLaW:Unifying Planning and Video Generation in a Latent Driving World

Tianze Xia, Yongkang Li, Lijun Zhou, Jingfeng Yao, Kaixin Xiong, Haiyang Sun, Bing Wang, Kun Ma, Guang Chen, Hangjun Ye, Wenyu Liu, Xinggang Wang

arXiv:2512.23421v3cs.CV

TL;DR

World models can address long-tail challenges in autonomous driving, but current approaches often leave video prediction and motion planning decoupled. DriveLaW conditions a diffusion planner on video-generator latents, and reports state-of-the-art results in nuScenes video generation and NAVSIM planning.

  • Problem

    Existing autonomous-driving world models provide indirect or parallel planning support rather than tightly coupling video-generation representations to decision-making.

    Source: §1 · p. 1

  • Method

    DriveLaW chains a video generator and a diffusion planner, conditioning trajectory generation on the video model’s latent representation and training the components with a three-stage progressive strategy.

    Source: §1 · pp. 1–2

  • Results

    DriveLaW reports state-of-the-art nuScenes video-generation results and a NAVSIM PDMS of 89.1 without post-training or post-processing.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    The authors conclude that unifying video generation and planning is effective across nuScenes generation and NAVSIM planning.

    Source: §5 · p. 8

    The authors acknowledge limitations and future research opportunities, but do not specify their scope in this passage.

    Source: D. Limitations and Future Work · p. 16

Abstract

from arXiv · show

World models have become crucial for autonomous driving, as they learn how scenarios evolve over time to address the long-tail challenges of the real world. However, current approaches relegate world models to limited roles: they operate within ostensibly unified architectures that still keep world prediction and motion planning as decoupled processes. To bridge this gap, we propose DriveLaW, a novel paradigm that unifies video generation and motion planning. By directly injecting the latent representation from its video generator into the planner, DriveLaW ensures inherent consistency between high-fidelity future generation and reliable trajectory planning. Specifically, DriveLaW consists of two core components: DriveLaW-Video, our powerful world model that generates high-fidelity forecasting with expressive latent representations, and DriveLaW-Act, a diffusion planner that generates consistent and reliable trajectories from the latent of DriveLaW-Video, with both components optimized by a three-stage progressive training strategy. The power of our unified paradigm is demonstrated by new state-of-the-art results across both tasks. DriveLaW not only advances video prediction significantly, surpassing best-performing work by 33.3% in FID and 1.8% in FVD, but also achieves a new record on the NAVSIM planning benchmark.

1. Introduction

World models can help address long-tail brittleness in autonomous driving, but existing approaches do not tightly connect video generation to planning. DriveLaW chains video generation and planning through shared latents, and reports strong results on both tasks.

1. Introduction

  • Existing world models guide planning indirectly or in parallel, leaving video-generator latents out of the planner’s state.The paper identifies this as a gap between visual imagination and action selection.Source: §1 · p. 1
  • DriveLaW chains generation and planning through a shared latent-space representation rather than treating them as parallel modules.Its Action DiT is conditioned on video-generator latents that encode scene semantics, agent dynamics, and physical regularities.Source: §1 · pp. 1–2
  • DriveLaW-Video uses a spatiotemporal VAE, an efficient Video DiT, and noise reinjection to balance compression with video fidelity.DriveLaW-Act uses a diffusion planner trained with flow matching to produce smooth, reliable trajectories.Source: §1 · p. 2
  • A three-stage progressive curriculum learns long-horizon motion, refines spatial detail, then chains video latents into the planner.The authors present the curriculum as a way to harmonize generation and planning optimization.Source: §1 · p. 2
  • DriveLaW achieves state-of-the-art single-view video-generation FID and FVD on nuScenes and a new record for closed-loop planning metrics on NAVSIM.The reported planning result requires neither post-training nor auxiliary scorers.Source: §1 · p. 2

2. Related Work

Prior autonomous-driving world models serve as generators, simulators, or sources of supervision, while unified systems often generate video and trajectories as separate streams. DriveLaW instead uses video-generator latents as planning representations.

2.1. World Models for Autonomous Driving

  • World models for autonomous driving span scene generation, simulation, and decision-making, with some systems providing indirect planning guidance through synthesized data or closed-loop environments.Other approaches predict future visual or affordance signals, while unified models generate videos and trajectories together.Source: §2.1 · p. 2
  • Unified systems can still treat video and trajectory generation as independent streams, leaving planning disconnected from the features governing video synthesis.DriveLaW exploits mid-level video-generator latents as planning representations, which the authors associate with more stable closed-loop driving.Source: §2.1 · pp. 2–3

2.2. Video Generation

  • Driving-video methods use autoregressive tokens or diffusion conditioned on geometric features, and newer latent-diffusion systems target long-horizon, high-fidelity videos.The paper characterizes many existing video generators primarily as renderers.Source: §2.2 · p. 3

2.3. Diffusion Policies for Autonomous Driving

  • Autonomous-driving diffusion policies model trajectories, interactive scenes, cognition priors, or goal guidance; DriveLaW chains a Video DiT with a diffusion planner.The paper describes this chain as distilling driving priors from large-scale videos and injecting them into the planner.Source: §2.3 · p. 3

3. Method

DriveLaW chains video generation and trajectory planning through shared video latents. Its method builds driving representations from video generation, refines video forecasts, conditions a diffusion planner on video features, and trains the components progressively.

3. Method

  • The method frames its overall pipeline as learning generalizable driving representations, joining video generation to planning, and training the video and action components progressively.The video model provides the representations used by the planner.Source: §3 · p. 3, §3.2 · p. 3, §3.5 · p. 6

3.1. Learning Generalizable Driving Representations from Video Generators

  • Mid-denoising video features form a perception latent for planning, intended to transfer driving cognition priors distilled from video generation.The method selects one or a small set of denoising timesteps to form the latent.Source: §3.1 · p. 3

3.2. DriveLaW

  • DriveLaW-Video passes denoised video latents to DriveLaW-Act, whose Action DiT conditions on them to predict trajectories aligned with scene evolution.The video model encodes past driving frames and denoises in latent space; the action model also encodes action noise, ego status, and high-level commands.Source: §3.2 · p. 3, Figure 1 · p. 4

3.3. DriveLaW-Video: Spatiotemporal World Generator

  • The video generator combines compressed spatiotemporal latents with Transformer modeling, motion-conditioned text prompts, and a final pixel-space refinement for fine details.The final refinement recovers highlights, dynamic shadows, and fine road textures without an extra super-resolution module.Source: §3.3 · p. 4
  • Noise reinjection targets high-frequency regions before denoising, prompting the model to regenerate details that long-range motion can blur or ghost.The method identifies high-frequency regions from a temporary pixel-space prediction, perturbs only their corresponding latent regions, and uses the generative prior to restore plausible detail.Source: §3.3 · pp. 4–5

3.4. DriveLaW-Act: A Diffusion-Based Planner

  • DriveLaW-Act encodes noisy actions and driving context, then uses a flow-matching objective to generate trajectories conditioned on cached video features.The context includes ego status and a high-level command; the Action DiT also receives continuous time and cached features from Video DiT blocks.Source: §3.4 · p. 5

3.5. Three-Stage Progressive Training

  • The three-stage curriculum learns long-horizon motion at reduced resolution, improves visual detail at higher resolution, then trains DriveLaW-Act on video latents for planning.The first two stages use clips of 740 × 352 × 121 and 1280 × 704 × 25, respectively.Source: §3.5 · pp. 5–6

4. Experiment

DriveLaW is evaluated for video generation on nuScenes and trajectory planning on NAVSIM, with setup, benchmark results, and ablations testing its training and latent-conditioning choices. It reports leading results on both tasks and examines how pretraining scale, representation, denoising step, and training stages affect performance.

4.1. Experimental Setup

  • Experiments train on nuPlan and nuScenes, evaluate video generation on nuScenes and planning on NAVSIM, and use FID/FVD and PDMS, respectively.Video training uses 8 Hz camera data; planning uses 2 Hz data, with the planner supervised on trajectories over the next 4 seconds.Source: §4.1 · p. 6

4.2. Main Results

  • 4.6 FID and 81.3 FVD on nuScenes validation outperform prior single-view approaches in video-generation quality.Qualitative comparisons with Epona show clearer vehicle details, preserved pedestrian shapes, and correct maintenance of an inconspicuous yellow van.Source: §4.2 · pp. 6–7, Table 1 · p. 7, Figure 3 · p. 8
  • 89.1 PDMS on NAVSIM sets a new state of the art without post-training or post-processing, surpassing world-model and traditional end-to-end planners.DriveLaW exceeds Epona by 2.9 PDMS, DriveVLA-W0 by 1.9, and PWM by 1.0.Source: §4.2 · p. 6, Table 2 · p. 7
  • DriveLaW achieves lower L2 errors and collision rates than Epona on NuScenes open-loop planning evaluation.The evaluation reports both measures at 1s, 2s, 3s, and averaged.Source: §4.2 · p. 6, Table 3 · p. 7

4.3. Ablation Study

  • +3.2 PDMS separates a fully pretrained video generator from one without driving-domain pretraining, while increasing pretraining samples consistently improves NAVSIM closed-loop performance.The authors attribute the gain to larger corpora deepening the model’s grasp of driving physics.Source: §4.3 · p. 7, Table 4 · p. 7
  • Video-generator latents improve NAVSIM PDMS by 5.0 over BEV features and 2.6 over VLM hidden states under the same diffusion planner.The representation comparison uses video latents, BEV features, and VLM hidden states as planner conditions.Source: §4.3 · p. 8, Table 5 · p. 8
  • Conditioning Action DiT on latents from early video-denoising steps yields stronger planning than using later-step latents.The authors note that raw pixel-format videos can contain redundant, non-essential information that hinders decision-making.Source: §4.3 · p. 8, Table 6 · p. 8
  • The complete multi-stage training strategy yields the best balance, with the lowest FID and FVD; omitting stage one raises FVD and weakens temporal coherence.Omitting stage two preserves temporal stability but moderately increases FVD, reflecting slightly degraded spatial detail.Source: §4.3 · p. 8, Table 7 · p. 8

5. Conclusion

DriveLaW unifies video generation and motion planning through shared video latents, pairing temporally consistent synthesis with a diffusion planner trained through three progressive stages.

5. Conclusion

  • Experiments on nuScenes and NAVSIM report state-of-the-art performance for video generation and motion planning.The paper presents the results as validating a shared latent representation for driving-world generation and decision-making.Source: §5 · p. 8

A. More Implementation Details

The implementation combines compressed spatiotemporal video generation with latent-conditioned trajectory planning and motion-guided prompts; qualitative comparisons find the video-generator latents more structured than BEV or VLM features.

A. More Implementation Details

  • DriveLaW-Video uses a diffusion architecture designed for high-compression spatiotemporal encoding and efficient chained generation under practical hardware constraints.The implementation details cover its generation backbone, planner, and motion-conditioned prompting.Source: §A.1 · p. 13, A. More Implementation Details · p. 13

A.1. DriveLaW-Video: Video Generation Backbone

  • The Video-VAE compresses by 1:192, using causal encoding, hybrid pixel-space decoding, reconstruction GAN training, learned decoder noise, balanced KL regularization, and Video-DWT loss.The stated aim is to preserve fidelity and structural detail under aggressive compression.Source: §A.1 · p. 13
  • The video diffusion backbone adapts PixArt-α with 28 self–cross attention blocks, hidden size 2048, and RMSNorm instead of LayerNorm.Tokens are serialized directly from VAE latents at 1 × 1 × 1 granularity, avoiding patchification.Source: §A.1 · p. 13
  • Normalized fractional RoPE with exponential frequency spacing supports varying resolutions and durations while preserving geometric consistency.Direct serialization of VAE-latent tokens at 1 × 1 × 1 granularity eliminates redundant patchification.Source: §A.1 · p. 13

A.2. DriveLaW-Act: Trajectory Planning Module

  • DriveLaW-Act is a 133M-parameter diffusion planner conditioned on cached first-step video latents, ego kinematics, navigation commands, and trajectory noise.It flow-matches continuous (x, y, θ) positions at 2 Hz over a 4 s horizon, runs without video decoding at inference, and preserves gradient isolation during training.Source: §A.2 · p. 13

A.3. Motion-Conditioned Prompting Mechanism

  • The motion-conditioned prompt converts ego speed, steering, and displacement into semantic bins, then combines them with numerical grounding in a fixed template.Frozen T5-XXL embeddings enter every Video-DiT layer via cross-attention to align generated video with ego dynamics.Source: §A.3 · pp. 13–14

B.1. Qualitative analysis of latent representations.

  • PCA visualizations show VGM latents as sharper, less noisy, and more semantically coherent with stronger spatial structure than BEV or VLM features.The comparison uses three principal components mapped to RGB; BEV features are taken before the BEV query transformation for fair comparison.Source: §B.1 · p. 14

B.2. Quantitative Evaluation of Video Generation

The video-generation evaluation covers NuScenes and OpenDV, plus varying NuPlan prediction horizons; reported comparisons favor DriveLaW-Video over Epona in several settings, but not at the longest horizon.

B.2. Quantitative Evaluation of Video Generation

  • Table 8 reports FID and FVD on NuScenes and OpenDV, and FVD across NuPlan prediction horizons.The supplied passages do not provide the table's metric values.Source: Table 8 · p. 14, §B.2 · p. 14
  • DriveLaW-Video outperforms Epona on zero-shot OpenDV generalization and NuPlan FVD through 80 frames, while Epona performs better at 100 frames.Epona is substantially slower, with the speed gap increasing at longer horizons; the paper describes DriveLaW as a more practical quality-efficiency trade-off for realistic horizons.Source: §B.2 · p. 14

B.3. Inference Speed Analysis

DriveLaW generates video faster than Epona in reported 4090 comparisons, including at a higher output resolution; the paper also reports inference speed on an H20 GPU.

B.3. Inference Speed Analysis

  • At 1280 × 704, DriveLaW generates 0.39 s per frame versus Epona’s 0.88 s, and it reports 0.12 s at 768 × 512 and 0.18 s at 1024 × 512.The comparison uses one NVIDIA 4090 GPU and 30 DiT sampling steps; the authors attribute runtime efficiency across resolutions to higher compression and hybrid decoding.Source: Table 9 · p. 14, §B.3 · p. 14

B.4. Runtime Performance on H20 GPU

  • Inference speed is also reported for an NVIDIA H20 GPU.The supplied passage gives no H20 timing values.Source: §B.4 · p. 14

B.5. Ablation on Noise Reinjection Usage

The ablation examines noise reinjection, while the accompanying qualitative results show DriveLaW’s video and latent representations across diverse driving conditions.

B.5. Ablation on Noise Reinjection Usage

  • DriveLaW’s VGM features appear sharper, less noisy, and more spatially structured than BEV and VLM features in the PCA visualization.The figure maps each representation’s top three principal components to RGB channels.Source: Figure 4 · p. 15
  • Removing noise reinjection degrades temporal coherence and slightly reduces spatial fidelity.The comparison is reported in Table 11.Source: §B.5 · p. 15, Table 11 · p. 15
  • Selective perturbation of high-frequency regions before denoising prompts detail regeneration, improving sharpness and temporal stability while reducing artifacts.Source: §B.5 · p. 15

C. More Qualitative Results

  • The additional qualitative-results section presents examples intended to illustrate DriveLaW’s video-generation and planning capabilities.Source: C. More Qualitative Results · p. 15

C.1. Video Generation on nuScenes

  • On nuScenes validation examples, DriveLaW maintains temporal coherence, fine spatial detail, and robustness across visual conditions, including night and complex urban scenes.Figure 5 includes conventional urban driving, dense interactions and occlusions, and night driving.Source: §C.1 · p. 15

C.2. Planning Results Visualization

The section presents representative Navtest planning cases and supplementary videos for visual assessment of generated scenes and planning outputs.

C.2. Planning Results Visualization

  • Representative Navtest cases illustrate predicted future trajectories with safety and smoothness.Figure 6 presents qualitative planning results.Source: §C.2 · p. 15

C.3. Supplementary Video Demonstrations

  • The supplementary material provides six demo videos: four normal-driving and two rainy-weather scenarios.The examples are offered for visual assessment of temporal consistency, spatial detail, and planning outputs.Source: §C.3 · p. 16

D. Limitations and Future Work

The authors identify reconstruction artifacts and inference latency as limitations, while describing scalability as an opportunity for future work.

D.1. Motion Artifacts in High-Compression VAE.

  • The 32 × 32 × 8 Video-VAE compression introduces reconstruction artifacts that propagate into generated video during high-motion driving.Noise reinjection mitigates but does not fundamentally resolve distortions during rapid ego-motion or dynamic-agent interactions.Source: §D.1 · p. 16

D.2. Inference Latency.

  • Inference remains slower than end-to-end planners that bypass video generation because diffusion-based video generation is computationally demanding.The authors note optimizations including high-compression VAE, resolution scaling, and hybrid decoding.Source: §D.2 · p. 16

D.3. Scalability and Future Outlook.

  • The authors expect DriveLaW’s performance to advance with foundational video models without architectural redesign, and propose pretrained generators as a route to planners needing minimal domain-specific training.They envision onboard viability with inference acceleration methods such as distillation, quantization, and dedicated hardware.Source: §D.3 · p. 16

Related papers

Loading 2512.23421v3…