PaperlayerView exact revision on arXiv

Source-linked AI summary

DriveVA: Video Action Models are Zero-Shot Drivers

Mengmeng Liu, Diankun Zhang, Jiuming Liu, Jianfeng Cui, Hongwei Xie, Guang Chen, Hangjun Ye, Michael Ying Yang, Francesco Nex, Hao Cheng

arXiv:2604.04198v2cs.CVcs.RO

TL;DR

Autonomous driving models must generalize to unseen scenarios and sensor configurations, while existing loosely coupled approaches can misalign visual imagination with planned actions. DriveVA jointly generates future video and trajectories in a shared latent process, achieving state-of-the-art NAVSIM planning and strong zero-shot transfer across nuScenes and Bench2Drive.

  • Problem

    Generalization across unseen traffic patterns, road layouts, and sensor configurations remains a central challenge, while existing methods often loosely couple visual prediction and planning.

  • Method

    DriveVA is a unified video-action world model that jointly generates future visual latents and trajectory tokens through a shared conditional generative process, with progressive video continuation for rollout consistency.

  • Results

    DriveVA achieves 90.9 PDMS on NAVSIM and reduces average L2 error and collision rate by 78.9% and 83.3% on nuScenes and 52.5% and 52.4% on Bench2Drive, respectively.

  • Takeaways & Limitations

    DriveVA shows strong zero-shot and cross-domain transfer without target-domain fine-tuning while maintaining video–trajectory consistency during real-world rollout.

  • Takeaways & Limitations

    Failure cases show that causal or intention-level mistakes in future imagination can produce consistent but incorrect trajectories.

Abstract

from arXiv · show

Generalization is a central challenge in autonomous driving, as real-world deployment requires robust performance under unseen scenarios, sensor domains, and environmental conditions. Recent world-model-based planning methods have shown strong capabilities in scene understanding and multi-modal future prediction, yet their generalization across datasets and sensor configurations remains limited. In addition, their loosely coupled planning paradigm often leads to poor video-trajectory consistency during visual imagination. To overcome these limitations, we propose DriveVA, a novel autonomous driving world model that jointly decodes future visual forecasts and action sequences in a shared latent generative process. DriveVA inherits rich priors on motion dynamics and physical plausibility from well-pretrained large-scale video generation models to capture continuous spatiotemporal evolution and causal interaction patterns. To this end, DriveVA employs a DiT-based decoder to jointly predict future action sequences (trajectories) and videos, enabling tighter alignment between planning and scene evolution. We also introduce a video continuation strategy to strengthen long-duration rollout consistency. DriveVA achieves an impressive PDM-based planning performance of 90.9 PDM score on the NAVSIM benchmark. Extensive experiments also demonstrate the zero-shot capability and cross-domain generalization of DriveVA, which reduces average L2 error and collision rate by 78.9% and 83.3% on nuScenes and 52.5% and 52.4% on the Bench2Drive built on CARLA v2 compared with the state-of-the-art world-model-based planner.

1 Introduction

DriveVA addresses limited cross-domain generalization and weak consistency between imagined videos and planned trajectories by jointly generating both in a shared latent process. It reports strong NAVSIM planning and zero-shot transfer results.

  • Generalization remains challenging because autonomous-driving systems must handle unseen traffic patterns, road layouts, sensor configurations, and long-tail interactions.
  • Existing world-model planners often generalize poorly across datasets and loosely couple visual prediction with trajectory generation.
  • DriveVA jointly decodes future video latents and action tokens with a single DiT in a shared latent space, aligning planned actions with imagined scene evolution.
  • Video supervision increases NAVSIMv1 PDMS from 71.4 to 90.9 (+19.5) over action-only optimization by providing dense temporal grounding of scene dynamics.
  • DriveVA achieves 90.9 PDMS on NAVSIM and reduces average L2 error and collision rate by 78.9% and 83.3% on nuScenes, respectively.On Bench2Drive, it reduces the same metrics by 52.5% and 52.4%.

2 Related Work

Related work spans latent-dynamics and visually predictive driving world models, while DriveVA combines pretrained video priors with unified video–trajectory generation. This design targets transfer beyond benchmark-specific distributions and tighter planning alignment.

  • Zero-shot driving is relevant because it tests decisions in unseen corner cases and indicates transfer across platforms and scenarios.
  • Driving world models include latent-dynamics methods for compact future representations and visually predictive methods that forecast future observations for planning.
  • Many existing methods treat visual prediction as auxiliary or loosely coupled, maintaining separate video and action generation processes.
  • Video generative driving models can exploit pretrained representations encoding spatiotemporal and physical priors rather than learning dynamics entirely from scratch.
  • DriveVA’s pipeline uses history observations, ego state, and language instructions to jointly predict future video latents and action tokens with a DiT.

3 Preliminary

The preliminary formulation describes flow matching as continuous transport from noise to data, with training based on linear interpolation and target-velocity regression. Conditional video generation performs this process in a pretrained video-autoencoder latent space.

  • Flow matching learns a time-dependent velocity field that transforms samples from a simple source distribution into the target data distribution.
  • Training uses the linear interpolation x(s) = (1 − s)ϵ + sxdata, whose derivative is xdata − ϵ.
  • The velocity network is trained by minimizing squared error between its prediction and the interpolation target velocity.
  • Conditional video flow matching operates in the latent space of a pretrained video autoencoder and decodes generated latent sequences into pixels.
  • The latent trajectory starts from Gaussian noise at s = 0 and is integrated to s = 1 before decoding with D.

4 Method

DriveVA jointly predicts future video latents and action tokens from visual history, ego state, and text commands, using a shared generative process. Fixed history conditioning, rolling-horizon continuation, and flow matching support consistent short-window generation that can be chained into longer rollouts.

  • Problem Formulation: The predicted action chunk contains K sequential 3-D actions encoding ego position and yaw, while the future video clip depicts visual evolution after executing those actions.Future frames are predicted through latent representations rather than raw pixels.
  • Video Continuation: A rolling-horizon procedure updates the history buffer after each action chunk, reducing long-horizon prediction to progressively chained short video-continuation problems.The same short-continuation setup is used during training and inference.
  • Video Representation: The model uses a causal video VAE to temporally downsample clips into latent sequences, with the first latent depending only on the first frame under the original causal design.At inference, historical observations are encoded as conditioning latents and future latents are generated from them.
  • Joint Video–Action Modeling: DriveVA jointly predicts future video latents and action tokens conditioned on visual history, the current ego state, and text or command tokens.The model targets future video representations and sequential 3-D actions within one generative formulation.
  • Generative Decoder: A DiT decoder predicts a conditional velocity field for noisy future video-and-action targets while keeping the history condition fixed during training and inference.The target block concatenates future video latents with action tokens, and flow matching trains the velocity prediction.

5 Experiments

DriveVA is evaluated on NAVSIM and strict zero-shot transfers to nuScenes and Bench2Drive, with additional analyses of planning quality, cross-domain robustness, and video–trajectory consistency. Results show strong planning performance and alignment between generated videos and predicted trajectories.

  • Quantitative Comparison Results: 90.9 PDMS is achieved on NAVSIM, surpassing traditional end-to-end and world-model-based methods with only front-view camera input.The authors attribute the gains primarily to jointly modeling future video imagination and ego-trajectory prediction within a shared generative process.
  • Cross-Domain Generalization: 78.9% lower average L2 error and 83.3% lower collision rate are reported on zero-shot nuScenes versus PWM.DriveVA is trained on NAVSIM and evaluated directly on nuScenes without target-domain fine-tuning.
  • Video–Trajectory Consistency Analysis: 0.15 average L2 reconstruction error across NAVSIM and zero-shot nuScenes indicates that generated videos imply motion aligned with predicted trajectories.DPVO reconstructs camera trajectories from generated videos after 2D similarity alignment over the future 4s horizon.
  • Ablation Studies: Video supervision raises NAVSIMv1 PDMS from 71.4 to 90.9, while removing video continuation reduces PDMS to 84.6.The ablation results identify video supervision as the main driver of the gain and show that rollout coupling remains important.
  • Efficiency Analysis: Two flow-matching sampling steps reach 90.9 PDMS, while three steps provide no additional gain.The experiment indicates efficient planning with very few sampling steps.

6 Conclusion

DriveVA is a unified video–action world model that jointly generates future video latents and trajectory tokens, using shared generation and progressive continuation to improve rollout consistency and zero-shot transfer.

  • 6 Conclusion: DriveVA jointly generates future video latents and trajectory tokens within a shared conditional generative process.The formulation unifies video continuation and action grounding rather than using a cascaded pipeline.
  • 6 Conclusion: The model uses a large pretrained video-generation backbone and progressive video continuation to support consistent future visual rollouts.The target block contains future video latent tokens and action tokens, while video continuation progressively extends the rollout.
  • 6 Conclusion: DriveVA achieves state-of-the-art PDM-based performance on NAVSIM and strong zero-shot transfer to nuScenes and Bench2Drive without target-domain fine-tuning.The conclusion identifies zero-shot transfer across both real-world and simulation benchmarks as a central outcome.

C Additional Zero-Shot Results

Additional zero-shot examples show that DriveVA keeps generated future videos and planned trajectories better aligned than PWM in unseen transfer scenarios.

  • C Additional Zero-Shot Results: DriveVA maintains better alignment between generated future videos and planned trajectories than PWM under zero-shot transfer.The comparison concerns additional qualitative examples on unseen datasets and links transfer performance to video–action coupling.
  • C Additional Zero-Shot Results: PWM can generate future videos and trajectories implying different maneuvers, whereas DriveVA preserves closer correspondence between visual forecasting and action generation.This qualitative evidence suggests the transfer gain is not limited to endpoint accuracy.

D DPVO-based External Verification of Video–Trajectory Consistency

DPVO-based reconstruction externally verifies that DriveVA’s generated videos imply trajectories closely matching its predicted ego motion across NAVSIM, zero-shot nuScenes, and representative driving scenarios.

  • D DPVO-based External Verification of Video–Trajectory Consistency: DPVO reconstructs ego trajectories from ground-truth and generated future videos, which are aligned to reference trajectories before average L2 error is computed.A 2D similarity transform addresses monocular visual-odometry scale ambiguity, and error is measured over the future 4s horizon.
  • D DPVO-based External Verification of Video–Trajectory Consistency: Qualitative comparisons show stronger video–trajectory alignment for DriveVA than PWM in representative unseen nuScenes cases.DriveVA preserves aligned turning behavior across imagined frames at 1s, 2s, and 3s, unlike PWM’s mismatched visual and planned motion.
  • D DPVO-based External Verification of Video–Trajectory Consistency: 0.09 and 0.16 are the NAVSIM average L2 errors for ground-truth-video and predicted-video reconstruction, respectively.The corresponding zero-shot nuScenes errors are 0.07 and 0.14.
  • D DPVO-based External Verification of Video–Trajectory Consistency: 0.07 and 0.14 are the zero-shot nuScenes average L2 errors for ground-truth-video and predicted-video reconstruction, respectively.These values remain small across both reconstruction branches.
  • D DPVO-based External Verification of Video–Trajectory Consistency: Across lane changes, right turns, straight driving, and turning scenarios, DPVO reconstructions from generated videos closely overlap DriveVA’s predicted trajectories.The independent visual-odometry system recovers essentially the same ego motion from generated frames.

E.1 Preliminary closed-loop evaluation on Bench2Drive Dev10

A preliminary Bench2Drive Dev10 closed-loop study finds that DriveVA improves success rate over both baselines and achieves a competitive driving score, while remaining only an auxiliary transfer check.

  • E.1 Preliminary closed-loop evaluation on Bench2Drive Dev10: DriveVA improves success rate over both baselines and obtains a competitive driving score on the Bench2Drive Dev10 split.The study evaluates closed-loop transfer from NAVSIM-trained video–action modeling to simulation.
  • E.1 Preliminary closed-loop evaluation on Bench2Drive Dev10: Dev10 contains limited routes, so its results are treated as auxiliary closed-loop evidence rather than a substitute for full Bench2Drive evaluation.The table caption likewise identifies Dev10 as a small diagnostic split.

E.2 Additional design ablations

The ablations show that jointly predicting future video and actions improves planning, while restricting their interaction slightly reduces performance. Qualitative failures indicate that video–trajectory consistency can persist even when the predicted future mode is incorrect.

  • 90.1 PDMS results when the causal mask restricts interaction between future video and action tokens, down from 90.9 for the default model.The comparison suggests mutual interaction during denoising is helpful, although condition tokens remain fixed and noise-free.
  • The action-only variant performs much worse than the default video-plus-action target, confirming that future-video prediction is not merely auxiliary.
  • Consistent video–trajectory predictions can still be wrong when the generated future mode differs from ground truth.In both examples, the predicted trajectory follows the generated video rather than the ground-truth future.

F Failure Cases and Limitations

DriveVA’s visual and trajectory branches remain aligned in representative failure cases, but the shared rollout can consistently realize an incorrect future mode. The authors identify causal scene understanding and multimodal future reasoning as important directions for improvement.

  • DriveVA predicts conservative stopping when ground truth bypasses a cyclist or proceeds through an intersection.The second case produces a nearly stationary trajectory.
  • The video and trajectory branches remain aligned even when DriveVA selects the wrong future mode.
  • Causal scene understanding and multimodal future reasoning are identified as important directions for improving these errors.

G More Qualitative Results

Additional qualitative examples show that predicted trajectories generally evolve consistently with generated future frames, including zero-shot scenarios on unseen nuScenes and CARLA domains. These visualizations also illustrate the unified video–action rollout’s alignment between scene evolution and ego motion.

  • The qualitative figures use generated future videos with overlaid ground-truth and predicted trajectories to make their temporal alignment directly comparable.Across the examples, green denotes ground truth and red denotes prediction.
  • Predicted trajectories evolve consistently with generated future frames in additional qualitative examples.The examples support alignment between visual imagination and ego motion.
  • Zero-shot examples on unseen nuScenes and CARLA scenarios retain video–trajectory alignment under dataset and real-to-simulation domain shifts.The scenarios include turning, bypassing, and straight-driving behaviors.
  • Right-turn, parked-vehicle avoidance, and stopped-bus bypass visualizations show trajectories evolving with generated video over time.Green denotes the ground-truth trajectory and red denotes the prediction.
Loading 2604.04198v2…