Source-linked AI summary
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability
Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, Hongyang Li
TL;DR
Existing driving world models are limited in generalization, prediction fidelity, and action controllability. Vista addresses these gaps with high-fidelity prediction mechanisms, dynamic priors, and unified multimodal control, demonstrating realistic continuous futures and controllability across domains.
Problem
Existing driving world models remain constrained in generalization, critical-detail fidelity, control modalities, and controllability across unseen datasets.
Method
Vista combines explicit dynamics and structure-preservation losses, historical dynamic priors from consecutive frames, and a unified interface for versatile action formats.
Results
Vista predicts realistic high-resolution futures, supports consistent multimodal action control in nuScenes and Waymo, and generalizes control to unseen scenarios.
Takeaways & Limitations
Vista can serve as a generalizable driving world model and can be formulated as a reward function for evaluating actions without ground-truth actions.
Takeaways & Limitations
Vista remains limited in computation efficiency, quality maintenance, and training scale.
Abstract
from arXiv · showhide
World models can foresee the outcomes of different actions, which is of paramount importance for autonomous driving. Nevertheless, existing driving world models still have limitations in generalization to unseen environments, prediction fidelity of critical details, and action controllability for flexible application. In this paper, we present Vista, a generalizable driving world model with high fidelity and versatile controllability. Based on a systematic diagnosis of existing methods, we introduce several key ingredients to address these limitations. To accurately predict real-world dynamics at high resolution, we propose two novel losses to promote the learning of moving instances and structural information. We also devise an effective latent replacement approach to inject historical frames as priors for coherent long-horizon rollouts. For action controllability, we incorporate a versatile set of controls from high-level intentions (command, goal point) to low-level maneuvers (trajectory, angle, and speed) through an efficient learning strategy. After large-scale training, the capabilities of Vista can seamlessly generalize to different scenarios. Extensive experiments on multiple datasets show that Vista outperforms the most advanced general-purpose video generator in over 70% of comparisons and surpasses the best-performing driving world model by 55% in FID and 27% in FVD. Moreover, for the first time, we utilize the capacity of Vista itself to establish a generalizable reward for real-world action evaluation without accessing the ground truth actions.
1 Introduction
Existing driving world models struggle to generalize across environments, preserve critical details, and support diverse action formats. Vista addresses these gaps with large-scale training, high-resolution prediction, and unified multi-modal control.
- Limitations of existing models: Existing driving world models are constrained by data scale and geographical coverage, often losing critical details through low frame rates and resolutions.They also commonly support only one control modality, limiting compatibility with varied planning outputs and leaving action-control generalization understudied.
- Vista: Vista is trained on a large worldwide driving-video corpus to improve cross-domain generalization and predicts futures at 10 Hz and 576×1024 pixels.The model uses dynamic priors, explicit dynamics and structure losses, and multi-modal action conditioning through a unified interface.
- Vista: Vista integrates high-level commands and goal points with low-level trajectories, steering angles, and speeds through an efficient learning strategy.These controls are injected through a unified conditioning interface.
- Evidence and applications: Vista outperforms the most competitive general-purpose video generator and establishes a new state of the art on nuScenes across multiple datasets.The paper also reports that Vista can serve as a reward function for assessing actions.
2 Preliminary
Stable Video Diffusion provides a high-aesthetic-quality image-to-video foundation, but it lacks the temporal consistency, realistic driving dynamics, and action control required of a driving world model.
- Stable Video Diffusion: Stable Video Diffusion generates 25-frame videos by progressively denoising noisy latent sequences conditioned on an image latent.Its condition image is concatenated channel-wise to the noisy inputs as a content reference.
- Limitations: Stable Video Diffusion’s first predicted frame may differ from its condition image, making autoregressive rollout impractical because of content inconsistency.It also struggles with intricate driving dynamics and can produce implausible motions.
- Requirements for driving world models: Stable Video Diffusion cannot be controlled by any action format, whereas Vista targets realistic, continuously extendable, and multi-modally controllable futures.Vista’s capabilities include anticipating realistic continuous futures and accepting diverse actions.
3 Learning a Generalizable Driving World Model
Vista uses a two-phase pipeline to learn high-fidelity future prediction and versatile action control. It combines latent replacement and dynamic and structural losses for coherent, detailed rollouts, then adds heterogeneous action conditioning and an uncertainty-based reward.
- Pipeline: Vista uses two training phases: first learning predictive fidelity and coherent futures, then freezing pretrained weights to learn multi-modal action controls.The pipeline is trained on a large public driving dataset to support generalization to unseen scenarios.
- Dynamic Prior Injection: Three consecutive condition frames provide position, velocity, and acceleration priors for coherent future prediction.Vista replaces corresponding noisy latents with clean condition latents and uses the last three predicted frames as priors for the next rollout step.
- Dynamics Enhancement Loss: The dynamics enhancement loss adaptively emphasizes regions with motion discrepancies, including moving vehicles and sidewalks, rather than weighting all outputs uniformly.The loss uses adjacent-frame discrepancies to identify dynamic regions and penalizes the latter frame in each pair.
- Structure Preservation Loss: The structure preservation loss matches high-frequency features between predictions and ground truth to retain edges, textures, and other structural details.High-frequency components are extracted with Fourier transforms and an ideal high-pass filter.
- Unified Conditioning of Versatile Actions: Vista supports angle, speed, trajectory, command, and goal-point controls through unified Fourier embeddings and cross-attention conditioning.The action modes span fine-grained maneuvers and high-level intentions, while zero-initialized projections enable gradual learning from the pretrained state.
- Generalizable Reward Function: Vista estimates a generalizable action-evaluation reward from its own prediction uncertainty, avoiding external models and ground-truth actions.The reward is based on conditional variance estimated from repeated denoising with the same condition frame and action.
4 Experiments
Vista is evaluated for generalization, fidelity, action controllability, and its dynamic-prior and auxiliary-loss designs. Results show improved similarity to real videos, control consistency, long-horizon coherence, and realistic dynamics across datasets and action modalities.
- Generalization and fidelity: Human evaluation compares Vista with general-purpose video generators on visual quality and motion rationality across datasets.Participants use a Two-Alternative Forced Choice protocol, selecting the better video on each aspect.
- Generalization and fidelity: Vista outperforms existing works in both human-evaluation metrics.The reported value is the percentage of times one model is preferred over the other.
- Action controllability: Applying action controls produces predictions more similar to real data, and Vista supports multimodal conditions across diverse scenarios.Qualitative results include action controllability in unseen scenarios beyond the training domain.
- Action controllability: Vista’s action modalities yield more consistent motions with ground truth on nuScenes and Waymo.Control consistency is measured using trajectory differences over 2 seconds on subsets of 537 samples per dataset.
- Ablation study: More dynamic priors improve consistency between predicted and ground-truth futures in long-horizon rollouts.The evaluation uses IDM-inferred trajectories, and diminishing trajectory differences indicate improved consistency as priors increase.
- Ablation study: The dynamics enhancement loss promotes realistic motion, while the structure preservation loss reinforces structural details.Qualitative comparisons show normal front-car motion, geometrically consistent tree shifts, and clearer moving-object outlines.
5 Conclusion
Vista is presented as a generalizable driving world model with realistic, continuous high-resolution futures, versatile action controllability, and action evaluation through a learned reward. The authors identify computation efficiency, quality maintenance, and training scale as limitations for future work.
- Conclusion: Vista predicts realistic and continuous futures at high spatiotemporal resolution and supports action control that generalizes to unseen scenarios.The conclusion also states that Vista can be formulated as a reward function for action evaluation.
- Limitations and future work: Vista remains limited in computation efficiency, quality maintenance, and training scale.Future work targets scalable architectures.
A Discussions
Vista is positioned as a generalizable, high-fidelity, multi-modal driving world model, while its discussion covers applications, comparisons, scope choices, and limitations. The paper also frames world models as tools for future prediction, planning, control, and action evaluation.
- Applications: Vista could serve as a forward dynamics model for short-term simulation, visual-navigation planning, an implicit driving policy, or a component of model-based reinforcement learning.
- Comparison with GenAD: Vista differs from GenAD through zero-shot multi-modal control generalization, higher 10 Hz and 576×1024 output, and stronger FID and FVD with lower model complexity.
- Limitations: Vista remains computationally expensive, may degrade during long-horizon rollouts or drastic view shifts, and can fail on ambiguous action intentions such as commands and goal points.
- Scope: The paper focuses on front-view generation because diverse data sources are easier to combine than heterogeneous multi-camera videos in surround-view settings.
- Limitations: Direct deployment in autonomous vehicles could pose safety risks because diffusion introduces stochastic outcomes and non-negligible latency.
- Motivation: World models predict plausible future states under potential actions and can support representation learning, model-based reinforcement learning, and model-predictive control.
- Video generation: Vista is designed for realistic predictive dynamics and versatile vehicle-action control, unlike camera-motion-controlled text-to-video methods without prediction ability.
C Implementation Details
Vista uses the Stable Video Diffusion architecture and Fourier embeddings for action conditioning.
- Vista adopts Stable Video Diffusion as its architecture and encodes each action sequence value into 128-channel Fourier embeddings.
C.2 Dataset
Vista trains on filtered worldwide driving videos and incorporates balanced nuScenes data during action-control learning.
- Filtering removes 15 hours of irrelevant OpenDV-YouTube content, leaving approximately 1735 hours of unlabeled driving videos.
- The action-control phase incorporates nuScenes training data while balancing samples by command categories because nuScenes is heavily biased.
- Actions are defined as 25-frame sequences, with turn commands determined by final displacement exceeding 2 meters orthogonally from the initial heading.
C.3 Training
Vista uses large-scale two-phase training and specialized sampling procedures to learn prediction fidelity, long-horizon coherence, and action controllability.
- Predictive-model training: Vista trains all UNet parameters at 576×1024 resolution on 128 A100 GPUs for 20K iterations during predictive-model training.
- Action-control training: Action-control learning first trains LoRA and projection layers, then fine-tunes unfrozen weights at 576×1024 resolution.
- Sampling: Triangular classifier-free guidance assigns moderate guidance to temporally reused condition frames, mitigating saturation drift during long-horizon rollouts.
- Sampling: Three-frame-overlap decoding averages overlapping pixels to improve perceptual continuity in generated clips.
C.5 Human Evaluation
The experiments examine reward estimation, action-control learning, guidance, LoRA adaptation, and action-conditioned prediction. Results indicate reliable command rewards, improved action learning with the proposed constraint, and effective control across categories.
- Ground-truth commands generally receive higher rewards than random commands on Waymo, supporting the reward function for command selection.
- The proposed action-independence constraint expedites action learning, evaluated with trajectory control as a representative action.
- Triangular classifier-free guidance achieves the best trade-off between visual quality and saturation preservation.
- LoRA adaptation is essential for action-control learning, while training only newly added projection layers produces visual corruptions.
- All action-control types remain effective across the four command-category subsets according to complete FVD results.For the stop subset, goal-point conditioning often behaves like the action-free mode because final displacements are within 2 meters.
D.7 Human Evaluation with GenAD
Vista is evaluated against GenAD on unseen driving videos and through additional generalization, long-horizon, action-control, and counterfactual demonstrations. Human preferences and qualitative results favor Vista, including under domain transfer and abnormal interventions.
- Human Evaluation: 94.4% and 94.8% of volunteers preferred Vista over GenAD for visual quality and motion rationality, respectively.The evaluation used 25 diverse unseen OpenDV-YouTube-val samples and 20 volunteers.
- Human Evaluation: Vista shows higher resolution and fidelity than GenAD in qualitative comparisons after outputs are aligned to GenAD’s shorter video duration.
- Generalization: Vista produces high-fidelity predictions across a very diverse range of scenarios in the wild.
- Long-Horizon Prediction: Vista continuously predicts long-term futures with consistent content and motion.
- Action Controllability: Vista’s versatile action controllability transfers to OpenDV-YouTube-val and Waymo in a zero-shot manner.
- Counterfactual Reasoning: Vista predicts counterfactual consequences caused by abnormal actions, including traffic-rule violations.
E.5 Human Evaluation Cases
The appendix presents diverse evaluation cases spanning multiple datasets and driving environments. These cases illustrate long-horizon simulation, varied action control, generalization to unseen viewpoints, and counterfactual reasoning.
- Generalization Cases: Vista generalizes across countrysides, tunnels, and unseen camera poses such as a double-decker-bus perspective.The resulting futures contain high-resolution vehicle and pedestrian behaviors.
- Long-Horizon Prediction: Additional results show autoregressive 15-second driving simulations at 10 Hz with marginal quality decline.
- Action Controllability: Across OpenDV-YouTube-val and Waymo, different action conditions consistently control ego-vehicle behavior across multiple scenes.
- Counterfactual Reasoning: Abnormal interventions produce counterfactual outcomes, including boundary crossing and collision avoidance at a crossroads.
- Human Evaluation Cases: The human-evaluation cases comprise 60 scenes from OpenDV-YouTube-val, nuScenes, Waymo, and CODA.The datasets jointly represent diverse real-world environments for comprehensive human evaluation.