PaperlayerView exact revision on arXiv

Source-linked AI summary

CVD-STORM: Cross-View Video Diffusion with Spatial-Temporal Reconstruction Model for Autonomous Driving

Tianrui Zhang, Yichen Liu, Zilin Guo, Yuxin Guo, Jingcheng Ni, Chenjing Ding, Dan Xu, Lewei Lu, Zehuan Wu

arXiv:2510.07944v2cs.CV

TL;DR

Driving world models need controllable, long-term multi-view video with meaningful structural information, but current approaches have gaps in explicit 3D representation. CVD-STORM trains a reconstruction-enhanced VAE and uses it in cross-view video diffusion; it is reported to surpass prior methods particularly in image quality and temporal coherence.

  • Problem

    Driving world models need long-term, controllable, multi-view video, yet existing approaches lack explicit 3D information or rely on depth estimates that do not represent real-world geometry accurately.

    Source: §1 · p. 2

  • Method

    CVD-STORM fine-tunes a VAE with an auxiliary 4D reconstruction task, then uses its latents to train a conditional cross-view video diffusion model.

    Source: §1 · p. 2

  • Results

    CVD-STORM surpasses state-of-the-art methods, particularly in image quality and temporal coherence.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The Gaussian Splatting Decoder directly estimates absolute depth through neural rendering, providing richer 3D structural information than previous approaches.

    Source: §5 · p. 10

    Generated-result depth metrics use pseudo-ground-truth maps from Depth Anything V2 and cannot assess absolute depth accuracy, which remains an open challenge.

    Source: §6.1.2 · p. 14

Abstract

from arXiv · show

Generative models have been widely applied to world modeling for environment simulation and future state prediction. With advancements in autonomous driving, there is a growing demand not only for high-fidelity video generation under various controls, but also for producing diverse and meaningful information such as depth estimation. To address this, we propose CVD-STORM, a cross-view video diffusion model utilizing a spatial-temporal reconstruction Variational Autoencoder (VAE) that generates long-term, multi-view videos with 4D reconstruction capabilities under various control inputs. Our approach first fine-tunes the VAE with an auxiliary 4D reconstruction task, enhancing its ability to encode 3D structures and temporal dynamics. Subsequently, we integrate this VAE into the video diffusion process to significantly improve generation quality. Experimental results demonstrate that our model achieves substantial improvements in both FID and FVD metrics. Additionally, the jointly-trained Gaussian Splatting Decoder effectively reconstructs dynamic scenes, providing valuable geometric information for comprehensive scene understanding. Our project page is https://sensetime-fvg.github.io/CVD-STORM.

1 INTRODUCTION

Driving world models need long-term, controllable, multi-view videos with structural information. CVD-STORM combines video generation with 4D scene reconstruction through a reconstruction-trained VAE.

1 INTRODUCTION

  • Existing driving video approaches still lack explicit 3D information, while some depth estimates are relative and other generated LiDAR is poorly aligned with images.These limitations constrain their use as world models.Source: §1 · p. 2
  • STORM-VAE adds a Gaussian Splatting decoder for 4D reconstruction and integrates spatial and temporal information into latents beyond RGB-only encoding.The framework uses those latents to train a cross-view video diffusion model.Source: §1 · p. 2
  • CVD-STORM generates long sequential multi-view driving videos and reconstructs 4D scenes through a two-stage training strategy.Scene reconstruction is learned before training the conditional world model.Source: §1 · p. 2
  • CVD-STORM is reported to improve generative quality and address 4D absolute depth estimation.At inference, it supports long-term six-view video conditioned on text, bounding boxes, and high-definition maps.Source: §1 · p. 2

2 RELATED WORK

Prior driving video work advanced duration, multiple views, and control, while 3D reconstruction and representation learning developed along related tracks. The paper frames their integration with video generation as an underexplored opportunity.

2.1 VIDEO DIFFUSION AND DRIVING WORLD MODEL

  • Driving video models progressed toward longer, multi-view, controllable generation, but still inadequately capture structural information such as depth.Earlier systems often struggled with long sequences, while later work adopted DiT architectures and multi-stage training.Source: §2.1 · p. 3

2.2 4D RECONSTRUCTION IN DRIVING SCENARIOS

  • Prior depth-generation approaches rely on pseudo supervision or provide only relative depth, while reconstruction methods include Gaussian-based scene modeling.The integration of generation and reconstruction remains underexplored.Source: §2.2 · pp. 3–4

2.3 REPRESENTATION LEARNING IN DIFFUSION

  • Representation-learning work adds auxiliary tasks or aligns latent features, and this paper extends that direction with reconstruction while tuning the VAE for 4D reconstruction.The stated aim is to enhance generation performance while adding reconstruction capability.Source: §2.3 · p. 4

3 METHOD

CVD-STORM combines a spatial-temporal reconstruction VAE with cross-view video diffusion to generate conditioned multi-view driving videos and dynamic 3D Gaussian scene reconstructions. Its pipeline fine-tunes the VAE for reconstruction before using it as the diffusion model’s latent encoder.

3 METHOD

  • The overall pipeline conditions multi-view driving video generation on text, bounding boxes, HD maps, or reference frames while producing dynamic 3D Gaussian scene reconstructions.STORM-VAE is created by fine-tuning a pretrained image VAE with an auxiliary reconstruction task, then used to train the cross-view diffusion model.Source: §3 · p. 4

3.1 PRELIMINARY: STORM

  • STORM uses image features and camera poses to predict pixel-level Gaussians, including depth and velocity, then renders target views from Gaussians transformed to their target timestamps.Training samples target views within a predefined range; RGB rendering is supervised by target images, and projected LiDAR depth supplies depth supervision.Source: §3.1 · p. 5

3.2 STORM-VAE

  • STORM-VAE sends image latents through parallel branches for high-fidelity VAE reconstruction and Gaussian Splatting decoding, combining their objectives.The Gaussian decoder operates on sampled context latents, unlike STORM’s direct processing of RGB images; the implementation omits GAN loss because experiments found it compromised training stability and sets λ to 0.5.Source: §3.2 · p. 5

3.3 CVD-STORM

  • The diffusion model appends temporal and cross-view transformer blocks after each MM-DiT block, with attention operating over image, sequence, and view dimensions.The latent tensor spans frames, viewpoints, channels, and the spatial dimensions of each image; the training loss uses rectified flow with conditioning information.Source: §3.3 · pp. 5–6
  • CVD-STORM uses STORM-VAE latents in a DiT-based video diffusion model with temporal and cross-view blocks, jointly trained with MM-DiT blocks in one stage.The single-stage training replaces a multi-stage process and is described as simplifying training and reducing computational costs.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

CVD-STORM is evaluated for video realism, temporal coherence, controllability, and reconstruction, with comparisons and ablations on driving datasets. The reported results include gains over prior methods and improvements from STORM-VAE and additional reference frames.

4.1.1 DATASETS

  • Training uses OpenDV-Youtube for single-view data and nuScenes, Waymo, and Argoverse2 for multi-view data, with 19-frame sequences and three image resolutions.The resolutions are 144 × 256, 176 × 304, and 256 × 448, sampled at ratios of 0.1, 0.3, and 0.6.Source: §4.1.1 · p. 6

4.3 ABLATION STUDY

  • Qualitative examples show decoded depth at frames 0, 5, and 10, and reference-free generation at frames 1 and 10 with bounding-box and road-map overlays.The caption associates same-colored boxes with objects of the same class, such as cars in blue boxes.Source: Figure 3 · p. 7, Figure 5 · p. 9

4.1.2 EVALUATION METRICS

  • Evaluation uses FID for realism, FVD for temporal coherence, and object-detection and road-map segmentation tasks as proxies for controllability and spatial accuracy.The passage identifies BEVFormer for object detection and cross-view transformers for road-map segmentation.Source: §4.1.2 · pp. 6–7

4.1.3 IMPLEMENTATION DETAILS

  • STORM-VAE training uses four context frames and three randomly sampled target timesteps; diffusion training freezes its encoder and uses three reference frames at inference.Inputs are standardized to six views, with attention masking for redundant fusion; single-view OpenDV-Youtube omits the cross-view block.Source: §4.1.3 · p. 7

4.3 ABLATION STUDY

  • 15.38% lower FID and 34.97% lower FVD with STORM-VAE than with the standard VAE in a same-step comparison without pretrained diffusion weights.The ablation compares models trained for the same number of steps.Source: Table 2 · p. 8, §4.2.1 · p. 8, §4.3 · p. 9
  • With reference frames, the task is video prediction; FVD improves as their number increases, while FID stays about the same.Figure 4 illustrates prediction from three reference frames and reports strong temporal consistency.Source: Table 2 · p. 8, §4.2.1 · p. 8, §4.3 · pp. 8–9

4.2.1 COMPARISON

  • 34.48% lower FID and 61.21% lower FVD than the second-best method; CVD-STORM generates videos up to 20 seconds and ranks second in vehicle IoU, marginally below UniMLVG.It also outperforms competing methods on object-detection mAP and road IoU.Source: §4.2.1 · pp. 7–8

5 CONCLUSION

CVD-STORM combines long-sequence, multi-view video generation with dynamic 4D scene reconstruction. Its Gaussian Splatting Decoder estimates absolute depth, while the authors report gains in image quality and temporal coherence.

5 CONCLUSION

  • CVD-STORM unifies long-sequence, multi-view video generation with dynamic 4D scene reconstruction.The conclusion describes this as the framework’s central combination.Source: §5 · p. 9
  • STORM-VAE adds a Gaussian Splatting Decoder that estimates absolute depth through neural rendering and provides richer 3D structural information than previous approaches.The conclusion also attributes improved representation learning and downstream diffusion generation to this design.Source: §5 · p. 10
  • The authors report that CVD-STORM surpasses state-of-the-art methods, particularly in image quality and temporal coherence.Source: §5 · p. 10

6 APPENDIX

The appendix reports reconstruction comparisons and generation-side depth evaluation, followed by qualitative video and depth results.

6.1 ADDITIONAL EXPERIMENTS

  • STORM-VAE extends STORM to six-view rendering on nuScenes and slightly outperforms STORM in reconstruction, assessed with PSNR and D-RMSE.PSNR assesses image quality, while D-RMSE assesses depth accuracy.Source: §6.1 · p. 14, §6.1.2 · p. 14, Table 3 · p. 14

6.1.1 VIDEO RESULTS

  • Zero-shot depth evaluation uses Depth Anything V2 pseudo-ground-truth, and a supplementary video provides visualization.Source: §6.1.1 · p. 14

6.1.2 COMPARISON OF STORM AND STORM-VAE.

  • Generation is compared with UniMLVG followed by STORM using progressive four-frame context; generated-depth metrics cannot assess absolute depth accuracy.The evaluation uses AbsRel and δ1, and generated results lack ground-truth depth.Source: §6.1.2 · p. 14

6.1.3 MORE QUALITATIVE RESULTS

Related papers

Loading 2510.07944v2…