PaperlayerView exact revision on arXiv

Source-linked AI summary

DriveGen3D: Boosting Feed-Forward Driving Scene Generation with Efficient Video Diffusion

Weijie Wang, Jiagang Zhu, Zeyu Zhang, Xiaofeng Wang, Zheng Zhu, Guosheng Zhao, Chaojun Ni, Haoxiao Wang, Guan Huang, Xinze Chen, Yukun Zhou, Wenkang Qin, Duochao Shi, Haoyun Li, Yicheng Xiao, Donny Y. Chen, Jiwen Lu

arXiv:2510.15264v3cs.CV

TL;DR

Driving-scene synthesis still lacks an efficient framework joining long-horizon video generation with large-scale dynamic 3D reconstruction. DriveGen3D combines condition-guided video diffusion with feed-forward temporal-aware Gaussian reconstruction, outperforming state-of-the-art baselines in video quality and 3D reconstruction accuracy with end-to-end generation under six minutes.

  • Problem

    Driving-scene methods often address temporal video coherence or spatial reconstruction fidelity separately, and can be computationally costly or limited in scalability.

    Source: §I · p. 1

  • Method

    DriveGen3D uses text- and BEV-guided FastDrive-DiT video synthesis followed by FastRecon3D feed-forward reconstruction with temporal-aware Gaussian splatting.

    Source: §I · p. 1, A. Overview · p. 3

  • Results

    Across video quality and 3D reconstruction accuracy, DriveGen3D outperforms state-of-the-art baselines and completes end-to-end generation in under 6 minutes.

    Source: §I · pp. 1–2

  • Takeaways & Limitations

    The framework provides a unified route to generating long-duration driving videos and high-quality large-scale dynamic scenes from text prompts and BEV layouts.

    Source: §I · p. 1, §V · p. 6

    Generated videos may mismatch ground-truth 3D box layouts or appear blurry, indicating a need to optimize video generation for reconstruction.

    Source: 3) Visualization and Comparison · p. 6

Abstract

from arXiv · show

We present DriveGen3D, a novel framework for generating high-quality and highly controllable dynamic 3D driving scenes that addresses critical limitations in existing methodologies. Current approaches to driving scene synthesis either suffer from prohibitive computational demands for extended temporal generation, focus exclusively on prolonged video synthesis without 3D representation, or restrict themselves to static single-scene reconstruction. Our work bridges this methodological gap by integrating accelerated long-term video generation with large-scale dynamic scene reconstruction through multimodal conditional control. DriveGen3D introduces a unified pipeline consisting of two specialized components: FastDrive-DiT, an efficient video diffusion transformer for high-resolution, temporally coherent video synthesis under text and Bird's-Eye-View (BEV) layout guidance; and FastRecon3D, a feed-forward module that rapidly builds 3D Gaussian representations across time, ensuring spatial-temporal consistency. DriveGen3D enable the generation of long driving videos (up to $800\times424$ at $12$ FPS) and corresponding 3D scenes, achieving state-of-the-art results while maintaining efficiency.

I. INTRODUCTION

DriveGen3D combines accelerated, multimodally guided driving-video synthesis with feed-forward dynamic 3D reconstruction. It targets the gap between long-horizon video generation and scalable 3D scene reconstruction, reporting stronger quality and accuracy with end-to-end generation under six minutes.

  • An integrated, efficient framework remains lacking: prior work tends to prioritize temporal video coherence or spatial reconstruction fidelity, with high costs or limited scalability.MagicDriveDiT [3] may take up to 30 minutes for one 20-second video, while optimization-based reconstruction can take 30 minutes per scene and faster feed-forward methods rarely integrate with dynamic video generation.Source: §I · p. 1
  • DriveGen3D pairs FastDrive-DiT for high-resolution driving-video generation with FastRecon3D, which reconstructs dynamic 3D scenes from multi-view video frames under text and BEV layout control.The overview depicts the video synthesis stage followed by per-frame 3D Gaussian Splatting reconstruction.Source: §I · p. 1, Figure 1 · p. 2
  • FastDrive-DiT uses conditional-branch caching and targeted 8-bit quantization of cross-view attention to reduce inference time by over 2× while preserving visual fidelity and layout control.The authors identify caching the conditional branch as a way to maximize speedup without compromising layout control.Source: §I · pp. 1–2
  • FastRecon3D uses temporal-aware Gaussian splatting and cross-attention over past and future context to reconstruct spatially accurate, temporally coherent primitives and mitigate motion blur in dynamic elements.The approach contrasts with treating frames independently or optimizing each scene individually.Source: §I · p. 1
  • On generated inputs, DriveGen3D achieves PSNR 23.25, SSIM 0.820, and LPIPS 0.176, while outperforming state-of-the-art baselines in video quality and 3D reconstruction accuracy; end-to-end time is under 6 minutes.The framework also reports high-quality video generation and complete large-scale street-scene reconstruction within 6 minutes.Source: §I · pp. 1–2

II. RELATED WORK

Prior driving-scene work advances controllable video generation and dynamic-scene reconstruction through distinct approaches. The review identifies scale and integration limits across feed-forward reconstruction and conditional scene synthesis.

  • Driving-video methods include MagicDrive’s camera-pose and 3D-box controls [25], MagicDriveDiT’s high-resolution long-video generation [3], and UniScene’s joint generation of occupancy, video, and LiDAR [26].The passage describes MagicDriveDiT as using flow matching and spatial-temporal conditional encoding, and UniScene as using progressive generation.Source: §II · p. 2
  • Dynamic reconstruction progressed from instance-specific optimization in StreetGaussian to feed-forward multiview inference in PixelSplat and MVSplat [19], while temporal methods include InfiniCube [31], Drive3R [20], and DrivingForward [13].PixelSplat and MVSplat reduce reconstruction from minutes to seconds; the passage notes fidelity challenges in geometrically complex regions or under sparse observations.Source: §II · p. 2
  • The stated gaps are constrained spatial resolution in feed-forward 3D reconstruction, typically at or below 512 × 512, and substantial compute and integration complexity in prior conditional scene-generation frameworks.The review also notes that feed-forward methods may sacrifice fidelity in complex regions or with sparse observations.Source: §II · p. 2

III. METHOD

DriveGen3D first synthesizes condition-guided driving videos and then reconstructs dynamic 3D scenes with temporal-aware Gaussian splatting. FastDrive-DiT combines caching and attention quantization to accelerate video generation, while FastRecon3D fuses temporal context for coherent per-frame Gaussians.

A. Overview

  • The overview pipeline sends high-resolution, temporally coherent videos synthesized under conditional guidance to FastRecon3D for feed-forward dynamic-scene reconstruction with temporal-aware Gaussian splatting.The stages are presented as a unified, rapid and scalable 3D scene-generation framework.Source: A. Overview · p. 3

B. FastDrive-DiT

  • FastDrive-DiT is built on MagicDriveDiT [3] to address the high cost of diffusion-based driving-video generation, which can take up to 30 minutes per video.The motivating cost is associated with the multi-view nature of driving-scene generation and its diffusion process.Source: B. FastDrive-DiT · p. 3
  • TeaCache accelerates diffusion generation by caching outputs using timestep-embedding-based estimates of changes, achieving up to 4.41× acceleration while maintaining visual quality.The adaptation computes caching coefficients from the conditional branch alone, reducing TeaCache forwarding time without obvious performance degradation.Source: B. FastDrive-DiT · p. 3
  • SageAttention quantizes attention at inference; applying it to transformer blocks saves an additional 30 seconds with nearly no performance degradation.The method focuses quantization on attention to address its high computational cost in video DiT.Source: B. FastDrive-DiT · p. 3

C. FastRecon3D

FastRecon3D is a feed-forward, temporal-aware Gaussian Splatting module that reconstructs dynamic 3D scenes from video using past and future context. Its recursive reconstruction produces complete 3D models in seconds.

C. FastRecon3D

  • Each timestep is represented by per-frame 3D Gaussian primitives, with the formulation designed to maintain 3D consistency.The method targets motion blur in dynamic elements and limits on large-scale reconstruction in prior approaches.Source: C. FastRecon3D · p. 3
  • Cross-attention fuses information from other frames into the current frame’s multi-view features for temporal reconstruction.The model extracts features from frames around timestep t before reconstructing the scene at t.Source: C. FastRecon3D · p. 4
  • The network predicts Gaussian parameters and saves them as a 3D model.The reconstruction uses a Gaussian network and a pretrained image encoder to extract features.Source: C. FastRecon3D · p. 4
  • Seconds: FastRecon3D produces complete 3D models using past and future context, avoiding costly scene-by-scene optimization.The approach is designed for simulation and other real-time applications.Source: C. FastRecon3D · pp. 3–4

A. Experimental Setup

Experiments use nuScenes data and evaluate video generation and 3D reconstruction with separate metrics and protocols. The reported setup includes the training resolution, inference hardware, and video lengths used for timing comparisons.

A. Experimental Setup

  • 700 training videos and 150 validation videos come from nuScenes; the reconstruction model uses 20,000 short training sequences.The dataset is identified as nuScenes.Source: A. Experimental Setup · p. 4
  • Video generation is evaluated with FVD for quality and mAP and mIoU for controllability using BEVFormer pretrained on real-world data.Perception metrics are applied to generated videos corresponding to validation-set annotations.Source: A. Experimental Setup · p. 4
  • For reconstruction, the model uses frames at t−2 and t+2 to synthesize the intermediate frame, scored with PSNR, SSIM, and LPIPS against ground truth.The protocol evaluates both ground-truth-image inputs and synthesized-video outputs.Source: A. Experimental Setup · p. 4
  • Training uses 800 × 424 resolution, inference runs on NVIDIA H20 GPUs, and timing comparisons include 17-frame and 233-frame videos.The reconstruction model is trained on one NVIDIA H20 GPU for two days; inference timing tables cover attention-block costs and acceleration.Source: A. Experimental Setup · p. 4, Table II · p. 5, Table III · p. 5

1) Novel View Synthesis:

With generated-image inputs, DriveGen3D maintains competitive reconstruction quality and achieves the highest reported SSIM among the compared methods.

1) Novel View Synthesis:

  • 22.84 PSNR and 0.811 SSIM are reported for generated-image inputs, with the highest SSIM among compared methods.The comparison includes optimization-based dynamic methods and feed-forward reconstruction methods.Source: 1) Novel View Synthesis · p. 5

2) Experimental Analysis and Ablation Study:

The ablations examine conditional-branch caching and attention quantization as ways to accelerate video generation. The visualization compares the baseline with the accelerated and quantized variants.

2) Experimental Analysis and Ablation Study:

  • Nearly 3× and 2× speedups on 17- and 233-frame videos, respectively, came from applying TeaCache only to MagicDriveDiT’s conditional branch.The perception metrics mAP and mIoU showed only slight decreases.Source: 2) Experimental Analysis and Ablation… · p. 5
  • Cross-view attention is MagicDriveDiT’s most computationally costly process; its V range is 10 times smaller than spatial and 5 times smaller than temporal.The smaller range motivates lower-resolution quantization for cross-view V.Source: 2) Experimental Analysis and Ablation… · p. 5
  • SageAttention on transformer blocks saves another 30 seconds for 233 frames after diffusion-step acceleration while maintaining performance.The figure compares MagicDriveDiT video generation with diffusion-step acceleration and further quantization.Source: 2) Experimental Analysis and Ablation… · p. 5, Figure 5 · p. 5

3) Visualization and Comparison:

Visual comparisons report no obvious change from the acceleration and quantization techniques, while the full pipeline generates and reconstructs street scenes for more than 20 seconds at 12 FPS. The conclusion frames DriveGen3D as an integration of long-duration video generation and large-scale dynamic scene reconstruction.

3) Visualization and Comparison:

  • The accelerated and quantized variants show no obvious visual change and can still generate videos longer than 15 seconds at faster speed.The comparison is between the baseline, diffusion-step acceleration, and further SageAttention quantization.Source: 3) Visualization and Comparison · p. 5
  • DriveGen3D generates and reconstructs street scenes for more than 20 seconds at 12 FPS.This is a typical full-pipeline output.Source: 3) Visualization and Comparison · p. 6
  • Generated-video reconstruction shows bus-location mismatch and a blurry back-left view, while pole flicker appears with both generated and GT videos.The text attributes the flickering-pole drawback to the second-stage reconstruction model.Source: 3) Visualization and Comparison · p. 6

V. CONCLUSION

  • DriveGen3D integrates longitudinal video generation and scene reconstruction, bypassing voxel-based generation paradigms for real-world driving-scenario reproduction and dynamic world modeling.The conclusion identifies autonomous-vehicle simulation and dynamic world modeling as potential applications.Source: §V · p. 6

Related papers

Loading 2510.15264v3…