GD-VDM: Generated Depth for better Diffusion-based Video Generation

1 Introduction

• The paper frames the challenge as generating diverse, complex videos while maintaining high quality, since earlier diffusion video results on driving scenes were often repetitive and empty.Video generation must learn both image generation and the dynamical model.Source: §1 · p. 1
• The approach first generates a depth video, then uses it to condition a video-to-video diffusion model that generates a real-looking video.The depth stage is intended to focus on scene composition and dynamics rather than fine details and textures.Source: §1 · p. 1
• GD-VDM produced more complex scenes with naturally moving objects, such as other vehicles.The paper also reports greater diversity and scene complexity than current VDM in its contributions.Source: §1 · p. 2
• Training with noised and denoised real depth videos was used to reduce the domain shift between training inputs and generated depth at inference.The authors report that this adjustment improved generated video quality after identifying a quality degradation relative to VDM.Source: §1 · p. 2

2.1 Denoising Diffusion Probabilistic Models (DDPM) [32, 13]

• DDPMs add Gaussian noise over T time steps using a predefined schedule, with the forward transition specified by q(xt|xt−1).The marginal q(xt|x0) and reparameterized sample express the noisy data directly from the original sample.Source: §2.1 · pp. 2–3
• DDPM training learns a reverse transition from noise toward natural data by predicting added noise rather than directly parameterizing the transition mean.The stated objective minimizes squared error between the added noise and its prediction.Source: §2.1 · p. 3

2.2 Classifier-Free Guidance [12]

• Classifier-free guidance conditions on context during training and combines conditional and unconditional noise estimates during sampling.The passage states that larger guidance weights make generation depend more heavily on context.Source: §2.2 · p. 3

3 Method

• The pipeline uses a VDM trained on depth videos to generate synthetic depth, then conditions Vid2Vid-DM on that depth to generate real-world video.The method is motivated by limited availability of videos with depth and the aim not to depend on additional depth information.Source: §3 · p. 3, §3.1 · p. 4

3.1 Depth Generation

• Depth videos are generated with a VDM trained on artificial depth maps estimated frame by frame; the authors report no observed temporal discontinuities from this process.The depth VDM uses a U-Net with 3D convolutional residual blocks and spatial-temporal attention.Source: §3.1 · p. 4

3.2 Video to Video Diffusion Model

• Vid2Vid-DM uses two 3D U-Nets: one extracts multiscale features from the conditioning video, and the other predicts noise from the noisy target video and those features.Features are concatenated at corresponding resolution scales.Source: §3.2 · p. 4

3.3 Generated Depth for better Diffusion-based Video Generation

• At inference, generated depth differs from the depth used to train Vid2Vid-DM, so training adds forward noise to real depth and denoises it with the depth model to simulate inference artifacts.This procedure is intended to align training depth more closely with depth generated by the model.Source: §3.3 · pp. 4–5, §3 · p. 4

• GD-VDM adapts the two-phase segmentation-conditioned idea of SB-GAN [1] to diffusion and proposes a way to address training–generation domain shift.The paper notes that concatenating two U-Nets had previously been proposed as W-Net.Source: §4 · p. 5
• Diffusion models have been used for image synthesis, image-to-image and text-to-image translation, and audio generation; this work uses DDPM [13].DDPM generates samples by progressively denoising white noise through a Markov chain.Source: §4 · p. 5
• Prior GAN video methods model video through structures such as static backgrounds with moving foregrounds, temporal and image generators, or separate content and motion spaces.The passage also describes video-discriminator approaches and global latent-code control.Source: §4 · p. 5
• VDM uses a 3D U-Net for temporal video modeling, but on complex driving scenes it often generated simple, empty roads despite good video quality.The paper describes later VDM extensions as orthogonal architectural and training modifications that can be combined with its contribution.Source: §4 · p. 5
• Video-to-video synthesis has received less attention than image-to-image translation and conditional video generation, and most existing solutions rely on GANs.The passage identifies potential for further development of more efficient video-to-video models.Source: §4 · p. 5

5 Experiments

• The evaluation included human ratings and FVD, but weak correlation between FVD and perceived quality led the authors to rely mainly on human evaluation.FVD results were provided for completeness.Source: §5 · pp. 5–6

5.1 Training details

• Training used 10-frame, 64×64 videos; the second-stage depth U-Net was initialized from the first-stage model and then updated during training.The first stage trained a VDM on depth videos, followed by Vid2Vid diffusion-model training.Source: §5.1 · p. 6

5.2 Datasets

• Cityscapes depth videos were estimated with Monodepth2, and a denoised-depth dataset was created by diffusing and denoising those videos for Vid2Vid-DM training.The Cityscapes training set had 2,975 videos; validation had 500 and test had 1,525.Source: §5.2 · p. 6
• On SUN3D, all methods generated visually pleasing videos, which the authors attribute to scenes with a moving camera and static background.SUN3D contains 415 sequences captured across 254 areas in 41 buildings.Source: §5.2 · p. 6

5.4 Human evaluation and analysis

• Human ratings favored GD-VDM over baselines for diversity and scene complexity, with twice as many moving vehicles, while VDM scored better on video quality.The study used 100 videos per approach, each reviewed by three annotators; denoised-depth training improved all reported human-evaluation metrics.Source: Table 1 · p. 7, §5.4 · p. 7

5.3 Baselines

• The Cityscapes baselines were Video Diffusion Model, a 3D U-Net video diffusion model, and MoCoGAN, a GAN-based model separating motion and content representations.Both baselines were trained on the Cityscapes video dataset.Source: §5.3 · p. 7

6 Conclusion

• GD-VDM first generates depth that defines scene layout, then uses it to generate a fully detailed video.The paper presents this as a two-phase diffusion model for video generation.Source: §6 · p. 8
• A diffusion Vid2Vid model generates coherent and realistic videos conditioned on the depth video.Source: §6 · p. 8
• The authors report that GD-VDM produces more diverse videos and presents complex scenes.They evaluated the approach against VDM using human evaluation and FVD.Source: §6 · p. 8

B Fréchet Video Distance (FVD) results

• GD-VDM outperformed VDM and MoCoGAN on FVD, but its denoised variant scored worse than the standard method.The authors mainly relied on human evaluation because FVD did not strongly correlate with perceived video quality.Source: B Fréchet Video Distance (FVD) results · p. 12, Table 2 · p. 13

C.1 SUN3D

• On SUN3D, VDM produced videos with impressive visual quality, which the authors attribute to moving-camera scenes with static backgrounds.Figures 6 and 7 show generated examples from the methods trained on SUN3D.Source: §C.1 · p. 13, Figure 6 · p. 14, Figure 7 · p. 15

C.2 Cityscapes

• The appendix also shows randomly generated Cityscapes videos, including five-frame sequences from three methods alongside two ground-truth sequences.Source: Figure 8 · p. 16, §C.2 · p. 16, Figure 9 · p. 17

D Human Evaluation

• Figures 10–13 depict the Mechanical Turk evaluation screenshots, including instructions for both tasks.Source: D Human Evaluation · p. 18, Figure 11 · p. 18, Figure 13 · p. 19
• Raters evaluated 10 videos from the same method for video quality and overall diversity.Source: Figure 10 · p. 18
• A second experiment asked raters to identify vehicles, pedestrians, and cyclists in the scene.Source: Figure 12 · p. 19