PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion Probabilistic Modeling for Video Generation

Ruihan Yang, Prakhar Srivastava, Stephan Mandt

arXiv:2203.09481v5cs.CVcs.LGstat.ML

TL;DR

Video prediction requires stochastic, multimodal forecasts that retain long-term dynamics and sharp high-resolution content. The paper proposes an autoregressive diffusion model that corrects deterministic next-frame predictions with generated residuals, achieving state-of-the-art perceptual quality and competitive or better probabilistic forecasting across evaluated datasets and baselines.

  • Problem

    Video prediction must produce stochastic multimodal forecasts, capture high-dimensional dynamics over time, and scale to high-resolution content without blurry artifacts.

  • Method

    Residual Video Diffusion combines deterministic convolutional-RNN next-frame prediction with a conditional denoising diffusion process that generates additive residual corrections.

  • Results

    Across four datasets and comparisons with two VAE and three GAN baselines, the method achieves state-of-the-art perceptual quality and competitive or better probabilistic forecasting.

  • Takeaways & Limitations

    Residual prediction provides a probabilistic video-generation approach whose perceptual quality is better than or comparable with likelihood-free methods and whose forecasting is competitive with strong VAE and GAN baselines.

  • Takeaways & Limitations

    Unconditional complete-video generation requires an auxiliary image model for initial context frames, and experiments are limited to single-domain datasets with monotonic content.

Abstract

from arXiv · show

Denoising diffusion probabilistic models are a promising new class of generative models that mark a milestone in high-quality image generation. This paper showcases their ability to sequentially generate video, surpassing prior methods in perceptual and probabilistic forecasting metrics. We propose an autoregressive, end-to-end optimized video diffusion model inspired by recent advances in neural video compression. The model successively generates future frames by correcting a deterministic next-frame prediction using a stochastic residual generated by an inverse diffusion process. We compare this approach against five baselines on four datasets involving natural and simulation-based videos. We find significant improvements in terms of perceptual quality for all datasets. Furthermore, by introducing a scalable version of the Continuous Ranked Probability Score (CRPS) applicable to video, we show that our model also outperforms existing approaches in their probabilistic frame forecasting ability.

1 Introduction

Video prediction must generate stochastic, multimodal futures while preserving long-term dynamics and sharp high-resolution frames. The paper addresses these goals with Residual Video Diffusion, combining deterministic next-frame prediction with diffusion-generated residual corrections.

  • Video prediction supports applications including event anticipation, reinforcement learning, interpolation, pedestrian forecasting, precipitation nowcasting, and neural video compression.
  • Existing approaches trade off multimodal stochastic prediction, long-term dynamics, and scalable high-resolution generation without blurry artifacts.Sequential VAEs tend to perform better on the first two goals, while sequential GANs tend to perform better on scalability and sharpness.
  • Residual Video Diffusion predicts the next frame deterministically, then corrects it with an additive residual generated by a conditional denoising diffusion process.The model uses convolutional RNNs to predict the next frame and diffusion context from preceding frames.
  • The approach is designed to be scalable to high-resolution video, stochastic, and based on likelihood-based principles.
  • Diffusion probabilistic models provide a route to probabilistic video forecasting with perceptual quality better than or comparable with likelihood-free methods such as GANs.
  • The paper introduces a marginal Continuous Ranked Probability Score for video and reports better probabilistic forecasting than modern GAN and VAE baselines.The named baselines include IVRNN, SVG-LP, RetroGAN, DVD-GAN, and FutureGAN.
  • Ablation studies show that modeling residuals from predicted next frames performs better than directly modeling the next frames.

2 A Diffusion Probabilistic Model for Video

The model extends diffusion probabilistic modeling to video by representing each future frame as a deterministic prediction plus a stochastic residual. Its generative and inference processes condition residual diffusion on temporal context from preceding frames.

  • 2.1 Background on Diffusion Probabilistic Models: DDPMs use a forward process that destroys data structure incrementally and a reverse denoising process to generate structure.Their maximum-likelihood training is described as stable while producing perceptual quality comparable with GANs.
  • 2.2 Residual Video Diffusion Model: Residual Video Diffusion generates stochastic residual corrections to deterministically predicted next frames, with both stages conditioned on long-range convolutional-RNN context.The approach is named Residual Video Diffusion (RVD).
  • 2.2 Residual Video Diffusion Model: The model defines scaled frame residuals as latent variables associated with the video frame sequence.
  • 2.2 Residual Video Diffusion Model: An autoregressive masked autoregressive flow predicts each frame using a mean µφ and scale σ, while the inverse transform decorrelates frames into residuals.The resulting temporally decorrelated, sparse residuals are described as simpler to model than the frames themselves.
  • 2.2 Residual Video Diffusion Model: The residual generative process conditions denoising factors on past frames and uses prediction networks Mθ with a fixed multivariate Gaussian top-level prior.The denoising variance parameter γ is treated as a hyperparameter.
  • 2.2 Residual Video Diffusion Model: Inference uses a deterministic residual derived from observed and predicted frames, followed by diffusion factors with a cosine variance schedule.
  • 2.2 Residual Video Diffusion Model: The architecture generalizes TimeGrad by modeling residuals and temporal dependency in the posterior, identified as crucial for competitiveness with strong VAE and GAN baselines.
  • 2.2 Residual Video Diffusion Model: Training uses stochastic-gradient optimization with sampled noise and diffusion steps, plus teacher forcing for multiple time steps.

3 Experiments

Experiments evaluate RVD across four video datasets against five baselines using perceptual and probabilistic forecasting metrics. RVD achieves consistently strong perceptual quality, competitive CRPS forecasting, and benefits from residual modeling, particularly on complex high-resolution videos.

  • Experimental Setup: RVD is compared with five baselines across four datasets using FVD, LPIPS, and video CRPS.The datasets span simpler 64×64 videos and complex 128×128 Cityscape and Simulation videos.
  • Evaluation Metrics: CRPS is computed per pixel and averaged across spatial and temporal dimensions to evaluate marginal probabilistic forecasts at video scale.The spatially resolved version compares RVD and IVRNN on the 10th future Cityscape frame, with darker regions indicating larger disagreement.
  • Perceptual Quality: RVD performs best in FVD on all datasets and best in LPIPS on 3 out of 4 datasets.Figure 2 also shows higher clarity and less blurriness for RVD on Cityscape frames.
  • Probabilistic Forecasting: RVD performs best in CRPS forecasting in 3 out of 4 cases, while inverse CRPS scores decrease monotonically with future-frame index.The strongest performances are obtained by RVD and IVRNN, reflecting worsening predictions farther into the future.
  • Ablation Studies: Across all datasets and perceptual metrics, residual modeling outperforms direct next-frame modeling; CRPS gains are strongest on complex Simulation and Cityscape data.The ablation compares RVD with the VD variant that directly models future frames.
  • Ablation Studies: RVD maintains robust performance with shorter training sequences, whereas IVRNN is more sensitive to sequence length.RVD outperforms IVRNN in most experiments despite being trained on shorter sequences.

4 Related Work

Prior video-generation research spans supervised metrics, sequential latent-variable models, GANs, diffusion models, and neural video compression. These approaches expose trade-offs among perceptual quality, stochasticity, long-term consistency, and scalability.

  • Video prediction is evaluated either with supervised error metrics such as PSNR and SSIM or with distributional and perceptual metrics for stochastic generation.
  • Sequential variational models are a major video-generation approach, extending recurrent networks with latent variables and increasingly expressive priors.
  • GAN-based sequential models avoid blurry artifacts but tend to suffer from long-term consistency.
  • Diffusion models progressed from thermodynamic and score-based formulations to high-resolution image generation and applications beyond images.
  • TimeGrad introduced sequential diffusion forecasting for lower-dimensional time series, while concurrent work studied video diffusion with an alternative architecture focused primarily on perceptual metrics.
  • Neural video compression models separately encode predictions and residual errors, achieving strong rate-distortion performance with hierarchical structures.

5 Discussion

Residual Video Diffusion combines autoregressive frame prediction with diffusion-generated residuals for stochastic video generation. Across varied datasets and baselines, it improves perceptual quality and remains competitive or better for probabilistic forecasting, while its scope and computational demands remain bounded.

  • Residual Video Diffusion generates a residual with a context-conditioned denoising process and adds it to a deterministic next-frame prediction.
  • The evaluation covers datasets with different complexity and pixel resolution, including Cityscape and turbulent-flow physics simulations, against two VAE and three GAN baselines.
  • The method achieves state-of-the-art perceptual quality and is competitive with or better than hierarchical VAE and GAN baselines for probabilistic forecasting.
  • Conditional generation requires at least one context frame pre-selected from the test dataset, while unconditional complete-video generation requires an auxiliary image generator.
  • Experiments use single-domain datasets with monotonic content, and diffusion models tend to be slow to train.
  • Video-generation models can be misused to create deepfakes and spread misinformation, although probabilistic prediction may support likelihood-based anomaly detection.

A Architecture

The architecture extends DDPMs with temporal conditioning for denoising and next-frame transformation. Its U-Net-style modules combine recurrent temporal processing, attention, residual blocks, positional conditioning, and spatial scaling.

  • The architecture extends the DDPM architecture to a temporally conditioned version with separate denoising and transform modules.
  • Channel dimensions are multiplied across subsequent denoising and transform downsampling layers, with reverse multipliers during upsampling.
  • Residual blocks use 3×3 kernels, LeakyReLU activation, and Group Normalization, while ConvGRUs use 3 × 3 kernels for temporal information.
  • Linear attention uses four heads with 16 dimensions per head, and denoising-step n is encoded positionally and added to residual blocks.
  • Upsampling and downsampling spatially scale feature maps by factors of 2 and 1/2, respectively.
  • The denoising module predicts noise from a noisy residual at diffusion step n while conditioning on all past context frames, using N = 1600 steps in the described setup.
  • The transform module uses a U-Net-style structure with four downsampling and upsampling layers in the low-resolution setting.

B Deriving the Optimization Objective

The optimization objective derives a variational lower bound for the sequential diffusion model and simplifies its terms using Gaussian structure and fixed, untrainable parameters. The resulting objective is a denoising score-matching objective tied to residual prediction.

  • The derivation follows DDPM methodology to obtain a variational lower bound for the sequential generative model.
  • Observed frames x0:T and diffusion variables y1:T define the sequential model, with latent diffusion variables separated from observed scaled residuals.
  • The scaled residual is defined as yt0 = (xt−µφ(x<t))/σ for t = 1, ..., T.
  • The third variational-bound term matches the diffusion model output to the frame residual.
  • A KL-divergence term between Gaussians with fixed variances simplifies to an L2 distance between their means.
  • The simplified stochastic objective drops untrainable parameters and yields the paper's denoising score-matching objective.

C Additional Generated Samples

Qualitative examples cover simulation, BAIR Robot Pushing, and KTH Actions, comparing generated frames with ground truth and VAE/GAN baselines. The method is perceptually strongest across all datasets, while CRPS performance is strongest on high-resolution data and competitive with IVRNN elsewhere.

  • Additional Generated Samples: Qualitative examples compare ground-truth frames with outputs from the method and VAE/GAN baselines across three additional datasets.The datasets are Simulation data, BAIR Robot Pushing, and KTH Actions.
  • Additional Generated Samples: The method and IVRNN are the strongest qualitative contenders in the displayed examples.
  • Additional Generated Samples: FVD indicates that the method outperforms on every dataset.
  • Additional Generated Samples: CRPS shows top performance on high-resolution datasets and competitive performance against IVRNN on the other two datasets.
  • Simulation Data: Simulation data uses four context frames to predict the next sixteen frames at 128 × 128 resolution.The simulation depicts a von Kármán vortex street generated with a Lattice Boltzmann Solver.
  • BAIR Robot Pushing: BAIR Robot Pushing uses four context frames to predict the next sixteen frames at 64 × 64 resolution, with temporal consistency and occlusion handling as challenges.
Loading 2203.09481v5…