Source-linked AI summary
MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation
Vikram Voleti, Alexia Jolicoeur-Martineau, Christopher Pal
TL;DR
Video synthesis remains challenging because models must generate high-quality, temporally plausible futures, while existing frameworks typically do not handle prediction, generation, and interpolation together. MCVD trains one masked conditional diffusion model for these tasks using blockwise convolutional processing and autoregressive generation, achieving SOTA prediction and interpolation results. The method is designed to work within modest computational resources, but its frame-block capacity is constrained by GPU and budget limits.
Problem
Video synthesis requires high-quality, temporally plausible frames, while existing specialized frameworks typically handle prediction, unconditional generation, or interpolation separately.
Method
MCVD randomly masks past and/or future frame blocks during conditional score-based diffusion training so one convolutional model supports prediction, generation, and interpolation.
Results
MCVD achieves SOTA video prediction and interpolation results across reported benchmarks, including an FVD of 145.5 on Cityscapes versus 418 for the best previous result reported by the authors.
Takeaways & Limitations
Past masking improved performance across all tested model variants and configurations, while strong interpolation results suggest a route to longer consistent videos through sparse-frame generation followed by interpolation.
Takeaways & Limitations
Models predict only 4–5 current frames at a time and rely on autoregressive rollout, while larger-frame models were constrained by computational-budget limits.
Abstract
from arXiv · showhide
Video prediction is a challenging task. The quality of video frames from current state-of-the-art (SOTA) generative models tends to be poor and generalization beyond the training data is difficult. Furthermore, existing prediction frameworks are typically not capable of simultaneously handling other video-related tasks such as unconditional generation or interpolation. In this work, we devise a general-purpose framework called Masked Conditional Video Diffusion (MCVD) for all of these video synthesis tasks using a probabilistic conditional score-based denoising diffusion model, conditioned on past and/or future frames. We train the model in a manner where we randomly and independently mask all the past frames or all the future frames. This novel but straightforward setup allows us to train a single model that is capable of executing a broad range of video tasks, specifically: future/past prediction -- when only future/past frames are masked; unconditional generation -- when both past and future frames are masked; and interpolation -- when neither past nor future frames are masked. Our experiments show that this approach can generate high-quality frames for diverse types of videos. Our MCVD models are built from simple non-recurrent 2D-convolutional architectures, conditioning on blocks of frames and generating blocks of frames. We generate videos of arbitrary lengths autoregressively in a block-wise manner. Our approach yields SOTA results across standard video prediction and interpolation benchmarks, with computation times for training models measured in 1-12 days using $\le$ 4 GPUs. Project page: https://mask-cond-video-diffusion.github.io ; Code : https://github.com/voletiv/mcvd-pytorch
1 Introduction
Video generation must produce both high-quality frames and plausible temporal changes, while long-horizon, unconditional, prediction, and interpolation tasks remain difficult to handle together. MCVD addresses these challenges with masked conditional diffusion, blockwise conditioning, and autoregressive generation using a simple convolutional architecture.
- Video generation requires high-quality frames and plausible temporal changes drawn from complex distributions over possible futures.
- Long-horizon generation is difficult because the number of possible futures grows exponentially with time.
- Existing specialized tools rarely support video prediction, unconditional generation, and interpolation within one framework.
- MCVD uses masked past and/or future frame conditioning to support future and past prediction, unconditional generation, and interpolation with a single model.
- MCVD uses a sliding-window blockwise autoregressive procedure for fast, coherent long-term generation with a convolutional U-net and SPATIN normalization.
- The approach conditions on frame blocks, transfers temporal dynamics implicitly, and avoids explicit optical-flow, derivative, and recurrent components.
2 Conditional Diffusion for Video
The section develops conditional diffusion for video prediction, then extends it with masking so one model supports prediction, generation, and interpolation. It uses blockwise autoregression and a 2D-convolutional U-Net architecture conditioned on past and future frames.
- Conditional diffusion: Score-based diffusion reverses a data-to-noise process, generating samples by denoising from Gaussian noise with a time-conditional neural network.The network estimates added noise, which supports reverse diffusion from noise to data.
- Video prediction: Conditioning diffusion on past frames enables direct modeling of immediate future video frames and blockwise autoregressive prediction of subsequent frames.Previously predicted frames become conditioning frames for later blocks.
- Blockwise generation: The model generates arbitrary-length videos by conditioning on frame blocks and autoregressively generating blocks rather than individual frames.This blockwise strategy is illustrated with ground-truth and generated frame rows in Figure 2.
- Masked conditional diffusion: Masking past frames during training enables the same network to perform both conditional video prediction and unconditional frame generation.Past frames are zeroed with probability pmask = 1/2 using a binary mask.
- Masked conditional diffusion: Randomly and independently masking past and future frames supports future prediction, past prediction, unconditional generation, and interpolation within one model.Interpolation uses unmasked past and future frames, while unconditional generation masks both sets.
- Network architecture: The denoising network is a U-Net using 2D convolutions, self-attention, adaptive group normalization, noise-level embeddings, and SPATIN conditioning.Past and future frames are concatenated separately from noisy current frames; SPATIN uses conditional embeddings to influence normalization.
3 Related work
The related work spans diffusion, recurrent, transformer, GAN-based, and specialized interpolation approaches. MCVD’s block-autoregressive design occupies a middle ground between generating all frames simultaneously and generating one frame at a time.
- Diffusion models: Prior score-based diffusion work covers image editing and video generation, while contemporary video diffusion methods also study conditional sampling and generation.MCVD is positioned as a directly conditional diffusion model rather than an unconditional model adapted through gradient-based conditional sampling.
- Recurrent models: Earlier video prediction and generation methods include stochastic and variational recurrent models that use latent variables and recurrent dynamics.Examples include SV2P, SVG, SAVP, SRVP, and SLAMP.
- Transformer models: Transformer-based methods model video in pixel or latent representations, including VQ-VAE codes, spatio-temporal volumes, and adversarially trained autoencoder latents.Some latent-space approaches additionally use optical-flow modules.
- GAN-based models: GAN-based video methods have used spatio-temporal CNNs, convolutional LSTMs, and temporal generators to model video content and motion.These approaches include early spatio-temporal GANs and models separating motion from content.
- Interpolation: Video interpolation methods are often specialized around optical flow or motion-field modeling, while frame interpolation is also studied for video compression.This specialization contrasts with MCVD’s shared masked-conditioning setup across tasks.
- Block-autoregressive generation: Block-autoregressive generation combines the speed and extensibility of block generation with recurrently extending videos, using k < n frames per iteration.It lies between non-recurrent all-frame generation and recurrent one-frame-at-a-time prediction.
4 Experiments
Experiments evaluate MCVD on prediction, unconditional generation, and interpolation across progressively difficult video datasets. The reported results show strong prediction and interpolation performance, while ablations examine masking, architecture, sampling, and computational constraints.
- Experimental setup: Experiments cover prediction on SMMNIST, KTH, BAIR, and Cityscapes, plus unconditional generation on BAIR and UCF-101 and interpolation on SMMNIST, KTH, and BAIR.The datasets progress from simple black-and-white digits to complex natural driving scenes and diverse natural videos.
- Experimental setup: 100 sampling steps are used for evaluation, although models are trained with 1000; DDPM generally produces better metrics than DDIM except on UCF-101.Using 1000 sampling steps could yield better results.
- Limitations and efficiency: Models predict only 4-5 current frames at a time and generate longer sequences autoregressively; larger-frame experiments did not substantially improve results under computational budget constraints.The authors emphasize that the method performs well with fewer computational resources.
- Results: MCVD achieves SOTA prediction performance on SMMNIST, BAIR, and Cityscapes; on Cityscapes, it obtains FVD 145.5 versus 418 for the best previous result reported by the authors.The Cityscapes evaluation conditions on 2 frames and predicts 28 frames.
- Results: Interpolation results are SOTA by a wide margin across SMMNIST, KTH, and BAIR, including comparisons with architectures specialized for interpolation.The result is reported in Table 7.
- Ablations: Training with random masking across multiple tasks improves generated-frame quality over training on individual tasks, and the general past-future-mask model outperforms task-specific models in ablations.The ablations compare concatenated-pixel inputs, SPATIN variants, and masking configurations.
5 Conclusion
The paper reports state-of-the-art video prediction and interpolation using randomly masked conditional video diffusion with a relatively simple architecture. It also identifies longer consistent generation and computational efficiency as important future directions and considerations.
- MCVD achieves SOTA video prediction and interpolation results with randomly masked conditional video diffusion and a relatively simple architecture.
- Past-masking improved performance across all tested model variants and configurations.
- Strong interpolation results suggest generating sparse distant frames first, then filling missing frames with an interpolative diffusion step.
- High-quality video generation may support autonomous-vehicle applications, while also posing risks from fake video content and high computational resource use.
Checklist
The checklist reports affirmative coverage of claims, limitations, societal impacts, reproducibility materials, training details, error bars where possible, and compute reporting, while noting that asset licenses were not mentioned.
- The authors report that the paper’s claims and scope accurately reflect its contributions and that its limitations were described.
- The paper reports discussing potential negative societal impacts and following the ethics review guidelines.
- The authors report providing code, data, instructions, and training details through the supplementary material or a URL.
- Error bars were reported whenever possible, and total compute and resource types were included.
- Existing assets were cited, but their licenses were not mentioned.
A Appendix
The appendix provides additional compute, memory, batch-size, and timing information alongside results and visualizations across five video datasets. It also highlights MCVD configurations that achieve state-of-the-art results across many benchmark settings.
- Additional information on model size, memory requirements, batch size, and computation times is provided in Table 8.
- Additional results and visualizations cover SMMNIST, KTH, BAIR, UCF-101, and Cityscapes.
- MCVD concat past-future-mask and past-mask configurations yield SOTA results across many benchmark configurations.
- The authors had difficulty integrating older FID and IS implementations for UCF-101 into their PyTorch codebase.
A.1 Computational requirements
Table 8 reports the compute used for training, defining the selected checkpoint and the training-time measure used for comparison.
- Table 8 reports compute used, including the checkpoint with the best approximate FVD and total training time up to that checkpoint.
- The table defines “steps” as the checkpoint with the best approximate FVD.
- The table defines “GPU hours” as total training time up to the reported steps.
A.2 Stochastic Moving MNIST
The SMMNIST evaluation reports results for models conditioned on five past frames and predicting ten frames, alongside visual prediction examples.
- The appendix provides additional SMMNIST configurations and visual results for the evaluation.
- Table 9 evaluates MCVD on SMMNIST with five past frames as conditioning and ten predicted frames.
- Figure 5 compares real SMMNIST ground-truth sequences with MCVD-predicted sequences.The top row shows real ground truth, while the bottom row shows model predictions.
A.3 KTH
The KTH evaluation includes long-horizon prediction results and visual examples comparing ground-truth frames with MCVD predictions.
- Table 10 reports KTH results for predicting 30 and 40 frames with models trained to predict k frames at a time.All models condition on ten past frames, and the table notes that SSIM values were updated after fixing a calculation bug.
- The KTH evaluation therefore presents both quantitative long-horizon results and qualitative prediction comparisons.
- Figure 6 shows KTH prediction examples in which MCVD predicts 20 frames from five-frame training configurations.Only two conditional frames are displayed in the figure, with real ground truth above generated predictions.
A.4 BAIR
The appendix presents BAIR and Cityscapes prediction examples, UCF-101 prediction and generation examples, and a full BAIR evaluation table.
- BAIR: Table 11 reports BAIR results across conditioning on p past frames, predicting pr future frames, and varying the training block size k.
- BAIR: Additional BAIR-related text reports a 94 on only the first frames and 96 on all subsequences of test frames.
- BAIR: Figure 7 visualizes BAIR prediction from two conditioning frames to 28 future frames, comparing real ground truth with MCVD predictions.
- UCF-101: Figure 8 presents UCF-101 prediction with four conditioning frames and 16 predicted frames, while Figure 9 shows unconditional four-frame generation.
- Cityscapes: Figure 10 shows Cityscapes prediction conditioned on two frames and generating the next seven frames with the MCVD concat variant.
- More prediction examples are available on the project page.