Source-linked AI summary
Diffusion Models for Video Prediction and Infilling
Tobias Höppe, Arash Mehrjou, Stefan Bauer, Didrik Nielsen, Andrea Dittadi
TL;DR
Video prediction and infilling require temporally coherent generative models, yet diffusion models have not been extensively explored for video. RaMViD extends diffusion models with 3D convolutions and random-mask conditioning, achieving competitive or state-of-the-art results across video tasks while retaining a shared architecture for conditional and unconditional training.
Problem
Video prediction and infilling are important for intelligent agents, but diffusion models have not been extensively explored for video and infilling research remains limited.
Method
RaMViD extends diffusion models to videos with 3D convolutions and randomized masking that conditions the same architecture for prediction, infilling, and unconditional training.
Results
RaMViD achieves competitive performance across multiple datasets, including state-of-the-art video-prediction results and competitive unconditional video generation.
Takeaways & Limitations
Random-mask conditioning provides a simple, effective way to use one diffusion architecture across conditional video-completion tasks and unconditional generation.
Takeaways & Limitations
Performance varies across tasks and masking schemes, and the method focuses on diffusion-based video generation rather than optimizing per-frame quality or adding super-resolution.
Abstract
from arXiv · showhide
Predicting and anticipating future outcomes or reasoning about missing information in a sequence are critical skills for agents to be able to make intelligent decisions. This requires strong, temporally coherent generative capabilities. Diffusion models have shown remarkable success in several generative tasks, but have not been extensively explored in the video domain. We present Random-Mask Video Diffusion (RaMViD), which extends image diffusion models to videos using 3D convolutions, and introduces a new conditioning technique during training. By varying the mask we condition on, the model is able to perform video prediction, infilling, and upsampling. Due to our simple conditioning scheme, we can utilize the same architecture as used for unconditional training, which allows us to train the model in a conditional and unconditional fashion at the same time. We evaluate RaMViD on two benchmark datasets for video prediction, on which we achieve state-of-the-art results, and one for video generation. High-resolution videos are provided at https://sites.google.com/view/video-diffusion-prediction.
1 Introduction
Video prediction supports applications including autonomous driving, planning, and reinforcement learning, while video infilling remains comparatively underexplored. RaMViD extends diffusion models to videos with random masking, enabling prediction and infilling using one architecture.
- Video prediction can support autonomous driving, event anticipation, planning, and reinforcement learning.
- Video infilling generates missing frames from observed video and can support planning, trajectory estimation, video processing, action recognition, and pose estimation.
- Diffusion models have advanced static visual generation but have not been extensively studied for videos.
- RaMViD extends diffusion models to videos with 3D convolutions and randomized conditioning, supporting future-frame prediction and arbitrary-position infilling.
- RaMViD introduces a diffusion-based architecture for video prediction and infilling, a random-masking schedule, and competitive performance across multiple datasets.
2 Background and related work
Diffusion models learn to reverse a noise-adding stochastic process so they can generate samples from a complex data distribution. Related video work addresses prediction and infilling with deterministic, stochastic, GAN-based, and emerging diffusion-based approaches.
- Diffusion models: Diffusion models transform data into unstructured noise and learn the reverse process to recover the data-generating distribution.
- Diffusion models: The forward diffusion process moves data toward a simple noise distribution, while learning the score enables reversing this process to generate data.
- Diffusion models: Denoising Score Matching provides a tractable alternative to the otherwise intractable original score-matching objective.
- Video prediction and infilling: Video prediction may be deterministic or stochastic, but deterministic approaches can average future states and produce blurry predictions.
- Video prediction and infilling: Video infilling research is substantially scarcer than prediction research, with much prior work focusing on frame interpolation.
- Video prediction and infilling: Concurrent diffusion-based video studies address autoregressive prediction, unconditional generation, long-video prediction, or prediction and infilling with differing evaluations.
3 Random-Mask Video Diffusion
RaMViD conditions video diffusion by selectively masking frames, using the same architecture for prediction, completion, and mixed conditional–unconditional training. Randomized conditioning frames let the model generate unknown frames while preserving known-frame information.
- Randomization: The randomized mask supports video prediction and infilling with the same model by fixing known-frame indices and generating the unknown frames.The conditioning set can be any subset of the video sequence, allowing missing frames at arbitrary positions to be completed.
- Conditional training: RaMViD partitions each video into unknown frames U and conditioning frames C, then adds noise only to the unknown frames.The network receives a sequence containing noisy unknown frames and clean conditioning frames, and the loss is computed on the unknown portion.
- Conditional training: An unconditionally trained model produces predicted frames that do not harmonize well with conditioning frames, motivating conditional training with randomized masking.Re-sampling can improve harmonization but becomes computationally too expensive for videos, especially with few conditioning frames.
- Conditional training: Unlike standard conditional diffusion, RaMViD feeds the entire sequence to the network while conditioning through clean and noisy frame positions.The score output and loss are restricted to the unknown-frame indices rather than requiring concatenation or zero-padding to match dimensions.
- Randomization: During training, RaMViD randomly chooses the number and positions of conditioning frames, leaving those frames clean while diffusing the remaining frames.After training, any arbitrary subset of frame indices can be fixed as C, with the remaining indices generated as U.
- Randomization: RaMViD uses the exact architecture of unconditional models and mixes conditional and unconditional training by setting C = ∅ with probability pU.When C = ∅, the objective becomes the same as the unconditional-training objective.
4 Experiments
Experiments evaluate RaMViD for prediction, infilling, and unconditional generation across BAIR, Kinetics-600, and UCF-101. The model achieves strong prediction and completion results, while generation remains limited by dataset complexity and temporal degradation.
- Experimental setup: RaMViD is evaluated on BAIR, Kinetics-600, and UCF-101 using prediction, infilling, and unconditional-generation protocols.BAIR evaluates prediction and infilling; Kinetics-600 evaluates prediction and two completion tasks; UCF-101 evaluates unconditional generation.
- BAIR: On BAIR, all tested unconditional rates achieve state-of-the-art prediction, with pU > 0 improving performance until a tipping point.The fixed prediction-specific variant does not improve performance and is unreliable on other completion tasks.
- BAIR: Randomized masking lets the same BAIR models perform video infilling without retraining, with similar performance across pU values.Infilling conditions on the first and last frames and generates the 14 intervening frames.
- Kinetics-600: On Kinetics-600, RaMViD achieves state-of-the-art prediction results by a significant margin and produces temporally coherent outputs with detailed backgrounds.Performance improves with pU > 0 up to a tipping point, but fast movements can deform and pU = 0.75 causes a significant FVD drop.
- Kinetics-600: On Kinetics-600, RaMViD excels at completion from endpoint or evenly distributed conditioning frames, with the latter also supporting upsampling.The model is reported to fill missing frames with very high quality and coherence when conditioning frames are distributed evenly.
- UCF-101: On UCF-101, RaMViD generates scenes with static backgrounds and foreground actions, but actions can lack coherence, moving objects can deform, and the model overfits.The authors report competitive unconditional-generation performance but not state-of-the-art, attributing limitations partly to the dataset’s small size and complexity.
- Long-horizon generation: Autoregressive sampling can generate arbitrarily long videos, but quality often becomes poor after about 30 frames as errors accumulate.Each step conditions on the latest sampled frames, and sequence quality slightly deteriorates at every step.
5 Conclusion
RaMViD extends diffusion models to videos for prediction and completion tasks using simple randomized conditioning. The paper identifies masking-schedule selection, cross-domain completion, frame quality, and dataset-related deployment concerns as remaining boundaries.
- RaMViD extends diffusion models to videos and supports several video completion tasks through simple conditioning that requires no major architectural modification.The method also uses an alternative masking schedule intended to improve unconditional generation without sacrificing conditional performance.
- The method's performance varies across tasks and masking schemes, motivating research into task-specific masking schedules.
- Applying the conditioning technique to completion in other data domains remains an open direction.
- The work focuses on the diffusion algorithm rather than optimizing frame quality, making super-resolution integration a relevant future direction.
- The models require large video datasets, whose group distributions may be nonuniform and should be evaluated before deployment.
A Implementation details
RaMViD uses the same architecture as unconditional models while introducing conditioning through randomized masks and noise-corrupted-frame losses. The implementation uses 3D convolutions, fixed diffusion schedules, and dataset-specific sampling settings.
- RaMViD introduces conditioning frames without changing the architecture used for unconditional models.
- Training computes loss only for frames corrupted with noise, with c = 128 channels and video length l in the method sketch.
- BAIR unconditional samples range from pU = 0 to pU = 0.75, with detail quality increasing as pU increases.
- All models use 1000 diffusion steps for training, with 750 sampling steps on BAIR and 500 on Kinetics-600 and UCF-101.
- The implementation uses 3 × 3 × 3 convolution kernels, spatial downsampling to 8 × 8, and four-head attention at 16 and 8 resolutions.
B.1 Results on BAIR
RaMViD can generate unconditional BAIR videos across all tested masking probabilities. Higher pU improves object-detail generation, although no quantitative unconditional BAIR results are reported.
- All tested RaMViD models generate unconditional BAIR videos, including RaMViD with pU = 0.The authors attribute this to the dataset's low diversity.
- Models with higher pU generate finer details, and only pU ≥0.25 models properly generate the different objects in the box.
- No quantitative results are available for unconditional generation on BAIR.
B.2 Results on Kinetics-600
Kinetics-600 is the most difficult evaluated dataset and exposes failure cases despite state-of-the-art results. RaMViD reduces deformed objects relative to other methods, but fast object or camera movement remains problematic.
- Kinetics-600 is the most difficult dataset considered, although RaMViD achieves state-of-the-art results there.
- Fast-moving objects commonly become deformed in Kinetics-600 predictions.
- On Kinetics-600, the background is preserved well during fast-object prediction, but the moving object can become unrealistically deformed.
- Fast camera movement can make infilling fail to harmonize object motion, causing disappearance in one frame and appearance in another.
- RaMViD's main qualitative improvement is fewer deformed objects than other methods, with deformations mainly remaining for fast-moving objects.
D Compute
Training used 8 NVIDIA A100 GPUs, with dataset-specific batch sizes, iteration counts, and runtimes.
- Each model was trained on 8 NVIDIA A100 GPUs with 40 GB of memory.
- BAIR models used batch size 32 and micro-batch size 16 for 250k iterations, taking approximately 3 days.
- Kinetics-600 models used batch size 64 and micro-batch size 16 for 500k iterations, taking approximately 10 days.
- UCF-101 models used batch size 64 and micro-batch size 16 for 450k iterations, taking approximately 9 days.
E Datasets
The experiments use BAIR, Kinetics-600, and UCF-101, with random subsequences during training and dataset-specific evaluation procedures.
- Dataset preparation: Training samples random subsequences because every dataset contains more frames than the model's training sequence length.
- BAIR robot pushing: BAIR uses 64 × 64 robot-pushing videos, trains on sequences of length 20, and evaluates FVD across 100 runs on 256 test videos.
- Conditional tasks: Figures illustrate Kinetics-600 video infilling with observed frames C = {0, 1, 14, 15} and completion with C = {0, 5, 10, 15}.
- Qualitative generation: Unconditional UCF-101 generation performs well for scenes with less movement, but outputs are often close to the training set.
- Kinetics-600: Kinetics-600 videos are center-cropped to 64 × 64; evaluation predicts sequences for 50,000 test videos and trains on length-16 sequences.
- UCF-101: UCF-101 videos are resized and center-cropped to 64 × 64; evaluation samples 10,000 videos unconditionally and computes Inception Score three times.
F Sampling Speed
Sampling speed affects generation quality, with at least 500 DDPM sampling steps recommended, while qualitative comparisons vary by motion complexity and competing method.
- Sampling steps: An FVD of 14 / 16 is achieved with 750 / 500 sampling steps on Kinetics-600, but FVD rises to 49 with 250 steps.
- Sampling steps: RaMViD recommends a minimum of 500 DDPM sampling steps for video generation.
- Comparisons: RaMViD and VideoGPT appear visually similar, but VideoGPT produces artefacts especially when objects interact.
- Comparisons: RaMViD models motion better than CCVS, while both perform equally poorly for fast object or camera movement.
- Prior work: Ho et al. appears to outperform RaMViD on unconditional UCF-101 generation, whereas RaMViD reportedly outperforms Voleti et al. on BAIR.