Source-linked AI summary
Flexible Diffusion Modeling of Long Videos
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, Frank Wood
TL;DR
Long-duration photorealistic video generation must handle increasing computational demands and long-range temporal dependencies. FDM uses a DDPM-based model that flexibly samples arbitrary frame subsets conditioned on other subsets, enabling exploration and optimization of resource-constrained sampling schedules. The paper reports improved long-range video modeling over prior work and samples temporally coherent videos exceeding 25 minutes, alongside a CARLA-based dataset and semantically meaningful metrics.
Problem
Long photorealistic videos remain difficult to generate because computation scales with frame count and frames may depend on arbitrarily distant earlier frames.
Method
FDM trains a DDPM-based video model to condition on and sample arbitrary subsets of frames, using a broad distribution of resource-constrained training tasks.
Results
FDM improves long-duration video modeling over prior work across several tasks and supports sampling temporally coherent videos exceeding 25 minutes.
Takeaways & Limitations
Flexible post-training sampling-scheme exploration enables dataset-dependent choices, with at least one FDM scheme outperforming VDM on each dataset.
Takeaways & Limitations
Sampling is slow: generating a 300-frame video takes approximately 16 minutes on a GPU.
Abstract
from arXiv · showhide
We present a framework for video modeling based on denoising diffusion probabilistic models that produces long-duration video completions in a variety of realistic environments. We introduce a generative model that can at test-time sample any arbitrary subset of video frames conditioned on any other subset and present an architecture adapted for this purpose. Doing so allows us to efficiently compare and optimize a variety of schedules for the order in which frames in a long video are sampled and use selective sparse and long-range conditioning on previously sampled frames. We demonstrate improved video modeling over prior work on a number of datasets and sample temporally coherent videos over 25 minutes in length. We additionally release a new video modeling dataset and semantically meaningful metrics based on videos generated in the CARLA autonomous driving simulator.
1 Introduction
Long, coherent photorealistic video generation remains difficult because resource demands and long-range dependencies grow with video length. FDM addresses this by flexibly conditioning and marginalizing video frames, enabling exploration of resource-constrained sampling schemes.
- Motivation: Long photorealistic videos remain an open challenge because memory and processing requirements scale with frame count.Long-range coherence further complicates generation because frames may depend statistically on arbitrarily distant earlier frames.
- Motivation: Fixed-lag autoregressive models impose unrealistic conditional independence assumptions that can impair long-range coherence.They treat the next frame as independent of frames beyond the chosen autoregressive lag.
- Research question: FDM asks how to allocate a jointly modeled limit K across videos of length N > K.The framework embraces finite-architecture constraints rather than assuming all frames can be modeled together.
- Approach: FDM uses a DDPM-based architecture that can sample any subset of frames conditioned on any other subset.This flexibility supports unconditional generation, video completion, and videos of different lengths.
- Contributions: The paper reports improvements over prior work on long-range video modeling and introduces a CARLA dataset with a semantically meaningful performance metric.These contributions accompany the flexible DDPM-based video model and its sampling-scheme exploration.
2 Sampling long videos
FDM generates long videos through sequential sampling schemes that allocate limited frame access across stages. Its flexible conditioning makes diverse schemes possible and enables dataset-dependent optimization for improved video quality.
- Sequential generation: Sequential sampling is required to generate arbitrarily long videos when the model can access only a small number of frames at once.An autoregressive example samples three frames conditioned on four preceding frames across seven stages for a 30-frame completion.
- Sequential generation: Autoregressive sampling is simple but assumes later frames are conditionally independent of earlier frames beyond the conditioning set.Sampling every second frame before infilling can partially ameliorate this restriction.
- Sampling schemes: A sampling scheme specifies frame-index vectors X_s to sample and Y_s to condition on at each stage.The procedure uses these stage-wise index sets to organize video completion.
- Sampling schemes: FDM is trained to sample arbitrarily chosen frame subsets conditioned on any other subset, rather than only consecutive or regularly spaced sequences.Valid schemes must respect |X_s| + |Y_s| ≤ K, sample every frame, and never condition on an unsampled frame.
- Optimization: Dataset-dependent optimization of sampling schemes improves video quality as measured by Fréchet Video Distance among other metrics.The best scheme varies by dataset, and optimized schemes improve the best Table 1 results on two of three datasets.
3 A review of conditional denoising diffusion probabilistic models
Conditional DDPMs learn to reverse a gradual diffusion process, generating video frames from noise while conditioning on observed frames. In this framework, variable frame subsets and conditioning inputs support flexible video-generation tasks.
- Conditional DDPM formulation: DDPMs represent video data as a 4-dimensional tensor and learn the probabilistic inverse of a process that transforms clean frames into noise.The diffusion process uses timesteps from 0 to T, with x0 denoting the data tensor.
- Conditional DDPM formulation: The forward diffusion process adds small amounts of Gaussian noise through transitions q(x_t|x_{t−1}) parameterized by α_t values slightly below 1.The α_t hyperparameters keep noise additions small at each timestep.
- Conditional DDPM formulation: To generate samples, the model draws x_T from a unit Gaussian and iteratively samples x_{t−1} from learned reverse transitions conditioned on x_t and y.The neural network parameterizes p_θ(x_{t−1}|x_t,y) as an approximation to the corresponding reverse diffusion distribution.
- Conditional DDPM formulation: The denoising network predicts the noise term ε_θ(x_t,y,t), which determines the mean of each Gaussian reverse transition.The U-net computes this prediction at every DDPM step while conditioning on observed frames.
- Flexible conditioning: Training samples variable latent and observed frame subsets so the model can adapt to different numbers and arrangements of represented frames.Frame-index information in X and Y is supplied to the network to support interactions between variable frame subsets.
4 Training procedure and architecture
The training procedure samples diverse latent and observed frame-index configurations under a computational limit, while the architecture extends image DDPMs with temporal processing. Test-time sampling schemes can be changed and optimized without retraining.
- Training procedure: Training samples latent and observed index vectors X and Y from a broad task distribution with |X| + |Y| bounded by K.Each task extracts the corresponding frames from a full training video, covering many plausible conditional-generation cases.
- Architecture: The video architecture extends a U-net with a frame dimension and uses temporal attention with a novel relative frame-position encoding network.The input is formed by concatenating the noisy latent frames and observed frames.
- Architecture: Additional data are processed instead of zero-padding variable-length index vectors, reducing wasted computation and providing a lower-variance loss estimate.The padding strategy is designed for batches whose sampled index-vector sizes vary up to K.
- Sampling schemes: FDM supports multiple test-time sampling schemes, whose relative performance is dataset-dependent and can be optimized without retraining.The optimized schemes select conditioning frames greedily using estimates from 100 training videos and 10 evenly spaced diffusion timesteps.
5 CARLA Town01 Dataset
The CARLA Town01 dataset provides long simulated driving videos and an interpretable benchmark for video-completion quality. A frame-to-position regressor enables route-based evaluation of sampled videos.
- Dataset: The dataset contains 408 training and 100 test videos, each 1000 frames long at 128 × 128 resolution, generated in CARLA’s Town01.Because all videos use one small town, expressive models can be evaluated through recognizable routes and memorized town layout.
- Evaluation benchmark: A neural-network regressor maps each rendered frame to the car’s (x, y) position, enabling sampled video trajectories to be plotted and quantitatively evaluated.The regressed coordinates are used to visualize routes and derive semantically meaningful measures of validity.
- Qualitative behavior: FDM completions usually preserve town landmarks with high fidelity, but some samples jump to an incorrect location for a single frame.Figure 5 visualizes observed-frame coordinates in red and sampled-frame coordinates in blue.
- Evaluation benchmark: The benchmark measures speed distributions and route validity, including the percentage of point-speed estimates exceeding 10m/s after positions are sampled ten frames apart.The simulator’s maximum speed is 3m/s, so unusually large estimates indicate occasional jumps between disparate town locations.
6 Experiments
Experiments evaluate flexible diffusion video completion across three datasets, comparing sampling schemes, baselines, offline optimization, and training-task ablations. Results show dataset-dependent sampling behavior and benefits from flexible task and scheme selection.
- Experimental setup: Video completion is evaluated on GQN-Mazes, MineRL, and CARLA Town01 using FDM with K = 20 represented frames.The datasets contain videos of 300, 500, and 1000 frames, respectively; evaluation conditions on the first 36 frames.
- Sampling schemes: Sampling-scheme performance varies significantly by dataset: hierarchical schemes achieve better FVDs on GQN-Mazes, autoregressive schemes tend to perform best on MineRL, and hierarchical schemes sometimes lose coherence on CARLA Town01.The reported behaviors are associated with different temporal dependency structures and, for CARLA, higher outlier percentages.
- Baselines: FDM has at least one sampling scheme that outperforms VDM on each dataset, although no FDM scheme consistently outperforms VDM.Both diffusion-based methods achieve significantly higher FVD scores than TATS and CWVAE.
- Optimized sampling schemes: Offline optimization of observed frame indices improves the best results from Table 1 on two of the three datasets.Table 2 evaluates dataset-specific optimized sampling schemes using FVD.
- Ablations: Replacing the structured training-task distribution with uniform sampling worsens FVD on all five tested schemes for both GQN-Mazes and MineRL.The average FVD reduction from the alternative distribution is 31% on GQN-Mazes and 52% on MineRL.
7 Related work
Related work includes conditional DDPMs that adapt unconditional sampling procedures and VAE-based approaches for video modeling. The paper positions explicit conditional DDPM training and flexible sparse-frame modeling within this landscape.
- Conditional diffusion models: Some conditional video models adapt unconditional DDPM sampling and therefore require approximations, whereas this paper explicitly trains a conditional DDPM.The paper states that further comparison of these approaches is outside its focus on modeling a small portion of video frames at a time.
- VAE-based video models: Prior VAE-based video models use autoregressive prediction with global or per-frame latent variables and convolutional LSTM architectures.These approaches differ in whether the prior is learned and whether latent variables are shared across frames.
8 Discussion
The method targets coherent photorealistic videos under finite computational resources, but diffusion sampling remains slow. The training datasets also encode action-generation policies that shape their trajectories.
- Discussion: The proposed method generates photorealistic videos with long-range coherence while respecting fixed, finite computational resources.The discussion describes this as an empirically explored method for long-duration video generation.
- Limitations: Approximately 16 minutes are required to generate a 300-frame video on a GPU.The paper identifies reducing diffusion integration steps as a possible direction for faster sampling.
- Dataset assumptions: MineRL and CARLA videos are generated from action policies, including agents seeking a goal block in Minecraft and a controller driving the CARLA camera.These policies causally produce the frame-to-frame changes in camera pose.
Checklist
The checklist reports that the paper addresses contributions, limitations, societal impacts, reproducibility, training details, error bars, compute, asset licensing, and data-consent considerations as applicable.
- Research and ethics: The paper reports discussing its contributions and scope, limitations, potential negative societal impacts, and compliance with ethics-review guidelines.Each corresponding checklist item is marked Yes.
- Reproducibility: The paper reports that code, data, and reproduction instructions, training details, and error bars were included.The checklist marks these experimental-reporting items Yes.
- Compute: The paper reports specifying total compute and resource types in the appendix.The corresponding checklist item is marked Yes.
- Assets and data: The paper reports citing existing assets, mentioning their licenses, and including new assets through supplemental material or a URL.The checklist marks each asset-reporting item Yes.
- Applicability: The checklist marks questions about personally identifiable information, offensive content, and human-subject procedures as not applicable.These items include consent, participant risks, institutional review, and compensation.
A Experimental details
The experiments evaluate FDM and baselines across video-completion metrics, training configurations, task-distribution ablations, and temporal-attention architectures. The proposed relative-position encoding network addresses long-video position representation while reducing parameters relative to lookup tables.
- Evaluation protocol: Metrics are computed over the first 100 test videos for GQN-Mazes and MineRL, while CARLA Town01 has a 100-video test set.The evaluation is limited by the expense of drawing samples from FDM and baselines.
- Evaluation metrics: Quantitative metrics include DDPM test Loss, LPIPS, SSIM, and PSNR, with lower Loss and LPIPS preferred and higher SSIM and PSNR preferred.The authors caution that SSIM and PSNR may favor blurry samples over realistic ones.
- Training-task ablation: The training-task-distribution ablation finds that the proposed structured distribution achieves better FVD than the uniform ablation across every tested dataset and sampling-scheme combination.The compared networks use the same hyperparameters, so FVD differences from Table 1 should not be conflated with this ablation.
- Relative position encodings: The relative-position encoding network represents p_ij as f_RPE(d_ij), where d_ij is the difference between frames’ positions in the full video.This replaces a lookup table for relative frame positions in temporal attention.
- Relative position encodings: For long videos and small K, the lookup table can be parameter-intensive and receive sparse learning signals because relative-position differences span the video length.The proposed network’s computation was not measurably slower than a lookup table in the reported experiments.
- Relative position encodings: After convergence, the RPE network showed no noticeable sample-quality difference from lookup tables while using 9.8 million fewer parameters.Early in training, the RPE network learned the relative importance of other frames more quickly.
C Explanation of our training task distribution
FDM trains on a broad but structured distribution of conditional frame-generation tasks, designed to support diverse test-time sampling schemes. The paper also describes hierarchy-based and optimized procedures for selecting conditioning frames.
- Training task distribution: The training distribution randomly samples latent and observed frame indices so the model can support many plausible test-time conditioning tasks.The distribution is broad enough to cover feasible frame choices while retaining structure for learning.
- Training task distribution: Frame spacing is sampled across multiple timescales with a log-uniform distribution to represent events occurring over seconds, minutes, or hours.The distribution parameters are chosen as broadly as the video length and frame rate allow.
- Training task distribution: The algorithm enforces a maximum of K sampled or observed frames, but the authors describe this mechanism as a design choice rather than a necessary or optimal component.
- Optimized sampling schemes: The optimized sampling procedure selects conditioning indices sequentially, beginning with neighboring observed or previously sampled frames around each latent index.At each iteration it estimates DDPM loss for every possible next conditioning index.
F Sampled videos
The paper presents long CARLA trajectories and qualitative completions and unconditional samples across GQN-Mazes, MineRL, and CARLA Town01. Hierarchy-2 avoids a long-range consistency failure observed in Autoreg on GQN-Mazes.
- Long video samples: FDM produces long CARLA Town01 samples lasting 30–40 minutes, alongside video completions and unconditional generations across the evaluated datasets.
- Video completions: Hierarchy-2 completions are shown for GQN-Mazes, MineRL, and CARLA Town01 after observing the first 36 frames.
- Video completions: Autoreg often introduces more than the two wall/floor colors expected within each GQN-Mazes video because it cannot track long-range dependencies.The caption contrasts this failure with Hierarchy-2 samples, where the issue is not seen.
- Unconditional generation: Unconditional FDM samples using Hierarchy-2 are shown for each dataset.