Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models

Songwei Ge; Seungjun Nah; Guilin Liu; Tyler Poon; Andrew Tao; Bryan Catanzaro; David Jacobs; Jia-Bin Huang; Ming-Yu Liu; Yogesh Balaji

Source-linked AI summary

Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models

Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming-Yu Liu, Yogesh Balaji

arXiv:2305.10474v3May 17, 2023cs.CVcs.GRcs.LG

TL;DR

Video diffusion requires photorealistic, temporally coherent sequences, but large-scale video data and training resources are limited. PYoCo finetunes a pretrained image diffusion model using a noise prior designed to preserve video-frame correlations, achieving state-of-the-art results on UCF-101 and MSR-VTT.

Problem
Large-scale text-to-video training is constrained by scarce billion-scale video datasets and substantially higher computational demands than image diffusion.
Method
PYoCo finetunes a pretrained text-to-image diffusion model with a video noise prior that models correlations among frames progressively along the temporal axis.
Results
PYoCo achieves state-of-the-art zero-shot text-to-video results on several benchmarks, including UCF-101 and MSR-VTT, while improving UCF-101 Inception Score from 33.45 to 47.76.
Takeaways & Limitations
A correlation-aware noise prior provides an efficient route to large-scale text-to-video generation by reusing a pretrained image diffusion model.
Takeaways & Limitations
The correlation parameter α trades temporal variation against shared noise: α →∞ produces frozen videos, while α = 0 recovers i.i.d. noise.

Abstract

from arXiv · show

Despite tremendous progress in generating high-quality images using diffusion models, synthesizing a sequence of animated frames that are both photorealistic and temporally coherent is still in its infancy. While off-the-shelf billion-scale datasets for image generation are available, collecting similar video data of the same scale is still challenging. Also, training a video diffusion model is computationally much more expensive than its image counterpart. In this work, we explore finetuning a pretrained image diffusion model with video data as a practical solution for the video synthesis task. We find that naively extending the image noise prior to video noise prior in video diffusion leads to sub-optimal performance. Our carefully designed video noise prior leads to substantially better performance. Extensive experimental validation shows that our model, Preserve Your Own Correlation (PYoCo), attains SOTA zero-shot text-to-video results on the UCF-101 and MSR-VTT benchmarks. It also achieves SOTA video generation quality on the small-scale UCF-101 benchmark with a $10\times$ smaller model using significantly less computation than the prior art.

1. Introduction

Video diffusion models must generate photorealistic frames with coherent temporal dynamics, but training them from scratch requires substantial data and computation. PYoCo finetunes a pretrained image diffusion model with a video-specific noise prior that preserves temporal correlations.

Motivation: Training large-scale video diffusion models from scratch is expensive because they must learn both photorealistic frames and coherent temporal dynamics.Prior approaches require hundreds of high-end GPUs or TPUs and several weeks of training.
Approach: Finetuning pretrained text-to-image models can transfer image-synthesis knowledge to video generation and improve training efficiency.
Noise Prior: Naively extending i.i.d. image noise to videos fails to model natural temporal correlations among frames from the same video.The authors visualize these correlations using noise maps obtained by reversing frames through a pretrained text-to-image model.
Contributions: The resulting model achieves state-of-the-art video generation performance on several benchmarks, with strong photorealism and temporal consistency.
Noise Prior: PYoCo injects correlations progressively along the temporal axis while finetuning a pretrained image diffusion model for text-to-video generation.

2. Related Work

Related work spans diffusion-based image and video generation, alongside strategies that reuse image-model knowledge to address the scarcity of large-scale video data. Prior video diffusion systems commonly train from scratch or combine image and video data.

Diffusion Models: Diffusion models have recently become a major approach for photorealistic text-to-image generation and have motivated analogous video-generation research.
Video Generation Models: Video-generation research has explored GANs, autoregressive models, implicit neural representations, and diffusion models.
Image Knowledge Transfer: Because video data are more limited in style, volume, and quality than image data, prior work leverages image datasets for text-to-video generation.

3. Preliminaries

The preliminaries describe diffusion denoising for noisy videos, EDM-based training, preconditioning, and ODE-based sampling. A denoiser receives a noisy video, text embedding, and noise level, then estimates the clean video.

Diffusion Formulation: The denoising network D processes a noisy video x, text embedding e, and noise level σ to estimate the clean video.The video has shape x ∈ R^(b×n_s×3×h×w), where n_s is sequence length and h × w is spatial resolution.
Training: EDM training corrupts a clean video with Gaussian noise σϵ and optimizes a σ-dependent weighted denoising objective.The loss weight is λ(σ) = (σ^2 + σ_data^2)/(σ · σ_data)^2, and σ follows a log-normal distribution.
Preconditioning: EDM preconditioning scales the denoiser inputs and outputs through explicit terms around the neural network F_θ.
Sampling: Sampling starts from a noise prior and solves a reverse-time ODE from σ = σ_max to σ = 0.The paper uses DEIS and its stochastic variant for synthesis.

4. Method

The method finetunes an image diffusion model for video while modifying the noise prior to preserve correlations observed across frames. It combines correlated-noise strategies with a cascaded architecture and joint image-video training.

Correlated noise model: Video-frame noise recovered through a pretrained image model is more similar within the same video than across different videos.This motivates preserving inter-frame correlation during video-model finetuning.
Correlated noise model: The proposed correlated-noise process addresses the mismatch between i.i.d. training noise and correlations present in reversed video-frame noise.The paper investigates mixed and progressive noising to preserve this correlation.
Mixed noise model: Mixed noising combines a noise vector shared across all frames with independent per-frame noise.The final noise is a linear combination of the shared and individual vectors.
Progressive noise model: Progressive noising generates each frame’s noise autoregressively by perturbing the noise from the preceding frame.The independent noise for frame i is used in the progressive formulation.
Correlation control: α controls inter-frame noise correlation: larger values increase correlation, α = 0 gives i.i.d. noise, and α →∞ produces frozen videos.The correlation parameter is used in both mixed and progressive noise models.
Noise validation: Progressive noise produces clustering patterns more similar to real video-frame noise than the independent alternative.The comparison is shown in Figure 2b against pretrained-image-model embeddings of real video-frame noise.
Model architecture: The model uses four cascaded networks: a base network, temporal interpolation network, and two spatial super-resolution networks.The cascade progresses from 16 × 64 × 64 frames with frameskip 5 to 76 × 1024 × 1024 video.
Training strategy: Training jointly finetunes on video and image data by concatenating them along the temporal axis and applying temporal modules only to video.The model uses T5 and CLIP text embeddings, each independently dropped at random during training.

5. Experiments

Experiments evaluate the proposed video noise priors across small-scale unconditional and large-scale zero-shot text-to-video settings. PYoCo improves benchmark performance, preserves image-model quality, and benefits from appropriately chosen noise correlation.

Experimental Setup: The experiments cover UCF-101 ablations and large-scale zero-shot text-to-video evaluation on UCF-101 and MSR-VTT.The study uses IS and FVD for small-scale experiments and evaluates large-scale generation on two zero-shot benchmarks.
Large-scale Text-to-Video Synthesis: 47.76 improves the zero-shot Inception Score from 33.45, with PYoCo outperforming all baselines on UCF-101.On MSR-VTT, the baseline configuration reaches a CLIP-FID of 10.21, while an ensemble further improves CLIP-FID and FID.
Small-scale Unconditional Video Synthesis: PYoCo achieves state-of-the-art unconditional UCF-101 generation with a model approximately 10× smaller and training time reduced from 925 to 75 GPU-days.The comparison is against a previous diffusion-based unconditional generation model.
Training Strategies: Finetuning an image diffusion model outperforms training from scratch, while correlated noise outperforms independent noise and yields lower frame FID.Correlated noise also better preserves image quality learned by the pretrained model.
Correlation Ratio: Too-small or too-large α degrades quality or motion, while α = 1 and α = 2 work best for mixed and progressive noising, respectively.The correlation ratio controls the correlation between noise across frames; larger α injects more correlation.
Model Size: Mixed and progressive noising consistently outperform training from scratch across model sizes, with progressive noising and α = 2 selected for large-scale experiments.The two correlated-noise strategies provide similar performance in the model-size ablation.

6. Conclusion

The paper proposes an efficient text-to-video training approach based on correlated noise priors for sequential frame generation. Applying progressive noising while finetuning a pretrained text-to-image model yields strong video quality and benchmark scores.

6. Conclusion: The authors observe clustered noise maps across video frames and propose mixed and progressive noise priors for sequential frame generation.These priors are designed as an efficient way to train text-to-video generation models.
6. Conclusion: Progressive noising is applied to finetune a state-of-the-art text-to-image diffusion model into a state-of-the-art large-scale text-to-video model.The reported generated-video quality and Inception and FID scores support the approach.

A. Experimental Setups

This section states that it provides additional experimental details covering implementation, datasets, evaluation, models, and training.

A. Experimental Setups: The experimental appendix covers implementation, dataset, evaluation, model, and training details.

A.1. Implementation details

The image-based U-Net is adapted for video by adding a temporal convolution dimension and decomposing attention into spatial and temporal components.

A.1. Implementation details: The 2D convolution layers are converted to 3D by adding a temporal dimension of size one.For example, a 3 × 3 convolution becomes a 1 × 3 × 3 convolution.
A.1. Implementation details: Attention layers are replaced with cascaded spatial and temporal attention layers, reusing spatial attention from eDiff-I.

A.2. Dataset and evaluation details

The evaluation uses caption templates and rewritten UCF-101 prompts to describe video categories more completely than raw class labels. The appendix provides category-specific examples and a copy-paste-friendly prompt list.

Caption templates for categorical video datasets: Caption templates combine category names with subjects and actions, including men, women, kids, and groups of people.The listed templates use forms such as “a man is [class]” and “a group of people are doing [class].”
Caption templates for categorical video datasets: The templates also vary whether the subject is performing an action, using constructions such as “is [class]” and “is doing [class].”Separate templates are provided for men, women, kids, and groups performing the category activity.
Prompts used for UCF-101 evaluation: For UCF-101, the authors replace original class labels with one-sentence captions because labels such as “jump rope” may describe an object rather than the complete video.The rewritten prompts are used to evaluate text-to-video generation models on the standard UCF-101 benchmark.
Prompts used for UCF-101 evaluation: The evaluation prompt list covers many activities, including makeup application, archery, sports, musical performances, and gymnastics.The appendix continues the list across multiple passages and provides a copy-paste-friendly version in a Google Spreadsheet.

A.3. Training details

Training uses separate optimization settings for image pretraining and video finetuning on UCF-101, while large-scale experiment hyperparameters are supplied in a table. Sampling uses stochastic DEIS variants with specified orders and step counts.

UCF-101 experiments: Image pretraining uses ADAM with a 2e−4 base learning rate, while video finetuning uses ADAM with a 1e−4 base learning rate.Both phases use a linear warmup of 5,000 steps.
UCF-101 experiments: UCF-101 sampling uses a stochastic DEIS sampler with 3kutta, order 6, and 25 steps.These settings are reported for the UCF-101 experiments.
Large-scale experiments: The hyperparameters for the large-scale text-to-video experiments are provided in Table F.The cited passage points to the table rather than listing the settings directly.

A.4. Architecture details

The appendix specifies architectures and optimization settings for small-scale UCF-101 and large-scale text-to-video experiments. It distinguishes model sizes and assigns separate architectural descriptions to the base, temporal interpolation, and spatial super-resolution models.

Large-scale architectures: The large-scale architecture descriptions cover the base model, temporal interpolation model, and two spatial super-resolution stacks.The relevant architectures are provided in Tables J, K, L, and M; Tables L and M describe spatial super-resolution models.
Large-scale experiment settings: Large-scale experiments use AdamW with learning rate 0.0001, weight decay 0.01, betas (0.9, 0.999), and EMA 0.9999.The configuration also enables gradient checkpointing and specifies 150K base-model and 220K super-resolution-model iterations.
Large-scale experiment settings: The large-scale base model uses stochastic DEIS with 3kutta, order 3, and 60 steps, while super-resolution models use DEIS with order 3 and 20 steps.The sampler settings are listed separately for the base and super-resolution models.
UCF-101 architectures: The small UCF-101 models include 69M-, 112M-, and 253M-parameter architectures.These correspond to the small, medium, and large UCF-101 configurations.
UCF-101 architectures: The small and medium UCF-101 models use two residual blocks, 128 channels, dropout 0.1, and 64 channels in attention heads.Both configurations specify spatial self-attention, spatial cross-attention, and temporal attention at resolutions [32] [16] [8], with scale-shift normalization enabled.
UCF-101 architectures: The small UCF-101 model uses channel multiplier [1] [2] [3], whereas the medium model uses [1] [2] [3] [4].The remaining listed attention and normalization settings are shared in the cited architecture descriptions.