Source-linked AI summary
VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation
Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, Tieniu Tan
TL;DR
VideoFusion targets the difficulty of generating temporally coherent videos with diffusion models that independently noise frames. It decomposes noise into shared base and time-varying residual components, and experiments report strong results across datasets, pretrained-image-DPM benefits, and text-conditioned video support.
Problem
Video diffusion is challenging because high-dimensional videos contain complex spatial-temporal correlations, while standard methods independently noise frames and ignore their temporal relationships.
Method
VideoFusion decomposes per-frame noise into a base noise shared by consecutive frames and a residual noise varying over time.
Results
VideoFusion achieves state-of-the-art results on different datasets and supports text-conditioned video creation, outperforming GAN-based and diffusion-based alternatives.
Takeaways & Limitations
The shared base noise can leverage pretrained image-generation DPM priors and provides a basis for controlling video content separately from motion.
Takeaways & Limitations
Sharing base noise can limit generated motions, and selecting suitable λ_i values for all videos is difficult because inter-frame differences vary.
Abstract
from arXiv · showhide
A diffusion probabilistic model (DPM), which constructs a forward diffusion process by gradually adding noise to data points and learns the reverse denoising process to generate new samples, has been shown to handle complex data distribution. Despite its recent success in image synthesis, applying DPMs to video generation is still challenging due to high-dimensional data spaces. Previous methods usually adopt a standard diffusion process, where frames in the same video clip are destroyed with independent noises, ignoring the content redundancy and temporal correlation. This work presents a decomposed diffusion process via resolving the per-frame noise into a base noise that is shared among all frames and a residual noise that varies along the time axis. The denoising pipeline employs two jointly-learned networks to match the noise decomposition accordingly. Experiments on various datasets confirm that our approach, termed as VideoFusion, surpasses both GAN-based and diffusion-based alternatives in high-quality video generation. We further show that our decomposed formulation can benefit from pre-trained image diffusion models and well-support text-conditioned video creation.
1. Introduction
VideoFusion addresses the difficulty of video diffusion by sharing base noise across consecutive frames while retaining time-varying residual noise. The approach also leverages pretrained image diffusion models and supports text-conditioned video creation.
- Video diffusion remains challenging because videos have high-dimensional data and complex spatial-temporal correlations.
- Independent per-frame noises destroy temporal correlations and force the denoising network to reconstruct coherent frames while modeling spatial and temporal relationships.
- VideoFusion decomposes each frame’s noise into shared base noise and time-varying residual noise, producing correlated noised latents that ease coherent-video reconstruction.
- A DALL-E 2 decoder experiment shows that shared base noise yields roughly correlated image sequences under the same condition, unlike independent noises.
- VideoFusion achieves state-of-the-art results across datasets and supports text-conditioned video creation.
2. Related Works
Diffusion probabilistic models learn generation through iterative denoising after progressively corrupting data with noise. Related video methods include GANs, autoregressive approaches, and diffusion models that often treat frames independently.
- DPMs combine a gradual noise-adding diffusion process with an iterative denoising process that generates new samples.
- DPMs have been applied to image generation, super-resolution, and image translation, and are easier to train and more diverse than GANs.
- Video-generation methods include GANs that learn joint frame distributions and decomposed models that separately represent content and motion.
- Recent video diffusion methods extend image denoising networks to 3D or condition denoising on known frames, but commonly treat video frames as independent diffusion samples.
3. Decomposed Diffusion Probabilistic Model
VideoFusion replaces independent per-frame diffusion noise with shared base noise plus frame-varying residual noise, aligning the diffusion process with video redundancy. Its two-generator denoising pipeline uses a pretrained image diffusion model for the base component and a residual generator for frame differences.
- Motivation: Standard video diffusion independently noises frames, destroying temporal correlations and forcing denoising networks to reconstruct coherent videos from unrelated noise samples.This increases the burden of jointly modeling spatial and temporal correlations.
- Decomposed Diffusion: VideoFusion decomposes each frame into a shared base component and a residual, then decomposes its noise into shared base noise and frame-specific residual noise.The shared component makes noised latent variables correlated across frames.
- Base Frame: The method uses the middle frame as the base frame, with zero residual, so one denoising pass can estimate base noise for all video frames.The base frame is set to x⌊N/2⌋ and λ⌊N/2⌋=1.
- Pretrained Image DPM: VideoFusion assigns a large pretrained image-DPM-based base generator to the difficult base-frame reconstruction and a smaller residual generator to frame differences.The reported configuration uses up to 2-billion parameters for the base generator and 0.5-billion parameters for the residual generator.
- Joint Training: Fixing the pretrained base generator performs poorly because of the image–video domain gap, so the base generator is finetuned jointly with a small learning rate.Stop-gradient prevents non-base-frame losses from updating the pretrained base generator, reducing the risk that noisy residual estimates damage it.
- Denoising Pipeline: At each DDIM or DDPM denoising step, the base generator removes shared base noise before the residual generator estimates the remaining noise for each frame.The residual generator is conditioned on frame number and operates on a less-noisy latent after base-noise removal.
- Discussion: Keeping the shared base noise fixed can preserve video content while changing motions, and explicit residual-noise guidance can associate residual noise with motion.The authors connect this control to generating longer coherent sequences.
4. Experiments
Experiments evaluate VideoFusion across multiple video datasets, conditioning settings, qualitative comparisons, efficiency tests, ablations, long-sequence generation, and motion-content decomposition. Results show strong generation quality, reduced inference cost, benefits from pretraining and joint finetuning, and improved coherence for extended videos.
- Experimental Setup: VideoFusion is evaluated on UCF101, Sky Time-lapse, TaiChi-HD, and WebVid-10M using unconditional, class-conditioned, and text-conditioned generation settings.Quantitative metrics include FVD, KVD, and IS; standard evaluations use 16-frame videos at 128 × 128 resolution.
- Quantitative Results: VideoFusion achieves the best quantitative results on UCF101 and outperforms the DPM-based VDM and GAN-based alternatives.Increasing resolution improves IS but worsens FVD, possibly because higher-resolution videos contain richer details and are harder to learn.
- Qualitative Results: VideoFusion generates more realistic videos with richer details than TATS and DIGAN in visual comparisons.The comparison displays the 4th, 8th, 12th, and 16th frames of 16-frame, 128 × 128 videos.
- Efficiency Comparison: VideoFusion reduces inference memory by 21.8% and latency by 57.5% relative to VDM.The shared base noise requires only one forward pass of the base generator, while the pretrained base generator permits a smaller residual generator.
- Ablation Study: The sharing coefficient λi must be balanced: values of 0.1 or 0.75 worsen UCF101 performance by under-sharing temporal information or restricting dynamics.A small λi shares less base noise, whereas a large λi assumes frames share most of their information.
- Ablation Study: Pretraining improves performance, but VideoFusion still surpasses VDM without pretrained initialization, supporting contributions from both image priors and decomposed diffusion.The pretrained image priors ease learning image content and help the model focus on temporal correlations.
- Ablation Study: Jointly finetuning the pretrained base generator improves performance, while removing stop-gradient worsens it because the residual generator can damage the pretrained model early in training.Fixing the pretrained generator performs poorly because of the domain gap between Laion-5B and UCF101.
- Generating Long Sequences: Keeping base noise fixed helps VideoFusion extend 16-frame videos to 512 frames while maintaining quality and coherence.This addresses the difficulty of preserving content coherence in autoregressively extended video frames.
5. Limitations and Future Work
VideoFusion’s shared base noise can improve temporal correlation but may restrict motion, and a single similarity setting may not suit all videos. The current text-conditioning design also leaves long-text video generation as an open challenge.
- Sharing base noise helps exploit temporal correlation but may limit motions in generated videos.
- A fixed λ_i is difficult to choose across videos because frame differences can be small or large.
- Future work will adaptively generate λ_i for each video and potentially each frame.
- Long-text video generation is difficult because DALL-E 2’s latent embedding may not encode long temporal caption information, while direct text conditioning burdens residual-generator training.
6. Conclusion
VideoFusion decomposes diffusion noise into shared base and time-varying residual components, producing correlated noise sequences for video frames. It also uses a pretrained image-generation DPM to estimate shared base noise, and reports competitive quantitative and qualitative results.
- VideoFusion decomposes standard diffusion into shared base noise and time-varying residual noise for consecutive frames.
- Correlated noise sequences may make it easier for the denoising network to reconstruct coherent videos.
- A pretrained image-generation DPM estimates base noise for all frames with one forward pass, leveraging its priors efficiently.
- Quantitative and qualitative results show VideoFusion produces results competitive with state-of-the-art methods.
A. DDPM Sampling of VideoFusion
During VideoFusion sampling, each added noise sample is decomposed into a frame-shared base component and a residual component varying along time.
- Each sampling step resolves added noise into base and residual components, sharing the base across frames while varying the residual over time.
- The sampling design preserves the same noise decomposition used by VideoFusion’s diffusion process.
- Residual noise varies along the time axis during sampling.
B. Details about VideoFusion
VideoFusion uses base and residual U-shaped generators, with smaller models for several datasets and larger models for WebVid-10M. The WebVid-10M setup includes a 2.00-billion-parameter pretrained base generator.
- Architecture: The base generator and residual generator are both U-shaped networks.
- Dataset settings: Experiments on UCF101, Sky Time-lapse, and TaiChi-HD use relatively smaller models.
- Dataset settings: Experiments on WebVid-10M use relatively larger models.
- Pretraining: 2.00 billion parameters: WebVid-10M uses a large pretrained base generator.
- Configuration references: Table 8 documents generator details for UCF101, Sky Time-lapse, and TaiChi-HD, while Table 9 documents WebVid-10M details.