Source-linked AI summary
NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation
Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang, Xiaodong Wang, Minheng Ni, Zhengyuan Yang, Linjie Li, Shuguang Liu, Fan Yang, Jianlong Fu, Gong Ming, Lijuan Wang, Zicheng Liu, Houqiang Li, Nan Duan
TL;DR
Long-video generation is constrained by the gap between short-video training and long-video inference, as well as inefficient sequential generation. NUWA-XL uses a coarse-to-fine Diffusion over Diffusion architecture with global keyframes and recursive local refinement, enabling direct long-video training and parallel inference. It reports a 94.26% inference-speed improvement when generating 1024 frames and introduces FlintstonesHD as a benchmark.
Problem
Existing long-video approaches train on short videos and generate segments sequentially, creating a training-inference gap and inefficient inference.
Method
NUWA-XL generates global keyframes as a coarse storyline, then recursively fills the intervening frames with local diffusion models in parallel.
Results
94.26% faster inference is reported when generating 1024 frames, alongside long-video generation with long-term coherence and realistic shot changes.
Takeaways & Limitations
The coarse-to-fine design supports direct training on 3376-frame videos, parallel long-video inference, and evaluation using the FlintstonesHD benchmark.
Takeaways & Limitations
Validation is limited to the public Flintstones cartoon because open-domain long-video datasets were unavailable, while direct long-video training requires substantial data and parallel inference requires reasonable GPU resources.
Abstract
from arXiv · showhide
In this paper, we propose NUWA-XL, a novel Diffusion over Diffusion architecture for eXtremely Long video generation. Most current work generates long videos segment by segment sequentially, which normally leads to the gap between training on short videos and inferring long videos, and the sequential generation is inefficient. Instead, our approach adopts a ``coarse-to-fine'' process, in which the video can be generated in parallel at the same granularity. A global diffusion model is applied to generate the keyframes across the entire time range, and then local diffusion models recursively fill in the content between nearby frames. This simple yet effective strategy allows us to directly train on long videos (3376 frames) to reduce the training-inference gap, and makes it possible to generate all segments in parallel. To evaluate our model, we build FlintstonesHD dataset, a new benchmark for long video generation. Experiments show that our model not only generates high-quality long videos with both global and local coherence, but also decreases the average inference time from 7.55min to 26s (by 94.26\%) at the same hardware setting when generating 1024 frames. The homepage link is \url{https://msra-nuwa.azurewebsites.net/}
1 Introduction
NUWA-XL addresses the difficulty of generating coherent, extremely long videos by replacing short-clip autoregression with a directly trained, parallel coarse-to-fine diffusion process. The paper also introduces FlintstonesHD as a benchmark for long video generation.
- Motivation: Long-video generation is increasingly important, but existing short-video methods face substantial computation demands when scaled to longer durations.Applications including films, cartoons, and short-form platforms commonly exceed the few-second lengths demonstrated by earlier video-generation systems.
- Challenges: Training on short videos while inferring 1024-frame videos creates a training-inference gap that can produce unrealistic shot changes and long-term incoherence.Phenaki and TATS, for example, are trained on fewer than 16 frames but can generate 1024 frames.
- Approach: NUWA-XL first generates global keyframes as a coarse storyline, then recursively fills intervals between adjacent frames with local diffusion models.With m depth and local diffusion length L, the process produces O(L^m) frames and supports a coarse-to-fine hierarchy.
- Advantages: Direct training on long videos enables NUWA-XL to reduce the training-inference gap, while parallel diffusion processes accelerate long-video generation.The model is reported as directly trained on 3376-frame videos and supports parallel inference.
- Results: 94.26% faster inference is reported for generating 1024 frames with NUWA-XL.This speedup is attributed to parallel inference.
- Benchmark: FlintstonesHD is introduced as a new dataset and benchmark for validating long-video generation.The dataset is intended to support evaluation of the model’s effectiveness.
2 Related Work
Prior video-generation work largely targets short videos or uses autoregressive-over-X architectures to extend short clips into long videos. NUWA-XL instead directly trains a Diffusion over Diffusion model on long videos and performs its diffusion processes in parallel.
- Short Video Generation: Image and video-generation research has used autoregressive, diffusion, GAN, VAE, and Transformer-based approaches, with many video methods focused on short videos.Examples include DDPM and LDM for images, and NUWA, VideoGPT, and related models for video.
- Long Video Generation: Most existing long-video methods use an “Autoregressive over X” architecture that extends short video clips through sliding-window or hierarchical generation.Here, X may be an autoregressive or diffusion model capable of generating short clips.
- NUWA-XL: NUWA-XL differs by directly training a Diffusion over Diffusion model on long videos to eliminate the training-inference gap.Its diffusion processes can also be executed in parallel to accelerate inference.
3 Method
NUWA-XL combines Temporal KLVAE with Mask Temporal Diffusion to model video latents and condition diffusion on prompts, timesteps, and optional visual frames. Its hierarchical inference generates long videos by first sampling sparse keyframes, then recursively filling intermediate frames in parallel.
- Temporal KLVAE: Temporal KLVAE extends a pre-trained image KLVAE with temporal convolution and attention while preserving its spatial modules and initializing temporal additions as identity functions.The video is encoded framewise by spatial convolutions, with temporal layers added to model temporal information without initially altering the original KLVAE behavior.
- Mask Temporal Diffusion: Mask Temporal Diffusion supports global diffusion without visual-frame conditions and local diffusion conditioned on the first and last frames while masking the middle frames.The same model accepts prompts with or without encoded visual conditions, enabling both stages of the hierarchical generator.
- Mask Temporal Diffusion: MTD encodes prompts with CLIP, encodes videos and visual conditions with T-KLVAE, and injects condition-derived scale and shift parameters into Mask 3D-UNet hidden states.The network additionally applies spatial self-attention, prompt cross-attention, and temporal self-attention before upsampling.
- Mask Temporal Diffusion: During inference, MTD iteratively denoises latent codes from noise and decodes the final latent representation into video pixels with T-KLVAE.The denoising process uses prompts, timestep embeddings, and encoded visual conditions when available.
- Diffusion over Diffusion Architecture: The Diffusion over Diffusion architecture first generates sparse keyframes globally, then applies local diffusion between adjacent keyframes to recursively produce increasingly fine frames.With depth m, the hierarchy generates video of length O(L^m); local diffusion stages can run in parallel after keyframe sampling.
4 Experiments
Experiments use FlintstonesHD, dense long-video annotations, and metrics for frame and clip quality. NUWA-XL outperforms sequential baselines on long-video quality and inference speed, while ablations examine architectural settings and qualitative coherence.
- 4.1 FlintstonesHD Dataset: FlintstonesHD provides densely annotated long videos to benchmark long-video generation.The dataset uses Flintstones episodes and frame-level captions to capture story and movement details.
- 4.2 Metrics: Block-FVD-X averages Fréchet Video Distance over short clips of length X to evaluate long-video quality.Avg-FID evaluates average generated-frame quality, while B-FVD-X evaluates generated video clips.
- 4.3 Quantitative Results: 85.09% and 94.26% are NUWA-XL's inference-speed improvements for 256 and 1024 generated frames, respectively.Unlike sequential AR-over-X models, NUWA-XL's frame quality does not decline with video length, and its B-FVD-16 declines more slowly.
- 4.3.2 Ablation study: NUWA-XL's depth enables increasingly longer videos while maintaining relatively high quality, with local diffusion length also improving quality at fixed video length.The depth comparison fixes local diffusion length L to 16, while the local-length comparison fixes depth m to 3.
- 4.4 Qualitative results: NUWA-XL preserves long-term coherence and realistic shot changes compared with AR over Diffusion in qualitative FlintstonesHD examples.The comparison highlights consistency between frames 22 and 1688 and a realistic shot change from frames 17 to 20.
5 Conclusion
The conclusion presents NUWA-XL as a coarse-to-fine Diffusion over Diffusion model that directly trains on long videos, supports parallel inference, and is evaluated with FlintstonesHD.
- 5 Conclusion: NUWA-XL views long-video generation as a coarse-to-fine Diffusion over Diffusion process and directly trains on 3376-frame videos.It also supports parallel inference and is evaluated using the FlintstonesHD benchmark.
6 Limitations
The paper identifies dataset coverage, data requirements, and hardware resources as limitations of NUWA-XL.
- 6 Limitations: NUWA-XL is validated only on the public Flintstones cartoon because open-domain long-video datasets are unavailable.The authors plan to extend evaluation to open-domain videos.
- 6 Limitations: Direct training on long videos creates a substantial data challenge.
- 6 Limitations: Parallel inference speedups require reasonable GPU resources.
7 Ethics Statement
The paper states that the research aligns with Microsoft’s responsible AI principles.
- 7 Ethics Statement: The research is conducted in alignment with Microsoft’s responsible AI principles.