Source-linked AI summary
LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, Yuwei Guo, Tianxing Wu, Chenyang Si, Yuming Jiang, Cunjian Chen, Chen Change Loy, Bo Dai, Dahua Lin, Yu Qiao, Ziwei Liu
TL;DR
High-quality text-to-video generation must balance visual realism and temporal coherence with the creative composition of pre-trained text-to-image models. LaVie addresses this with cascaded video latent diffusion models, joint image-video fine-tuning, and the Vimeo25M dataset, achieving strong quantitative and qualitative results while retaining known limitations in multi-subject scenes and human hands.
Problem
High-quality, temporally coherent T2V generation must also preserve the creative generation and compositional strengths of pre-trained T2I models.
Method
LaVie combines a pre-trained T2I model with cascaded Base T2V, temporal interpolation, and video super-resolution latent diffusion models, trained jointly on image and video data.
Results
LaVie achieves state-of-the-art quantitative and qualitative performance; under the same experimental setting, it outperforms Video LDM by 24.31.
Takeaways & Limitations
Simple temporal self-attention with RoPE and joint image-video fine-tuning support realistic, temporally coherent, creative, and concept-compositional video generation.
Takeaways & Limitations
LaVie struggles with scenes involving more than two subjects and often generates anatomically inaccurate hands, including incorrect finger counts.
Abstract
from arXiv · showhide
This work aims to learn a high-quality text-to-video (T2V) generative model by leveraging a pre-trained text-to-image (T2I) model as a basis. It is a highly desirable yet challenging task to simultaneously a) accomplish the synthesis of visually realistic and temporally coherent videos while b) preserving the strong creative generation nature of the pre-trained T2I model. To this end, we propose LaVie, an integrated video generation framework that operates on cascaded video latent diffusion models, comprising a base T2V model, a temporal interpolation model, and a video super-resolution model. Our key insights are two-fold: 1) We reveal that the incorporation of simple temporal self-attentions, coupled with rotary positional encoding, adequately captures the temporal correlations inherent in video data. 2) Additionally, we validate that the process of joint image-video fine-tuning plays a pivotal role in producing high-quality and creative outcomes. To enhance the performance of LaVie, we contribute a comprehensive and diverse video dataset named Vimeo25M, consisting of 25 million text-video pairs that prioritize quality, diversity, and aesthetic appeal. Extensive experiments demonstrate that LaVie achieves state-of-the-art performance both quantitatively and qualitatively. Furthermore, we showcase the versatility of pre-trained LaVie models in various long video generation and personalized video synthesis applications.
1 INTRODUCTION
LaVie addresses the challenge of generating realistic, temporally coherent videos while preserving the creative and compositional strengths of pre-trained text-to-image models. It combines a pre-trained T2I prior with architectural and training choices intended to balance video quality, training cost, and compositionality.
- Training T2V systems from scratch requires extensive computation to learn spatio-temporal distributions.
- Adapting pre-trained T2I models can accelerate convergence by transferring prior spatial knowledge to video data.
- LaVie is a 3B-parameter cascaded video latent diffusion framework built on Stable Diffusion for realistic, temporally coherent, and creative video synthesis.
- Simple temporal self-attention with RoPE adequately captures video temporal correlations, while joint image-video fine-tuning supports high-quality and creative results.The paper reports that direct video-only fine-tuning causes catastrophic forgetting and loss of concept-mixing ability.
2 RELATED WORK
Prior video-generation work extends image or video generative models with temporal mechanisms, while LaVie organizes text-to-video synthesis through a cascaded pipeline. The pipeline progressively improves temporal smoothness and spatial quality after generating key frames.
- The related work includes unconditional video generation methods based on GANs and other deep generative models.
- LaVie generates prompt-aligned key frames, interpolates additional frames for smoother motion, and super-resolves the result.
- Conditional text-to-video generation synthesizes videos from text descriptions, commonly extending text-to-image models with temporal convolutions or attention.
3 PRELIMINARY OF DIFFUSION MODELS
Diffusion models learn data distributions through noise addition and denoising, while latent diffusion performs these operations in compressed latent space. LaVie applies this latent-space formulation to per-frame video representations and their spatio-temporal distribution.
- Diffusion models add noise to data through a scheduled Markov chain and learn denoising over diffusion steps.The noisy sample is expressed as z_t = α_t z + σ_t ϵ, with ϵ sampled from a standard normal distribution.
- Latent diffusion models encode inputs into low-dimensional latent codes, perform diffusion and denoising there, and decode the final latent output.Operating in latent space substantially reduces training and inference time compared with data-space diffusion.
- LaVie encodes each video frame into a latent code and models the latent spatio-temporal distribution during diffusion.
4 OUR APPROACH
LaVie is a cascaded text-conditioned video latent diffusion system that generates low-resolution key frames, increases frame rate, and produces high-definition video. Its base model jointly fine-tunes on images and videos to preserve creative composition and transfer visual concepts.
- Overall framework: LaVie comprises a Base T2V model, Temporal Interpolation model, and Video Super Resolution model trained with text conditioning.The full system generates 61 frames at 1280×2048-pixel resolution.
- Base T2V model: The base model generates short, low-resolution key frames while preserving diversity and compositionality for creative prompts.
- Base T2V model: Joint image-video fine-tuning mitigates catastrophic forgetting, improves video quality, and transfers styles, scenes, and characters from images to videos.The resulting base model can handle both text-to-image and text-to-video tasks without modifying the LDM architecture.
- Temporal interpolation: The temporal interpolation network quadruples the base video's frame rate by converting 16 input frames into 61 output frames.
- Video super resolution: The VSR model uses a pre-trained image ×4 upscaler prior and temporal layers to increase video resolution to 1280×2048.Patchwise training on 320×320 patches and low-resolution video conditioning support the upscaler's operation.
- Dataset: Vimeo25M provides examples of the video-text data used to support diverse video generation.
5 EXPERIMENTS
LaVie is evaluated through dataset construction, implementation details, qualitative comparisons, zero-shot benchmarks, and human studies. Results indicate stronger visual quality and semantic performance, while motion smoothness and detailed face, body, and hand generation remain challenging.
- DATASETS: Vimeo25M was curated to address WebVid10M limitations in resolution, diversity, and aesthetics, yielding 25 million video segments from filtered scenes.The dataset filters captions and segments, and retains videos meeting resolution and aesthetic criteria.
- DATASETS: 16.89% of Vimeo25M videos scored above 6 in aesthetics versus 7.22% for WebVid10M, while scores between 4 and 6 were 79.12% versus 72.58%.Vimeo25M also contains predominantly higher-resolution videos than WebVid10M.
- QUALITATIVE EVALUATION: Qualitatively, LaVie synthesizes diverse content and outperforms Make-A-Video in visual fidelity, while capturing Van Gogh style more effectively than the other compared methods.The authors attribute these results to pretrained LDM initialization and joint image-video fine-tuning, but note that systematic comparison with two methods is constrained by unavailable testing code.
- QUANTITATIVE EVALUATION: 24.31 was the improvement over Video LDM under the same experimental setting in zero-shot evaluation, although Make-A-Video remained ahead overall.On MSR-VTT, LaVie achieved superior or competitive performance using CLIPSIM-based text-video semantic similarity evaluation.
- QUANTITATIVE EVALUATION: Human raters preferred LaVie over the other two approaches, but all three methods received unsatisfactory motion-smoothness scores.High-quality face, body, and hand visuals also remained difficult to produce.
6 LIMITATIONS
LaVie has two reported limitations: it struggles to keep multiple subjects distinct and to render anatomically correct hands. The authors suggest stronger language understanding and broader human-video data as possible remedies.
- Multi-subject generation: LaVie can mix the appearances of more than two subjects instead of generating distinct individuals.The paper gives “Albert Einstein discussing an academic paper with Spiderman” as an example.
- Multi-subject generation: Replacing CLIP with a more robust language model such as T5 could improve complex-description understanding and reduce subject mixing.
- Hands generation: LaVie often renders incorrect numbers of fingers, making human hands less realistic.
- Hands generation: Training on a larger, more diverse dataset containing human videos could improve realistic and anatomically correct hand generation.
7 CONCLUSION
LaVie is presented as a high-quality, temporally coherent text-to-video foundation model built from cascaded diffusion models and joint image-video fine-tuning. The paper reports strong zero-shot results and demonstrates extensions to long and personalized video generation.
- LaVie produces high-quality and temporally coherent videos using cascaded video diffusion models and temporal modules added to a pretrained LDM.
- Vimeo25M provides higher-resolution, more diverse video-text data with improved aesthetic scores for training.
- Joint image-video fine-tuning helps LaVie compose styles, characters, and scenes in generated videos.
- Comprehensive zero-shot evaluations validate LaVie’s superiority over state-of-the-art approaches quantitatively and qualitatively.
- Long video generation and personalized video synthesis demonstrate the versatility of LaVie’s pretrained base model.