Source-linked AI summary
VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, Ying Shan
TL;DR
High-quality text-to-video research is constrained by the inaccessibility of large, well-filtered video datasets and the low visual quality of WebVid-10M. The paper analyzes spatial-temporal coupling and uses low-quality videos with synthesized high-quality images, producing a generic high-quality video model. Its evaluations demonstrate superiority in picture quality, motion, and concept composition.
Problem
High-quality video models require large, well-filtered video datasets that are inaccessible to the community, while WebVid-10M has unsatisfactory picture quality.
Method
The method learns motion from low-quality videos, picture quality from synthesized high-quality images, and finetunes spatial modules after fully training the video model.
Results
The proposed model demonstrates superiority in picture quality, motion, and concept composition, while full training yields stronger spatial-temporal coupling than partial training.
Takeaways & Limitations
Data-level disentanglement provides a route to training generic high-quality video models without high-quality videos.
Takeaways & Limitations
The approach assumes that strong spatial-temporal coupling can tolerate parameter perturbations during finetuning with high-quality images without obvious motion degradation.
Abstract
from arXiv · showhide
Text-to-video generation aims to produce a video based on a given prompt. Recently, several commercial video models have been able to generate plausible videos with minimal noise, excellent details, and high aesthetic scores. However, these models rely on large-scale, well-filtered, high-quality videos that are not accessible to the community. Many existing research works, which train models using the low-quality WebVid-10M dataset, struggle to generate high-quality videos because the models are optimized to fit WebVid-10M. In this work, we explore the training scheme of video models extended from Stable Diffusion and investigate the feasibility of leveraging low-quality videos and synthesized high-quality images to obtain a high-quality video model. We first analyze the connection between the spatial and temporal modules of video models and the distribution shift to low-quality videos. We observe that full training of all modules results in a stronger coupling between spatial and temporal modules than only training temporal modules. Based on this stronger coupling, we shift the distribution to higher quality without motion degradation by finetuning spatial modules with high-quality images, resulting in a generic high-quality video model. Evaluations are conducted to demonstrate the superiority of the proposed method, particularly in picture quality, motion, and concept composition.
1. Introduction
The paper addresses the lack of accessible high-quality video data by combining low-quality videos with synthesized high-quality images. Its analysis motivates a fully trained video model followed by spatial-module finetuning.
- Motivation: Commercial text-to-video models achieve high visual quality, but their large, well-filtered video datasets are inaccessible to the research community.WebVid-10M is widely used academically, yet most clips are about 320p and have unsatisfactory picture quality.
- Analysis: The paper studies Stable Diffusion-based video models and analyzes how spatial and temporal modules interact under different training strategies.It compares full training with strategies that train only temporal modules.
- Analysis: Full training produces stronger coupling between appearance and motion than training only temporal modules.The stronger coupling supports more natural motion and greater tolerance to later spatial modifications.
- Method: The proposed method disentangles motion from appearance by learning motion from low-quality videos and picture quality from high-quality images.Synthesized images provide high resolution and complex concept composition while low-quality videos provide motion information.
- Method: The resulting pipeline first fully trains a video model, then finetunes its spatial weights with synthesized high-quality images.The paper identifies spatial-only finetuning as better than alternative modification strategies.
2. Related Work
Video generation research has progressed from earlier generative architectures to diffusion-based models, with latent-space extensions of text-to-image backbones becoming dominant. Related work also exposes limitations of combining motion modules with personalized image models.
- Video generation paradigms: Early video-generation methods used generative adversarial networks and variational auto-encoders, followed by transformer-based video synthesis.Examples include VGAN, TGAN, MoCoGAN, VideoGPT, TATS, and Phenaki.
- Diffusion video models: Diffusion-based video models operate in pixel or latent space, with many methods extending text-to-image diffusion backbones into video models.Latent-space approaches such as LVDM and MagicVideo extend latent diffusion models and form the dominant paradigm.
- Data limitation: Commercial systems generate plausible videos with strong details and aesthetics, but their high-quality training data and model weights are unavailable to researchers.This limits accessible development of video-generation and downstream research.
- Data limitation: WebVid-10M offers diverse, well-segmented clips but generally has unsatisfactory picture quality and approximately 320p resolution.These properties make it widely used yet insufficient for directly obtaining high-quality video models.
- Related adaptation: AnimateDiff combines WebVid-10M temporal modules with selected LoRA image models, but mismatches can degrade motion and concept composition.Its reliance on personalized LoRA models prevents it from serving as a generic solution.
3. Method
The method analyzes spatial-temporal coupling in SD-based video models and uses that analysis to separate motion learning from appearance-quality improvement. It first trains a video model with videos, then finetunes spatial modules with high-quality images.
- Base T2V Model: SD-based video models extend a pretrained text-to-image backbone by adding temporal modules, with full and partial training differing in whether spatial modules are updated.Full training updates spatial and temporal modules from pretrained SD weights; partial training updates only temporal modules while fixing the spatial modules.
- Spatial-temporal Connection Analyses: The analysis perturbs spatial or temporal parameters using high-quality image data while freezing or finetuning selected modules to compare model coupling.The experiments use synthesized JDB images and LORA for parameter perturbation, then examine visual quality, motion, and temporal consistency.
- Finetuning Strategy: Directly finetuning spatial modules of a fully trained model improves picture quality while preserving motion better than jointly finetuning spatial and temporal modules.F-Spa-DIR is more stable and temporally consistent, whereas F-Spa&Temp-DIR exhibits foreground and background flashes.
- Spatial-temporal Connection Analyses: Fully trained models have stronger spatial-temporal coupling than partially trained models and therefore tolerate parameter perturbations with less motion degradation.The partially trained model’s coupling is easily broken, causing rapid motion degeneration and picture-quality shifts when parameters are perturbed.
- Data-level Disentanglement: The resulting data-level pipeline fully trains a video model with low-quality videos and then directly finetunes only its spatial modules with high-quality images.This uses low-quality videos for motion consistency and high-quality images for picture quality and concept composition.
F (F-Spa-DIR) and M C
Among the tested finetuning choices, direct spatial finetuning gives the best balance of picture quality and motion. Synthesized images further improve concept composition compared with web-collected images.
- F-Spa-DIR and M C: F-Spa-DIR and F-Spa&Temp-DIR achieve the best picture quality among finetuned models, but F-Spa-DIR has better motion.F-Spa&Temp-DIR shows flashing foreground and background, while F-Spa-DIR provides better temporal behavior.
- F-Spa-DIR and M C: Directly finetuning spatial modules with high-quality images improves picture quality without marginal motion loss.The selected pipeline fully trains on low-quality videos before direct spatial finetuning on high-quality images.
- Promotion of Concept Composition: Synthesized images with complex concepts are used during partial finetuning to transfer concept-composition ability from successful text-to-image models.This avoids requiring real-world data to capture both complex concepts and motion simultaneously.
- Promotion of Concept Composition: Models trained with JDB synthesized by Midjourney have much better concept composition ability than models trained with LAION-aesthetics V2 web images.The comparison uses JDB and LAION-aesthetics V2 as alternative image datasets for the second finetuning stage.
4. Experiments
Experiments evaluate the method using WebVid-10M, synthesized high-quality images, EvalCrafter, qualitative comparisons, and user studies. Results show strong visual quality, alignment, motion, and strategy-level evidence for spatial-module finetuning.
- 4.1. Settings: EvalCrafter evaluates visual quality, content quality, motion quality, temporal consistency, and text-caption alignment across about 18 objective metrics.The evaluation uses roughly 512 prompts and supplements objective metrics with human preference studies for motion quality.
- 4.2. Comparison with State-of-the-Art T2V Models: Our method achieves visual quality comparable to VideoCrafter1 and Pika Labs, ranks second in text-video alignment, and surpasses Show-1 in motion quality.The reported comparison states that motion remains strong without significant degradation, although models trained with more video data perform better in motion.
- 4.2. Comparison with State-of-the-Art T2V Models: Our visual quality is on par with Gen2 and Pika Labs, while motion quality exceeds AnimateDiff and is comparable to Show-1.The qualitative comparison attributes the visual-quality shift to using JDB images rather than only WebVid-10M videos.
- 4.3. Strategy Evaluation: Fully trained models exhibit stronger spatial-temporal coupling, tolerate larger parameter perturbations, and retain better motion quality than partially trained models.Partially trained models are easier to shift toward improved visual quality, whereas fully trained models better preserve motion under perturbation.
- 4.3. Strategy Evaluation: Direct spatial-module finetuning gives the best motion among similarly visual-quality strategies, while JDB produces better concept composition and visual scores than LAION Aesthetics V2.F-Spa-DIR is more temporally stable than F-Spa&Temp-DIR, whose outputs show foreground and background flashes.
5. Conclusion
The paper concludes that high-quality video diffusion models can be trained without high-quality videos by separating motion learning from appearance learning. Its evaluations support using low-quality videos for motion and synthetic high-quality images for appearance and concept composition.
- 5. Conclusion: The method trains high-quality video diffusion models without high-quality videos by using low-quality videos for motion and high-quality images for appearance.The approach is based on analyzing coupling between spatial and temporal dimensions in Stable Diffusion-based video models.
- 5. Conclusion: Fully trained text-to-video models exhibit stronger spatial-temporal coupling than partially trained models.This observation guides the proposed training strategy.
- 5. Conclusion: Synthetic images with complex concepts are recommended for finetuning instead of real images.The conclusion frames this choice as part of separating appearance from motion at the data level.
- 5. Conclusion: Quantitative and qualitative evaluations demonstrate the effectiveness of the proposed method.The reported evaluation covers the method’s high-quality video generation objective.