Source-linked AI summary
VideoCrafter1: Open Diffusion Models for High-Quality Video Generation
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, Chao Weng, Ying Shan
TL;DR
Video generation lacks accessible open-source models that combine high quality with effective text or image conditioning. VideoCrafter1 introduces open-source T2V and I2V diffusion models, achieving strong reported quality while preserving reference-image content in I2V generation. The models remain limited to two-second videos, leaving duration, resolution, motion, and visual quality for future improvement.
Problem
Many high-quality video-generation systems are commercial or inaccessible, while existing open-source T2V and I2V models have quality, resolution, motion, artifact, or content-preservation limitations.
Method
VideoCrafter1 introduces open-source T2V and I2V diffusion models, using temporal attention for T2V and image embeddings injected through cross attention for I2V.
Results
The T2V model achieves the best visual and video quality among open-source models, while the I2V model preserves the content and structure of the reference image.
Takeaways & Limitations
The released models provide open-source foundations for researchers and practitioners to build upon in video generation.
Takeaways & Limitations
Both models currently generate videos limited to 2 seconds, with longer duration, higher resolution, and better motion and visual quality left for future work.
Abstract
from arXiv · showhide
Video generation has increasingly gained interest in both academia and industry. Although commercial tools can generate plausible videos, there is a limited number of open-source models available for researchers and engineers. In this work, we introduce two diffusion models for high-quality video generation, namely text-to-video (T2V) and image-to-video (I2V) models. T2V models synthesize a video based on a given text input, while I2V models incorporate an additional image input. Our proposed T2V model can generate realistic and cinematic-quality videos with a resolution of $1024 \times 576$, outperforming other open-source T2V models in terms of quality. The I2V model is designed to produce videos that strictly adhere to the content of the provided reference image, preserving its content, structure, and style. This model is the first open-source I2V foundation model capable of transforming a given image into a video clip while maintaining content preservation constraints. We believe that these open-source video generation models will contribute significantly to the technological advancements within the community.
1. Introduction
Video generation research has advanced, but open-source systems remain limited in accessibility, quality, resolution, and image-content preservation. VideoCrafter1 addresses these gaps with open-source T2V and I2V diffusion models.
- Commercial T2V systems produce high-quality videos, but their models remain inaccessible to researchers for further exploration.
- Existing open-source T2V models are constrained by low resolution, short clips, flicker, noise, or poor video quality.ModelScope generates 256 × 256 videos, Hotshot-XL generates 8-frame 512 × 512 GIFs, and Zeroscope V2 XL retains flicker and visible noise.
- Open-source I2V models generally provide limited motion, visible artifacts, or inadequate preservation of the reference image’s content and structure.I2VGen-XL matches semantic meaning but does not strictly follow the reference image’s content and structure.
- VideoCrafter1 introduces separate diffusion models for text-to-video and image-to-video generation.The T2V model builds on SD 2.1 with temporal attention, while the I2V model accepts text and image inputs.
- The T2V model generates cinematic-quality videos at 1024 × 576, while the I2V model is designed to preserve input-image content and structure.The models are trained using large-scale image and video datasets and are released for researchers and practitioners to build upon.
2. Related Works
Related work develops video diffusion models from text-to-image foundations, while conditional controls address the limits of text-only guidance. Visual comparisons situate VideoCrafter1 among commercial and open-source systems.
- Video diffusion models extend diffusion modeling from images to the spatial and temporal distributions of videos conditioned on text prompts.
- Figure 2 compares generated videos from Gen-2, Pika Labs, I2VGen-XL, and Zeroscope-XL.
- Subsequent approaches improve definition or efficiency through cascaded diffusion, v-prediction, and latent-space transfer of text-to-image knowledge.
- Text-only guidance can be verbose and may not accurately reflect users’ intentions, motivating conditional controls for video diffusion models.
3. Methodology
VideoCrafter1 uses latent video diffusion with a spatiotemporal U-Net, conditioning generation on text, frame rate, and— for I2V—rich image features. Its image branch projects patch-level visual tokens into a text-aligned space and fuses them with text through dual cross-attention.
- Latent video diffusion: The model encodes videos into compact latent representations with a pretrained Stable Diffusion VAE, performs diffusion and denoising there, then decodes clean latents into pixel-space videos.The VAE processes each frame independently without extracting temporal information.
- Denoising 3D U-Net: The denoiser is a 3D U-Net built from spatial-temporal blocks with skip connections, convolutional layers, spatial transformers, and temporal transformers.Temporal transformers model video dynamics alongside spatial processing.
- Conditioning: Text prompts provide semantic control through cross-attention, while FPS conditioning controls the motion speed of generated videos.FPS and timestep embeddings are projected and fused before being added to the convolutional features.
- Image conditioning: Figure 5 compares conditioning on a global semantic image token against conditioning on full patch visual tokens under the same text prompt.The comparison uses the prompt “a beautiful girl with colorful hair.”
- Image conditioning: For I2V generation, CLIP patch visual tokens are projected into a text-aligned image embedding space and combined with text embeddings through dual cross-attention.The same query is used for image and text cross-attention, with newly added image value projections.
4. Experiments
The experiments evaluate VideoCrafter’s training setup and compare its text-to-video and image-to-video models with open-source and commercial systems. VideoCrafter achieves strong visual quality, alignment, temporal consistency, motion, and reference-image fidelity, while retaining some limitations.
- Implementation Details: The models use joint image-video training with LAION COCO, WebVid-10M, and an additional high-quality video dataset.The training strategy combines large-scale image and video data for model development.
- Implementation Details: The T2V model is trained progressively from 256 × 256 to higher resolutions before evaluation.Training proceeds through staged low-resolution and higher-resolution video finetuning.
- Text-to-Video Evaluation: EvalCrafter evaluates visual quality and text-video alignment using quantitative metrics and user studies against Gen-2, Pika Labs, and ModelScope.The paper reports main results in Table 1 and Figure 6, with qualitative examples in Figures 2, 7, and 9.
- Text-to-Video Results: VideoCrafter outperforms open-source T2V models in visual quality and text alignment, with larger motion but occasional temporal-consistency errors.The authors attribute the larger motion to training that encourages substantial object movement.
- Image-to-Video Results: For I2V, VideoCrafter provides good temporal consistency and motion magnitude with acceptable visual fidelity, but successful rate and facial artifacts remain limitations.Pika and Gen-2 provide stronger reference-image fidelity in some comparisons, while their motion or stability has separate shortcomings.
5. Conclusion and Future Work
VideoCrafter1 introduces open-source T2V and I2V diffusion models, while identifying longer duration, higher resolution, and improved motion and visual quality as key future directions.
- The T2V model generates high-quality, high-resolution, cinematic-quality videos at 1024 × 576.
- The I2V model is the first open-source generic I2V foundation model reported to preserve the content and structure of a reference image.
- The two models currently produce videos limited to 2 seconds, motivating future work on longer duration, resolution, motion, and visual quality.Suggested directions include additional-frame training, frame interpolation, spatial upscaling, and higher-quality data.