Source-linked AI summary
ModelScope Text-to-Video Technical Report
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, Shiwei Zhang
TL;DR
Video generation suffers from fidelity and motion-continuity challenges, motivating a trainable and publicly available baseline. ModelScopeT2V extends latent diffusion with spatio-temporal blocks and multi-frame training across image-text and video-text data. It achieves similar or superior quantitative and qualitative performance to state-of-the-art methods, while longer-video generation remains a future direction.
Problem
Video generation remains difficult because training challenges can produce sub-optimal fidelity and motion discontinuity, and publicly available codebases have been lacking.
Method
ModelScopeT2V extends latent diffusion with spatio-temporal blocks and multi-frame training on image-text and video-text paired datasets.
Results
ModelScopeT2V achieves the best performance on FID-vid (11.09) and FVD (550), while obtaining a competitive CLIPSIM score of 0.2930.
Takeaways & Limitations
ModelScopeT2V provides an open-source baseline for future research in text-to-video synthesis.
Takeaways & Limitations
Future work should explore additional conditions and the generation of longer videos containing more semantic information.
Abstract
from arXiv · showhide
This paper introduces ModelScopeT2V, a text-to-video synthesis model that evolves from a text-to-image synthesis model (i.e., Stable Diffusion). ModelScopeT2V incorporates spatio-temporal blocks to ensure consistent frame generation and smooth movement transitions. The model could adapt to varying frame numbers during training and inference, rendering it suitable for both image-text and video-text datasets. ModelScopeT2V brings together three components (i.e., VQGAN, a text encoder, and a denoising UNet), totally comprising 1.7 billion parameters, in which 0.5 billion parameters are dedicated to temporal capabilities. The model demonstrates superior performance over state-of-the-art methods across three evaluation metrics. The code and an online demo are available at \url{https://modelscope.cn/models/damo/text-to-video-synthesis/summary}.
1 Introduction
Video generation remains difficult because training challenges can produce low-fidelity videos with discontinuous motion, while publicly available codebases have been lacking. ModelScopeT2V addresses this gap with spatio-temporal modeling and multi-frame training, achieving similar or superior quantitative and qualitative performance to state-of-the-art methods.
- Video generation is challenging because training difficulty can cause sub-optimal fidelity and motion discontinuity, while public codebases remain scarce.
- ModelScopeT2V introduces spatio-temporal blocks to model temporal dependencies in latent diffusion for text-to-video generation.
- Its multi-frame training strategy uses both image-text and video-text paired datasets to enhance semantic richness.
- ModelScopeT2V performs quantitatively and qualitatively similar or superior to other state-of-the-art methods.
- Qualitative examples show generated video frames adhering to the supplied prompts.
2 Related work
Diffusion-based video generation builds on progress in image synthesis but still faces fidelity and motion-continuity challenges. ModelScopeT2V extends latent diffusion with a text-conditioned video architecture and combines image-text with video-text data to improve semantic diversity.
- Diffusion models improve image-generation diversity and fidelity but remain inefficient because generation requires iterative denoising in high-resolution pixel space.
- Text-to-image diffusion methods condition generation with natural-language representations, including CLIP and T5, often through language-conditioned UNet architectures.
- Text-to-video diffusion methods target realistic video generation but must address both high fidelity and motion continuity.
- ModelScopeT2V combines image-text and video-text paired datasets to exploit their semantic diversity for video-generation training.
3 Methodology
ModelScopeT2V is a latent video diffusion model built around VQGAN, a text-conditioned denoising UNet, and spatio-temporal processing. Its design combines spatial and temporal operations with multi-frame training over image-text and video-text data.
- Overall architecture: ModelScopeT2V maps videos between visual and latent spaces with VQGAN, while the denoising UNet performs diffusion and denoising in latent space.VQGAN's encoder E converts visual data to latent variables and decoder D reconstructs generated videos.
- Text conditioning: Text conditioning uses CLIP ViT-H/14 embeddings as keys and values in spatial cross-attention, allowing intermediate UNet features to aggregate textual information.The embedding c represents the prompt with maximum token length Np and token-embedding dimension Nc.
- Overall architecture: The UNet predicts noise across diffusion steps, transforming noisy latent variables toward a denoised latent video representation before VQGAN decodes the result.During training, the diffusion process progressively transforms ground-truth latent variables; during inference, denoising begins from random noise.
- Spatio-temporal block: The spatio-temporal block factorizes convolution and attention over space and time using spatial convolution, temporal convolution, spatial attention, and temporal attention.Its default operation counts are (N1, N2, N3, N4) = (2, 4, 2, 2); spatial attention includes cross-attention and self-attention, while temporal attention uses self-attention.
- Spatio-temporal block: Spatial convolution uses a 3×3 kernel within frames, whereas temporal convolution uses a kernel of size 3 across F frames.The convolutions operate on H/8 × W/8 latent features within each frame and across the temporal dimension.
- Spatio-temporal block: Spatial attention processes latent features across the spatial dimension, while temporal attention processes the temporal dimension using Transformer architectures.The attention modules operate over spatial size HW/64 and temporal size F, respectively.
- Multi-frame training: Multi-frame training mixes image-text and video-text paired datasets because the model structure can adapt to any frame length, treating an image as a one-frame video.One eighth of GPUs are assigned to image-text data and the remainder to video-text data.
4.1 Implementation details
ModelScopeT2V is trained with image-text and video-text datasets, using a latent diffusion architecture with temporal convolution and attention. The experiments use specified datasets, sampling procedures, and training settings.
- Datasets: LAION-5B image-text pairs and almost 10 million WebVid video-text pairs provide the training data, while MSR-VTT is reserved for validation.WebVid training uses 16 randomly selected frames at 3 frames per second; MSR-VTT is not used for training.
- Architecture: The model consists primarily of a text encoder, VQGAN, and denoising UNet initialized from Stable Diffusion checkpoints.VQGAN parameters remain frozen during training and inference.
- Training settings: Training uses AdamW with a learning rate of 5 × 10^-5 for 267 thousand iterations on 80G NVIDIA A100 GPUs.Multi-frame training uses batch sizes of 1,400 for images and 3,200 for videos.
- Temporal modeling: Temporal convolution and temporal attention are jointly used to model temporal dependencies in the denoising UNet.The temporal blocks contain 552 million of the UNet’s 1,345 million parameters, or 39%.
- Evaluation: The evaluation compares models on MSR-VTT using FID-vid, FVD, and CLIPSIM.FID-vid and FVD use 2,048 randomly selected test videos, while CLIPSIM uses captions from the test split.
- Model scale: The full ModelScopeT2V model contains approximately 1.7 billion parameters.The total includes VQGAN and the text encoder.
4.2 Main results
ModelScopeT2V achieves the best reported FID-vid and FVD scores on zero-shot MSR-VTT evaluation, while obtaining a competitive CLIPSIM score. Its CLIPSIM is only marginally below Make-A-Video despite Make-A-Video using additional training data.
- Quantitative results: 11.09 FID-vid and 550 FVD are the best reported scores for ModelScopeT2V on MSR-VTT.The evaluation uses a zero-shot setting because MSR-VTT was not used for training.
- Quantitative results: 0.2930 CLIPSIM indicates competitive semantic similarity between generated videos and text prompts.This score is only marginally lower than Make-A-Video’s, although Make-A-Video uses additional HD-VILA-100M data.
4.3 Qualitative results
Qualitative comparisons show that ModelScopeT2V produces high-quality videos with realistic details and broad motion, while comparisons with Imagen Video reveal trade-offs in vividness and detail.
- Make-A-Video comparison: ModelScopeT2V and Make-A-Video both generate high-quality videos in the three-example comparison.The comparison presents the prompt and corresponding videos from both methods.
- Make-A-Video comparison: ModelScopeT2V produces more realistic robot and dog examples than Make-A-Video in two qualitative cases.The authors attribute this advantage to joint training with image-text pairs.
- Make-A-Video comparison: ModelScopeT2V generates a wider range of motion than Make-A-Video, whereas Make-A-Video more closely matches the industrial-site prompt in one example.The authors associate Make-A-Video’s difference with its use of image CLIP embeddings.
- Imagen Video comparison: Imagen Video generates more vivid and contextually relevant content, while ModelScopeT2V represents the prompt with rougher details.The comparison uses three examples with the prompt and videos from both methods.
- Imagen Video comparison: ModelScopeT2V avoids the significantly distorted dog tongue shown in one Imagen Video example and robustly represents the prompt content.The authors attribute Imagen Video’s stronger performance to its T5 encoder, larger parameter count, and larger training dataset.
4.4 Community development
ModelScopeT2V’s public code and online demos have supported community extensions and applications. These developments broaden the model’s usage, including short-video creation.
- Release: Public GitHub repositories, ModelScope, and HuggingFace demos make ModelScopeT2V available to the open-source community.The paper reports active community engagement with the model.
- Community extensions: Projects including sd-webui-text2video and Text-To-Video-Finetuning extend ModelScopeT2V’s usage and applicability.The model’s video-generation feature has also been used to create short videos.
5 Conclusion
ModelScopeT2V combines spatio-temporal modeling with multi-frame training over image-text and video-text pairs, achieving similar or superior quantitative and qualitative performance to state-of-the-art methods. Future work targets additional conditioning and longer videos with richer semantic information.
- ModelScopeT2V designs spatio-temporal blocks with convolution and attention to model temporal dynamics.
- Multi-frame training uses both text-image and text-video pairs to leverage semantics from comprehensive visual content-text data.
- ModelScopeT2V achieves similar or superior performance to other state-of-the-art methods in quantitative and qualitative comparisons.
- Future research will investigate additional conditions, including multi-condition approaches and LoRA, to enhance video generation quality.
- Generating longer videos that encapsulate more semantic information is identified as an additional research direction.