Source-linked AI summary
Make-Your-Video: Customized Video Generation Using Textual and Structural Guidance
Jinbo Xing, Menghan Xia, Yuxin Liu, Yuechen Zhang, Yong Zhang, Yingqing He, Hanyuan Liu, Haoxin Chen, Xiaodong Cun, Xintao Wang, Ying Shan, Tien-Tsin Wong
TL;DR
Text alone conveys overall scene context but may not provide precise control for customized video generation. Make-Your-Video combines textual and structural guidance in a latent-diffusion video generator adapted from a pretrained image model with temporal modules and causal masking. The method reports superior controllable text-to-video performance, particularly in temporal coherence and fidelity to user guidance, while supporting longer videos and practical applications.
Problem
Text-to-video prompts convey global context but may be insufficient for precise control, motivating customized generation with concrete motion-structure guidance.
Method
Make-Your-Video combines text with frame-wise depth guidance and adapts a pretrained image Latent Diffusion Model using temporal modules and causal attention masking.
Results
The method achieves top controllable text-to-video performance qualitatively and quantitatively, especially in temporal coherence and fidelity to users’ guidance.
Takeaways & Limitations
The approach transfers rich visual concepts from pretrained image models, enables longer video synthesis, and demonstrates potential for practical applications.
Takeaways & Limitations
The model lacks precise control over visual appearance, requires frame-wise depth guidance, and may face usage challenges when that guidance is intensive.
Abstract
from arXiv · showhide
Creating a vivid video from the event or scenario in our imagination is a truly fascinating experience. Recent advancements in text-to-video synthesis have unveiled the potential to achieve this with prompts only. While text is convenient in conveying the overall scene context, it may be insufficient to control precisely. In this paper, we explore customized video generation by utilizing text as context description and motion structure (e.g. frame-wise depth) as concrete guidance. Our method, dubbed Make-Your-Video, involves joint-conditional video generation using a Latent Diffusion Model that is pre-trained for still image synthesis and then promoted for video generation with the introduction of temporal modules. This two-stage learning scheme not only reduces the computing resources required, but also improves the performance by transferring the rich concepts available in image datasets solely into video generation. Moreover, we use a simple yet effective causal attention mask strategy to enable longer video synthesis, which mitigates the potential quality degradation effectively. Experimental results show the superiority of our method over existing baselines, particularly in terms of temporal coherence and fidelity to users' guidance. In addition, our model enables several intriguing applications that demonstrate potential for practical usage.
1. Introduction
Make-Your-Video targets customized video generation by combining text, which conveys global context, with structural motion guidance for more precise control. It uses an efficient latent-diffusion framework with temporal modules and masking to support coherent, longer videos.
- Motivation: Textual prompts provide global scene context but offer insufficient precise control for aesthetically expressing imagined events.The method supplements text with concrete motion structure to address this control need.
- Structural Guidance: Frame-wise depth maps provide 3D-aware structural guidance that can remain rough, allowing non-professionals to prepare inputs while guiding plausible content.The paper gives an example where office objects guide generation of a photorealistic outdoor landscape.
- Efficient Generation: The method formulates conditional video generation with a Latent Diffusion Model and separates spatial image-synthesis training from temporal coherence training to reduce computational requirements.This scheme transfers concepts learned from image data into video generation.
- Temporal Modeling: Temporal blocks learn inter-frame coherence alongside temporal and spatial convolutions, improving temporal stability through greater adaptability to pretrained modules.The spatial convolutions assist the temporal components in adapting to the pretrained architecture.
- Results: Qualitative and quantitative evaluations report top performance over existing baselines, especially for temporal coherence and fidelity to user guidance.Ablation studies support the effectiveness of the proposed designs, and applications indicate potential practical usage.
- Longer Videos: A causal attention mask enables video synthesis at four times the training length while mitigating quality degradation.The masking strategy is presented as a simple mechanism for longer generation.
2. Related Work
Related work has advanced text-to-image and text-to-video diffusion, but video generation remains constrained by data scarcity, temporal consistency, and training cost. Make-Your-Video addresses these concerns by transferring pretrained image-model knowledge through learnable temporal modules and joint text-structure guidance.
- Diffusion Models: Latent diffusion models compress image data into a latent space, improving efficiency compared with diffusion directly in pixel space.This efficiency motivates their use for transferring visual concepts to video generation.
- Paper Positioning: Make-Your-Video adapts a pretrained text-to-image model into a joint text-structure-guided video generator by adding temporal modules and training on text-video data.Its stated aim is efficient adaptation alongside general mechanisms for longer video synthesis.
- Text-to-Video Challenges: Text-to-video generation still lags behind text-to-image generation because high-quality paired text-video data are scarce, temporal consistency is difficult, and training is resource-intensive.These constraints motivate methods that reuse pretrained image-generation knowledge.
- Prior T2V Transfer: Several approaches transfer text-to-image knowledge to video by initializing from pretrained text-to-image models and fine-tuning the entire model.Training-free transfer methods can instead rely on manually specified pseudo-motion dynamics but may produce low-quality, inconsistent videos.
- Longer Video Synthesis: Long-video methods commonly use interpolation or prediction, but interpolation only smooths motion and existing prediction mechanisms remain limited to curated domains.The paper therefore investigates general video prediction mechanisms for longer synthesis.
- Video Editing: Video editing methods that process individual frames can create temporal inconsistency, while texture-based editing may struggle to accurately reflect intended edits.These limitations distinguish video generation and editing from frame-level image manipulation.
3. Method
Make-Your-Video performs controllable text-to-video generation by conditioning latent diffusion on text prompts and frame-wise depth, while adapting a pre-trained image model with temporal modules. A causal attention mask supports longer synthesis by making temporal receptive fields adjustable.
- Conditional video generation: The framework conditions video generation on text prompts for appearance and frame-wise depth sequences for motion structure.Depth guidance can be extracted from input videos and concatenated with diffused latent features during denoising.
- Inference: At inference, random noise is recurrently denoised under text and depth guidance, then decoded into a video in pixel space.The depth condition can come from multiple sources, and the pre-trained decoder converts the final latent into video frames.
- Adapting LDMs: The method promotes a pre-trained image-conditioned LDM to video generation by freezing its spatial weights and learning additional temporal modules on video data.This preserves the learned image prior while addressing temporal consistency and limited text-video data coverage.
- Temporal modules: STRB and TT provide temporal modeling through pseudo-3D convolutions and temporal self-attention with learnable positional embeddings.STRB combines spatial and temporal convolutions, while TT models motion dynamics along the temporal axis.
- Temporal masking: Temporal self-attention can degrade quality for sequences longer than the 16-frame training length because longer token sequences disturb one another.The paper identifies confused attention across frames as a possible reason for this degradation.
- Temporal masking: The causal attention mask limits each token to preceding temporal tokens, enabling adjustable receptive fields for longer video synthesis.The mask also injects a directional motion prior and is used during all experiments.
4. Experiments
The experiments evaluate Make-Your-Video against controllable and pure text-to-video baselines, then test its adapting strategy and causal attention mask. Results show strong zero-shot performance and improved quality for longer synthesis with CAM.
- 4.1. Experimental Setup: The evaluation compares controllable baselines using depth guidance with pure text-to-video methods, including LVDMExt+Depth Adapter, T2V-zero+Depth CtrlNet, CogVideo, MagicVideo, Make-A-Video, and Video LDM.The comparison is conducted because joint text-structure-conditioned video synthesis has relatively few publicly available works.
- 4.1. Experimental Setup: The method is evaluated with FVD and KVD by comparing 2K generated 16-frame samples against UCF-101 samples.FVD and KVD measure feature-level similarity between synthesized and real videos using Fréchet-distance and kernel-based methods.
- 4.2. Evaluation on Video Generation: In zero-shot evaluation, Make-Your-Video significantly outperforms all baselines with lower FVD and KVD.Its performance is notable because Make-A-Video reaches a similar FVD while training on the additional HD-VILA-100M dataset containing 100M text-video pairs.
- 4.3. Ablation Studies: The ablation study compares SD-Depth, Temporal Transformer variants, temporal convolutions, and Pseudo 3D modules to assess the adapting strategy.The full method introduces both Temporal Transformer and Pseudo 3D modules while fine-tuning the added modules.
- 4.3. Ablation Studies: CAM improves conditional video generation for 16 frames and enables plausible 64-frame videos with greater detail and cross-frame coherence.Without CAM, the baseline suffers significant quality degradation and temporal inconsistency when applied to longer videos.
5. Applications
The applications demonstrate customized video creation and video re-rendering guided by text and rough motion structure. Across quantitative, qualitative, and user evaluations, the method shows strong temporal coherence, prompt conformity, and frame quality.
- 5.1. Video Creation: The method supports customized video creation from rough motion structures, including manually constructed miniature setups and scenes built with 3D modeling software.The generated videos can adhere to target textual descriptions and scene structure while remaining temporally consistent.
- 5.2. Video Re-rendering: For video re-rendering, the method changes appearance according to text prompts while preserving the input video's motion structure.The evaluation uses 11 representative videos from DAVIS and in-the-wild sources with manually designed prompts.
- 5.2. Video Re-rendering: The model outperforms baseline models in temporal consistency and all baselines except SD-Depth in prompt conformity.SD-Depth fails to synthesize consistent video frames, while Tune-A-Video shows overfitting to the original video.
- 5.2. Video Re-rendering: Qualitative comparisons show stronger combined video quality, structure preservation, text-video conformity, and temporal coherence than competing methods.Competitors exhibit inconsistent frames, low conformity, visual concept forgetting, or structure deviations in the reported cases.
- 5.3. User Study: In a user study with 32 participants, the method achieves the best ranking scores and preference rates for temporal coherence, text-and-structure conformity, and frame quality.Participants ranked five randomly ordered re-rendered videos for each example.
6. Limitation
The method has two stated limitations: it lacks precise visual-appearance control and requires frame-wise depth guidance, which can make use more demanding.
- 6. Limitation: The model cannot precisely control visual appearance, making videos featuring a specific individual or object intractable to synthesize.The paper relates this limitation to concept customization, which is extensively studied in text-to-image synthesis but less explored in text-to-video synthesis.
- 6. Limitation: The controllable model explicitly requires frame-wise depth guidance, which may increase usage challenges in some situations.The paper suggests sparse keyframe guidance as a possible future direction for broadening applications.
7. Conclusion
The paper presents an efficient customized video-generation approach using textual and structural guidance, with frozen image-generation modules and temporal mechanisms. Comparisons and applications support its effectiveness for controllable video generation and practical usage.
- 7. Conclusion: The method uses a pre-trained image LDM as frozen spatial modules, transferring visual concepts while maintaining satisfactory temporal coherence.The approach separates spatial image-synthesis modules from temporal video-generation modules.
- 7. Conclusion: A temporal masking mechanism facilitates longer video synthesis, extending diffusion-model generation beyond the training length.The causal attention mask enables videos up to 4× the training length while mitigating quality degradation.
- 7. Conclusion: Ablation studies confirm the effectiveness of the proposed design choices.
- 7. Conclusion: Comparisons with state-of-the-art methods show superiority in controllable text-to-video generation both quantitatively and qualitatively.
- 7. Conclusion: Applications demonstrate practical usage across real-life scene setup, dynamic 3D scene modeling, and video re-rendering.