Source-linked AI summary
VideoComposer: Compositional Video Synthesis with Motion Controllability
Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, Jingren Zhou
TL;DR
Controllable video synthesis must handle varied temporal dynamics while preserving cross-frame consistency. VideoComposer composes textual, spatial, and temporal conditions, using motion vectors and an STC-encoder to guide dynamics and sequential relations. Experiments report flexible simultaneous control of spatial and temporal patterns, while watermarked training data and 256×256 resolution limit visual quality and detail.
Problem
Video synthesis remains difficult to control because temporal dynamics vary widely and generated frames must remain temporally consistent.
Method
VideoComposer decomposes videos into textual, spatial, and temporal conditions, using motion vectors for temporal guidance and an STC-encoder for spatio-temporal relations.
Results
VideoComposer flexibly composes videos with diverse conditions while maintaining synthesis quality and controlling spatial and temporal patterns, including hand-crafted motions.
Takeaways & Limitations
The framework supports simultaneous spatial and temporal control through text, sketches, reference inputs, and hand-crafted motion patterns.
Takeaways & Limitations
Watermarked WebVid10M training data affects generation quality, and 256×256 output resolution can leave delicate details unclear.
Abstract
from arXiv · showhide
The pursuit of controllability as a higher standard of visual content creation has yielded remarkable progress in customizable image synthesis. However, achieving controllable video synthesis remains challenging due to the large variation of temporal dynamics and the requirement of cross-frame temporal consistency. Based on the paradigm of compositional generation, this work presents VideoComposer that allows users to flexibly compose a video with textual conditions, spatial conditions, and more importantly temporal conditions. Specifically, considering the characteristic of video data, we introduce the motion vector from compressed videos as an explicit control signal to provide guidance regarding temporal dynamics. In addition, we develop a Spatio-Temporal Condition encoder (STC-encoder) that serves as a unified interface to effectively incorporate the spatial and temporal relations of sequential inputs, with which the model could make better use of temporal conditions and hence achieve higher inter-frame consistency. Extensive experimental results suggest that VideoComposer is able to control the spatial and temporal patterns simultaneously within a synthesized video in various forms, such as text description, sketch sequence, reference video, or even simply hand-crafted motions. The code and models will be publicly available at https://videocomposer.github.io.
1 Introduction
VideoComposer addresses limited video controllability by composing textual, spatial, and temporal conditions, including explicit motion guidance, while targeting temporal consistency and synthesis quality.
- Existing visual generative models achieve high fidelity and diversity but remain limited in controllability for practical applications.
- Video generation is difficult because temporal dynamics vary widely while adjacent frames must remain temporally continuous.
- VideoComposer decomposes videos into textual, spatial, and temporal factors, using motion vectors to guide inter-frame dynamics.
- Its unified STC-encoder captures spatio-temporal relations through cross-frame attention to enhance output consistency and fuse diverse condition sequences.
- VideoComposer composes videos from diverse conditions while maintaining synthesis quality and supports motion patterns specified by simple hand-crafted strokes.
2 Related work
Related work spans diffusion-based image and video synthesis, controllable editing, and motion modeling, motivating efficient temporal controls for video generation.
- Diffusion models have become prominent for image synthesis because of stable training, sample quality, and conditional-generation flexibility.
- Pixel-space video diffusion methods face high computational demands, whereas latent-space approaches reduce computational overhead.
- Motion cues support video understanding and generation, with prior work using optical flow or temporal structures to encode motion information.
- Compressed-video motion vectors offer a more efficient alternative to computationally demanding optical-flow extraction.
3 VideoComposer
VideoComposer combines latent video diffusion with composable textual, spatial, and temporal conditions. Its STC-encoder models sequential dependencies, while classifier-free guidance supports flexible condition combinations during inference.
- Video latent diffusion models: The framework uses latent diffusion to reduce video-processing cost while preserving local visual fidelity.
- Video latent diffusion models: A 3D UNet with temporal convolution and cross-attention denoises latent video representations while modeling locality and sequentiality.
- Videos as composable conditions: VideoComposer decomposes videos into textual, spatial, and temporal conditions that jointly determine spatial and temporal patterns.
- Temporal conditions: Motion vectors encode pixel-wise movements between adjacent frames and provide temporal-smooth motion control from compressed MPEG-4 videos.
- STC-encoder: The STC-encoder extracts spatial features and applies temporal Transformer modeling to embed space-time relations and provide a unified interface for sequential conditions.
- Condition fusion: Condition sequences are fused by element-wise addition, concatenated with noisy latents, and supplemented by cross-attention for textual and stylistic guidance.
- Training and inference: Classifier-free guidance extrapolates between two condition sets, emphasizing their differing elements for flexible applications such as text-driven video inpainting.
4 Experiments
VideoComposer is evaluated across compositional video-generation tasks, motion control, and STC-encoder ablations. The experiments show flexible conditioning and improved motion controllability, frame consistency, and structural fidelity.
- 4.1 Experimental setup: VideoComposer uses WebVid10M and LAION-400M, with frame consistency and motion controllability evaluated on generated videos.Frame consistency is computed from consecutive-frame CLIP similarity, while motion control uses endpoint error against ground-truth optical flow.
- 4.2 Composable video generation with versatile conditions: VideoComposer animates images and sketches according to text and initial spatial conditions, with additional temporal, mask, or style guidance refining structure and appearance.The model also fills masked video regions according to textual descriptions and temporal structural conditions.
- 4.3 Experimental results of motion control: The inclusion of motion vectors reduces motion control error, while the STC-encoder further advances motion controllability.The STC-encoder improves frame consistency across sketches, depth maps, and motion vectors, and qualitative examples show better condition fidelity and fewer structural defects.
- 4.3 Experimental results of motion control: Motion vectors support source-motion transfer, hand-crafted motion control, and static-background removal in video-to-video translation.Motion vectors prioritize moving visual cues because static regions are omitted from the representation.
- 4.4 Ablation study: The ablation study reports 0.915 without the STC-encoder and 0.927 with VideoComposer.The supplied table fragment identifies these values alongside the comparison between the ablated model and VideoComposer.
5 Conclusion
VideoComposer explores compositionality for flexible and controllable video synthesis. It combines motion-vector temporal guidance with an STC-encoder to aggregate spatial and temporal dependencies, and experiments support its controllability and creative range.
- 5 Conclusion: VideoComposer composes videos from textual, spatial, and temporal conditions within a latent diffusion model.Motion vectors provide temporal-dynamics guidance, while the STC-encoder aggregates spatial and temporal dependencies for inter-frame consistency.
- 5 Conclusion: Experiments combine diverse conditions to augment controllability and reveal the role of the proposed design choices.The conclusion characterizes VideoComposer as a flexible and controllable synthesis system with impressive creativity.
Appendix
The appendix provides implementation details, additional experimental results, and discussion of VideoComposer’s limitations and potential societal impact.
- Appendix: The appendix contains implementation details, additional experimental results, and discussion of limitations and potential societal impact.
A More implementation details
The appendix describes training, architecture, and additional compositional-generation examples. These materials include two-stage optimization, a 3D UNet with temporal processing, and demonstrations using depth maps, motion vectors, and style conditions.
- A More implementation details: VideoComposer is pre-trained with AdamW at 5 × 10−5 for 400k steps across image and video stages.The first and second stages use 132k and 268k steps, respectively, with one fourth of GPUs allocated to image pre-training.
- A More implementation details: Depth-map sequences can condition video generation, and an additional style condition can be incorporated in the final example.
- A More implementation details: WebVid10M frames are sampled with various strides to maintain a frame rate of 4.
- A More implementation details: The 3D UNet extends a 2D UNet with spatial and temporal convolution and transformer blocks.Spatial blocks are inherited from latent diffusion models, while temporal processing blocks are newly introduced.
- A More implementation details: Motion transfer uses motion vectors extracted from source videos or hand-crafted motions to generate videos from a single image.
B More experimental results
Additional experiments show that VideoComposer supports compositional control across sketch and depth sequences, motion transfer, and traditional text-to-video generation. These results indicate fidelity to multiple conditions and competitive text-to-video performance.
- Compositional sketch sequence-to-video generation: VideoComposer generates videos that adhere to compositional sketch sequences, emphasizing structure control in video-to-video translation.The demonstrated conditions include text, sketches, and style.
- Compositional depth sequence-to-video generation: VideoComposer effectively generates videos according to depth sequences while maintaining fidelity to text prompts, depth maps, and style.
- Motion transfer: Hand-crafted or source-video motion vectors transfer motion patterns to synthesized videos.The experiments demonstrate motion transfer using vectors extracted from off-the-shelf source videos as well as manually designed vectors.
- Text-to-video generation performance: VideoComposer achieves competitive performance against state-of-the-art text-to-video approaches on MSR-VTT using FVD and CLIPSIM.It also outperforms the first-stage text-to-video pre-training, indicating compositional generation does not sacrifice text-to-video capability.
C Discussion
The discussion identifies visual-quality limitations from watermarked training data and 256×256 output resolution, alongside societal risks from dual-use deployment. It recommends oversight and content filtering when the system is used beyond research.
- Limitations: Watermarked WebVid10M training data causes synthesized videos to contain watermarks that reduce generation quality and visual appeal.The dataset is used because a publicly available large-scale, high-quality alternative is absent.
- Limitations: Limiting generated videos to 256×256 resolution means some delicate details may remain unclear.The resolution constraint was adopted to reduce training cost.
- Potential societal impact: VideoComposer is a dual-use technology with risks including social-bias propagation and offensive-content creation.These risks are linked to implicit knowledge inherited from StableDiffusion, WebVid, and LAION.
- Potential societal impact: Deployment beyond research should include rigorous context evaluation, oversight, and filtering of prompts and generated content.