Source-linked AI summary
ControlVideo: Training-free Controllable Text-to-Video Generation
Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, Qi Tian
TL;DR
Text-to-video generation is limited by the cost of temporal modeling and by appearance inconsistency and structural flickers. ControlVideo adapts ControlNet without finetuning using fully cross-frame interaction, interleaved smoothing, and hierarchical sampling; experiments report better qualitative and quantitative performance while generating short and long videos within several minutes on one NVIDIA 2080Ti. Its scope remains constrained when videos must extend beyond the supplied motion sequences.
Problem
Text-to-video models require massive training resources and can suffer from appearance inconsistency and structural flickers, especially in long videos.
Method
ControlVideo inflates ControlNet for video and adds fully cross-frame interaction, interleaved-frame smoothing, and hierarchical sampling without finetuning.
Results
Experiments on extensively collected motion-prompt pairs report better qualitative and quantitative performance than alternative competitors, with short and long videos generated within several minutes on one NVIDIA 2080Ti.
Takeaways & Limitations
The framework provides consistent and efficient controllable text-to-video generation using commodity GPUs for both short and long videos.
Takeaways & Limitations
ControlVideo struggles to generate videos beyond the supplied motion sequences when the requested text describes substantially different motion.
Abstract
from arXiv · showhide
Text-driven diffusion models have unlocked unprecedented abilities in image generation, whereas their video counterpart still lags behind due to the excessive training cost of temporal modeling. Besides the training burden, the generated videos also suffer from appearance inconsistency and structural flickers, especially in long video synthesis. To address these challenges, we design a \emph{training-free} framework called \textbf{ControlVideo} to enable natural and efficient text-to-video generation. ControlVideo, adapted from ControlNet, leverages coarsely structural consistency from input motion sequences, and introduces three modules to improve video generation. Firstly, to ensure appearance coherence between frames, ControlVideo adds fully cross-frame interaction in self-attention modules. Secondly, to mitigate the flicker effect, it introduces an interleaved-frame smoother that employs frame interpolation on alternated frames. Finally, to produce long videos efficiently, it utilizes a hierarchical sampler that separately synthesizes each short clip with holistic coherency. Empowered with these modules, ControlVideo outperforms the state-of-the-arts on extensive motion-prompt pairs quantitatively and qualitatively. Notably, thanks to the efficient designs, it generates both short and long videos within several minutes using one NVIDIA 2080Ti. Code is available at https://github.com/YBYBZhang/ControlVideo.
1 Introduction
ControlVideo addresses the training and consistency challenges of controllable text-to-video generation by adapting ControlNet without finetuning. Its cross-frame interaction, interleaved-frame smoother, and hierarchical sampler target appearance coherence, structural smoothness, and efficient long-video synthesis.
- Training text-to-video models requires massive high-quality video datasets and computational resources, limiting further research and applications.
- Controllable text-to-video generation conditions videos on textual descriptions and motion sequences while leveraging pretrained text-to-image models and coarse temporal structure.
- Prior methods still exhibit inconsistent appearance, large-motion artifacts, and structural flickers, while sparse cross-frame attention impedes inheriting text-to-image quality.
- Fully cross-frame interaction concatenates all frames as a larger image, directly inheriting ControlNet's high-quality and consistent generation without finetuning.
- The interleaved-frame smoother interpolates middle frames in alternating three-frame clips at selected timesteps to reduce structural flickers across the video.
- The hierarchical sampler splits long videos into short clips, pre-generates key frames for long-range coherence, and synthesizes intermediate clips conditioned on key-frame pairs.
- Experiments on extensively collected motion-prompt pairs show better qualitative and quantitative performance, with short and long videos generated within several minutes on one NVIDIA 2080Ti.
2 Background
The background introduces latent diffusion models and ControlNet as the image-generation foundations adapted by ControlVideo. LDMs diffuse in latent space, while ControlNet adds task-specific conditioning to Stable Diffusion's U-Net.
- Latent diffusion models apply the diffusion process in latent space rather than image space.
- An LDM encodes images into latent codes, decodes them back into images, and learns latent-code distributions through forward and backward diffusion.
- During generation, DDIM sampling starts from Gaussian noise and predicts progressively less noisy latent states.
- Stable Diffusion is a text-guided LDM pretrained on billions of image-text pairs, with τ denoting the text prompt.
- ControlNet extends Stable Diffusion with task-specific conditions such as depth maps, poses, and edges through an auxiliary U-Net.
3 ControlVideo
ControlVideo adapts ControlNet for training-free controllable text-to-video generation by combining fully cross-frame interaction, interleaved-frame smoothing, and hierarchical sampling. These modules target appearance consistency, structural flicker, and efficient long-video synthesis.
- Fully cross-frame interaction: ControlVideo extends ControlNet with fully cross-frame self-attention that concatenates all frames to preserve appearance consistency and reduce quality degradation.All latent frames interact as a “larger image,” maintaining a smaller generation gap with text-to-image models than sparser cross-frame mechanisms.
- Interleaved-frame smoother: The interleaved-frame smoother interpolates middle frames in three-frame clips across sequential timesteps to reduce structural flickers throughout the video.Predicted RGB frames are smoothed, re-encoded into latent space, and used for subsequent DDIM denoising at selected intermediate timesteps.
- Qualitative evaluation: Qualitative comparisons evaluate ControlVideo with depth maps and Canny edges for appearance consistency and video quality against alternative methods.The cited comparison identifies failures of Tune-A-Video to inherit source structures and visible large-motion artifacts from Text2Video-Zero.
- Interleaved-frame smoother: The smoother is applied only at selected intermediate timesteps, limiting added computation while retaining interpolated-frame individuality and quality.Following denoising steps preserve the interpolated frames’ characteristics.
- Hierarchical sampler: The hierarchical sampler splits long videos into short clips, pre-generates key frames for long-range coherence, and synthesizes intermediate clips conditioned on key-frame pairs.This clip-by-clip process maintains holistic consistency while reducing the resource burden of long-video generation.
- Quantitative evaluation: Table 1 reports quantitative consistency comparisons on 125 motion-prompt pairs, with the best results bolded.The supplied table caption specifies consistency evaluation but does not provide the individual metric values.
4 Experiments
Experiments compare ControlVideo with prior methods on motion-prompt pairs using qualitative, quantitative, user-preference, ablation, and long-video evaluations. ControlVideo shows stronger consistency and quality while enabling efficient long-video generation on commodity hardware.
- Experimental setup: The evaluation uses 125 video-prompt pairs, CLIP-based frame and prompt consistency metrics, and comparisons with Tune-A-Video, Text2Video-Zero, and Follow-Your-Pose.The dataset contains depth- and edge-conditioned motion-prompt pairs; a user study also evaluates video quality, temporal consistency, and text alignment.
- Qualitative comparisons: ControlVideo produces more consistent appearance and structure than competing methods for depth, canny-edge, and human-pose conditions, including large-motion inputs.Prior methods show inconsistent appearance, ignored structure, or visible artifacts, whereas ControlVideo handles large motion with high-quality consistent videos.
- Quantitative comparisons: Depth-conditioned ControlVideo outperforms state-of-the-art methods on frame consistency and prompt consistency.Depth-conditioned models also achieve higher temporal consistency and text fidelity than canny-conditioned counterparts because depth provides smoother motion information.
- User study: Raters strongly favor ControlVideo across video quality, temporal consistency, and text alignment, especially for temporal consistency.Each of 125 structure-prompt pairs was evaluated by five raters using majority votes.
- Ablation study: Fully cross-frame interaction improves appearance coherence and video quality, while the interleaved-frame smoother mitigates structural flickers and improves smoothness.The fully cross-frame mechanism incurs an extra 1 ∼2× time cost, described as acceptable for high-quality generation.
5 Related work
Related work develops text-to-video generation by extending text-to-image models with temporal modeling, but large-scale training remains costly. Training-reducing approaches instead use motion information or sparse cross-frame mechanisms, motivating ControlVideo’s training-free design.
- Text-to-video synthesis: Text-to-video models extend text-to-image generation with temporal consistency, using autoregressive or diffusion-based approaches.These methods aim to synthesize videos from textual descriptions while modeling higher-dimensional video distributions.
- Training burden: Large-scale text-to-video training requires extensive video-text data and substantial computational resources.This cost limits accessibility for many researchers and users.
- Training-reducing approaches: Motion-conditioned and sparse cross-frame methods reduce some training or finetuning demands while preserving considerable appearance consistency.Gen-1 and Follow-Your-Pose use coarse temporal information, while Tune-A-Video and Text2Video-Zero use sparse cross-frame mechanisms.
6 Discussion
ControlVideo is a training-free framework for consistent and efficient controllable text-to-video generation. Its experiments report better video quality and temporal consistency than prior state-of-the-art methods, while its main limitation is dependence on the input motion sequences.
- Framework: ControlVideo combines fully cross-frame interaction, an interleaved-frame smoother, a hierarchical sampler, and memory-efficient designs without training.The framework is adapted from ControlNet for controllable text-to-video generation.
- Reported outcome: Quantitative and qualitative experiments on extensive motion-prompt pairs report better video quality and temporal consistency than previous state-of-the-art methods.The framework can generate short and long videos in several minutes with commodity GPUs.
- Limitation: ControlVideo remains constrained by its input motion sequences and struggles to produce videos beyond those sequences.The paper gives the example that moonwalk poses are difficult to adapt into an Iron Man running video from text prompts.
- Broader impact: The method is presented as an efficient step toward accessible video creation on commodity hardware.The broader-impact discussion emphasizes accessibility for researchers and users who cannot access large-scale video models.