Source-linked AI summary
Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, Mike Zheng Shou
TL;DR
Large-scale video-data training makes text-to-video generation expensive. The paper proposes One-Shot Video Tuning with Tune-A-Video, which adapts pretrained image diffusion models using one text-video pair and structural guidance, reporting remarkable results across varied applications.
Problem
Existing T2V approaches achieve promising results through large-scale text-video training, but this paradigm is expensive and time-consuming.
Method
Tune-A-Video adapts pretrained T2I diffusion models to one text-video pair using sparse spatio-temporal attention, efficient attention tuning, and DDIM inversion.
Results
Extensive qualitative and quantitative experiments demonstrate remarkable results across a wide range of text-driven video-generation applications.
Takeaways & Limitations
One-Shot Video Tuning eliminates the burden of training with large-scale video datasets while enabling temporally coherent video generation.
Takeaways & Limitations
The method can fail when input videos contain multiple objects and occlusion, reflecting limitations in handling object interactions.
Abstract
from arXiv · showhide
To replicate the success of text-to-image (T2I) generation, recent works employ large-scale video datasets to train a text-to-video (T2V) generator. Despite their promising results, such paradigm is computationally expensive. In this work, we propose a new T2V generation setting$\unicode{x2014}$One-Shot Video Tuning, where only one text-video pair is presented. Our model is built on state-of-the-art T2I diffusion models pre-trained on massive image data. We make two key observations: 1) T2I models can generate still images that represent verb terms; 2) extending T2I models to generate multiple images concurrently exhibits surprisingly good content consistency. To further learn continuous motion, we introduce Tune-A-Video, which involves a tailored spatio-temporal attention mechanism and an efficient one-shot tuning strategy. At inference, we employ DDIM inversion to provide structure guidance for sampling. Extensive qualitative and numerical experiments demonstrate the remarkable ability of our method across various applications.
1. Introduction
The paper introduces One-Shot Video Tuning to avoid expensive large-scale video training, building Tune-A-Video on pretrained T2I diffusion models. It combines spatio-temporal attention, efficient tuning, and structural inversion to generate temporally coherent videos from one text-video pair.
- Large-scale text-video training produces promising T2V results but requires extensive computation on expensive hardware and substantial time.
- One-Shot Video Tuning trains a T2V generator using only a single text-video pair and pretrained T2I knowledge.The setting is motivated by whether T2I models can infer novel videos from one example.
- Pretrained T2I models represent verb terms in still images, while spatio-temporal attention preserves consistent objects across concurrently generated frames.The resulting motion is not necessarily continuous without further tuning and guidance.
- Tune-A-Video uses sparse spatio-temporal attention and updates only attention projection matrices to preserve T2I knowledge while reducing tuning and attention costs.The sparse mechanism visits the first and former video frames; full space-time attention has quadratic computational growth.
- DDIM inversion supplies structure guidance at inference, producing temporally coherent videos with smooth movement after tuning maintains object consistency but lacks continuous motion.
- The method reports remarkable results across a wide range of text-driven video applications and extensive qualitative and quantitative experiments.
2. Related Work
Related work spans text-to-image and text-to-video generation, text-driven video editing, and models trained from a single video. Existing approaches face challenges including limited text-video data, temporal inconsistency, and substantial computational demands.
- Overview: The paper positions its method at the intersection of diffusion-based image and video generation, text-driven editing, and single-video generative modeling.The related fields are presented as connected areas with differences from the proposed method.
- Text-to-Image diffusion models: Text-to-image diffusion models have become a major foundation for generating images from text prompts.The surveyed models include approaches using classifier-free guidance, CLIP feature space, and cascaded diffusion.
- Text-to-Video generative models: Text-to-video generation remains challenging because high-quality large-scale text-video datasets are scarce and temporal consistency is difficult to model.Earlier systems often targeted simple domains such as moving digits or specific human actions.
- Text-to-Video generative models: Recent video diffusion systems transfer text-to-image progress to video but generally depend on extensive video-data training.The passage contrasts this reliance with the proposed framework’s efficient tuning of pre-trained models.
- Text-driven video editing: Applying image-editing models independently to video frames can produce inconsistency because those models lack temporal awareness.Text2Live is described as handling some texture-based editing but depending on Layered Neural Atlases, whose generation can take about 10 hours.
- Generation from a single video: 8 days of training for a 13-frame video illustrates the computational burden of some single-video GAN approaches.Patch nearest-neighbour methods reduce computation by orders of magnitude but remain limited in generality, according to the passage.
3. Method
Tune-A-Video adapts a pretrained T2I diffusion model to generate edited videos from one text-video pair. It inflates the model into the spatio-temporal domain, tunes attention efficiently, and uses DDIM inversion to preserve source-video structure during sampling.
- Problem: The task generates a novel video from an edited prompt while preserving the original video's motion.The source prompt can be modified to alter appearance or replace the subject while retaining the source motion.
- One-Shot Video Tuning: One-Shot Video Tuning achieves this goal using a publicly available T2I model and a single text-video pair.This avoids training a T2V model on large-scale video datasets.
- Network Inflation: The method inflates a 2D latent diffusion model with pseudo-3D convolutions and temporal self-attention for video generation.Spatial 3 × 3 kernels become 1 × 3 × 3 kernels, and a temporal self-attention layer is added to each transformer block.
- Spatio-Temporal Attention: Sparse causal ST-Attn queries the current frame while using the first and immediately preceding frames as keys and values.This design retains computational complexity O(2m(N)^2) rather than using full space-time attention.
- Fine-Tuning: ST-Attn updates only W Q, while temporal self-attention layers and cross-attention query projections are fine-tuned for temporal modeling and text-video alignment.The method fixes W K and W V in ST-Attn layers and fine-tunes newly added temporal self-attention layers.
- Inference: DDIM inversion converts the source video into latent noise that initializes sampling under the edited prompt, providing structural guidance for the generated video.The same input video requires inversion only once, and experiments report accurate transfer of structural movements.
4. Applications of Tune-A-Video
Tune-A-Video supports text-driven video generation and editing across object, background, and style changes while preserving video structure and motion consistency. It also works with personalized and conditional T2I models.
- Object editing: Tune-A-Video supports replacing, adding, or removing objects by modifying corresponding words or phrases in the text prompt.Examples include changing a man into Spider Man, adding a cowboy hat, or deleting an object description.
- Background change: Users can change video backgrounds or locations while preserving the consistency of the object’s movements.Examples include moving a skiing man to a beach or sunset setting and changing a countryside road into a sea view.
- Style transfer: The method transfers real-world videos into styles such as comic or Van Gogh by appending a global style descriptor to the prompt.The paper attributes these applications to the open-domain knowledge of pretrained T2I models.
- Personalized and controllable generation: Tune-A-Video integrates with personalized and conditional T2I models to create videos with specified subjects, styles, or controls.Examples include DreamBooth, T2I-Adapter, and ControlNet.
5. Experiments
Experiments evaluate Tune-A-Video on 42 DAVIS videos and 140 edited prompts against three baselines using qualitative comparisons, automatic metrics, user studies, and ablations. The method achieves stronger temporal consistency and textual faithfulness, while each core design contributes to its results.
- Dataset: The benchmark contains 42 DAVIS videos, automatically captioned footage, and 140 manually designed edited prompts.The evaluation uses representative videos and applications described in Section 4.
- Baselines: Tune-A-Video is compared with CogVideo, Plug-and-Play, and Text2LIVE across text-driven video generation and editing tasks.CogVideo is a zero-shot T2V model, while Plug-and-Play and Text2LIVE edit videos using different frame or atlas-based approaches.
- Qualitative results: Tune-A-Video produces temporally coherent videos that preserve input structure and align with edited words and details.CogVideo reflects general concepts but cannot take video input; Plug-and-Play lacks frame consistency; Text2LIVE struggles with accurate prompt edits.
- Quantitative results: Tune-A-Video outperforms baselines in both frame consistency and textual faithfulness under automatic metrics.CogVideo has consistent frames but weak textual representation, whereas Plug-and-Play has high textual faithfulness but inconsistent content.
- User study: User studies find higher preference for Tune-A-Video in both temporal consistency and alignment with textual descriptions.Five participants rated each example by majority vote, with CogVideo and Plug-and-Play less preferred for frame-wise or frame-text inconsistency.
- Ablation study: Ablations show that ST-Attn, DDIM inversion, and finetuning each contribute to successful results.Removing ST-Attn causes content discrepancies, removing inversion harms motion replication, and removing finetuning causes flickering from unsmooth motion.
6. Limitations and Future Work
Tune-A-Video can produce unpleasant results when input videos contain multiple objects and occlusions. The authors attribute this limitation to difficulties handling object interactions and leave additional conditioning as future work.
- Limitations: Multiple objects and occlusions can cause Tune-A-Video to mix objects and produce unpleasant results.The paper illustrates this failure with two pandas being mixed together.
- Future work: The authors suggest depth or other conditional information as a possible way to distinguish objects and their interactions.This direction is left for future work.
7. Conclusion
The paper introduces One-Shot Video Tuning and Tune-A-Video for text-driven video generation and editing from pretrained T2I models. Efficient tuning and structural inversion enable temporally coherent videos, with experiments showing results across many applications.
- Conclusion: One-Shot Video Tuning trains a T2V generator using one text-video pair and pretrained T2I models.This establishes a new task setting for T2V generation.
- Conclusion: Tune-A-Video uses efficient tuning and structural inversion to generate temporally coherent videos.The framework is presented for text-driven video generation and editing.
- Conclusion: Extensive experiments demonstrate results across a wide range of applications.
A. Dataset Details
The dataset uses 42 DAVIS videos spanning animals, vehicles, and humans, with automated captions and 140 manually designed edited prompts.
- 42 DAVIS videos cover animals, vehicles, and humans.The selected video items are listed in Table 2.
- BLIP-2 generates automated captions for the selected video footage.
- 140 edited prompts apply object editing, background changes, and style transfers across the videos.Three edited prompts were manually designed for each video.
- Table 2 lists the names of the videos selected from DAVIS.
B. User Study Details
A user study compares Tune-A-Video with Plug-and-Play and CogVideo on 140 edited prompts, using five raters to assess consistency and text matching.
- The study compares Tune-A-Video against Plug-and-Play and CogVideo on 140 edited prompts.
- Five raters evaluate each prompt by comparing two randomly ordered videos from different methods.
- Raters judge which video has higher consistency by selecting the smoother-looking video.
- Raters judge text matching by selecting the video that better represents the given description.
C. Additional Results
Additional figures provide more method examples, baseline comparisons, and ablation results.
- Figures 10 and 11 present additional video examples of the method.
- Figure 12 provides additional qualitative comparisons with baseline methods.
- Figure 13 presents additional ablation-study results.