Source-linked AI summary
DreamVideo: Composing Your Dream Videos with Customized Subject and Motion
Yujie Wei, Shiwei Zhang, Zhiwu Qing, Hangjie Yuan, Zhiheng Liu, Yu Liu, Yingya Zhang, Jingren Zhou, Hongming Shan
TL;DR
Customized video generation requires simultaneous control of subject identity and motion, a combination that existing approaches do not fully provide. DreamVideo decouples the task into subject and motion learning with lightweight adapters, and experiments report superior customization over existing state-of-the-art methods. Its scope is demonstrated on joint and individual customization using 20 subjects and 30 motion patterns.
Problem
Customized video generation remains challenging because it must control diverse spatial subject content and intricate temporal motion simultaneously.
Method
DreamVideo learns subjects with textual inversion and an identity adapter, learns motions with an appearance-guided motion adapter, and combines the two adapters using a pretrained video diffusion model.
Results
DreamVideo demonstrates superior customization performance over existing state-of-the-art methods in joint and individual subject and motion customization experiments.
Takeaways & Limitations
The two-stage adapter design supports flexible customization of subjects and motions, with experiments covering 20 subjects and 30 motion patterns.
Abstract
from arXiv · showhide
Customized generation using diffusion models has made impressive progress in image generation, but remains unsatisfactory in the challenging video generation task, as it requires the controllability of both subjects and motions. To that end, we present DreamVideo, a novel approach to generating personalized videos from a few static images of the desired subject and a few videos of target motion. DreamVideo decouples this task into two stages, subject learning and motion learning, by leveraging a pre-trained video diffusion model. The subject learning aims to accurately capture the fine appearance of the subject from provided images, which is achieved by combining textual inversion and fine-tuning of our carefully designed identity adapter. In motion learning, we architect a motion adapter and fine-tune it on the given videos to effectively model the target motion pattern. Combining these two lightweight and efficient adapters allows for flexible customization of any subject with any motion. Extensive experimental results demonstrate the superior performance of our DreamVideo over the state-of-the-art methods for customized video generation. Our project page is at https://dreamvideo-t2v.github.io.
1 Fudan University 2 Alibaba Group 3 Huazhong University of Science and Technology 4 Zhejiang University
The paper lists affiliations with Fudan University, Alibaba Group, Huazhong University of Science and Technology, and Zhejiang University, alongside a figure illustrating customized subjects and motions.
- Figure 1 presents customized video results organized by specific subjects on the left and motions across the top.
- The examples demonstrate customization of subject identity and motion pattern across varied context descriptions.
1. Introduction
Customized video generation must control spatial subject content and temporal motion simultaneously, but existing methods commonly optimize only one aspect or provide limited motion diversity. DreamVideo addresses this gap by decoupling subject and motion learning into two lightweight adapters.
- Customized video generation is challenging because videos combine diverse spatial content with intricate temporal dynamics.
- Existing methods often optimize only subject appearance or motion, which can reduce generalization to the other aspect.
- AnimateDiff targets generalized animation but can lack motion diversity, particularly by emphasizing camera movements.
- DreamVideo learns a textual identity and identity adapter for subject appearance, then trains a motion adapter on target videos with appearance guidance.
- The two-stage design enables flexible composition of customized subjects and motions from a few images and videos.
- Experiments use 20 customized subjects and 30 motion patterns, with reported customization performance surpassing state-of-the-art methods.
2. Related Work
Related work spans text-to-video generation and customized image or video generation. DreamVideo is situated among approaches that extend diffusion-based generation toward joint subject and motion customization.
- Text-to-video generation: Recent text-to-video systems increasingly use diffusion models to generate high-quality and diverse videos from textual prompts.
- Figure 2 depicts DreamVideo’s two-stage pipeline for learning subject identity and motion with a frozen pretrained video diffusion model.
- Customized video generation remains less explored than customized image generation, despite substantial progress in image customization.
- Dreamix applies a DreamBooth-like paradigm to subject-driven video generation but can overfit and produce small or missing motions.
3. Methodology
DreamVideo builds on video diffusion models and separates subject learning from motion learning through lightweight bottleneck adapters. Appearance guidance, selective conditioning, and parameter analysis are used to preserve identity while modeling motion.
- 3.1. Preliminary: Video Diffusion Models: Video diffusion models generate videos by progressively denoising Gaussian-sampled variables while predicting timestep-specific noise from text conditions.
- 3.2. DreamVideo: Both adapters use bottleneck structures, while the motion adapter adds a condition layer to incorporate appearance guidance.
- 3.2. DreamVideo: The proposed task combines a few subject images with one or more motion videos to generate videos containing both the specified subject and motion.
- 3.2. DreamVideo: Subject learning first optimizes a pseudo-word textual identity, then trains an identity adapter while freezing the text embedding and video diffusion model.
- 3.2. DreamVideo: Cross-attention changes most during spatial subject learning, whereas parameter contributions are similar across layers during temporal motion learning.
- 3.2. DreamVideo: Appearance guidance is incorporated into motion learning so the motion adapter focuses on motion rather than appearance features from training videos.
4. Experiment
DreamVideo is evaluated for joint, subject-only, motion-only, and component-level customization. Across qualitative, quantitative, user, and ablation studies, it preserves identities and motions while supporting diverse, text-aligned videos.
- Joint customization: DreamVideo harmoniously combines arbitrary subjects and motions, whereas AnimateDiff, ModelScopeT2V, and LoRA exhibit motion or fusion failures.AnimateDiff preserves appearance but produces limited motion diversity; ModelScopeT2V and LoRA can corrupt identities or damage motions.
- Joint customization: DreamVideo outperforms other methods across CLIP-T, CLIP-I, and DINO-I, while remaining comparable to Dreamix in Temporal Consistency with fewer parameters.AnimateDiff achieves the highest Temporal Consistency but tends to generate small motions.
- Subject customization: DreamVideo preserves precise subject appearances under varied text contexts while mitigating the small-motion and prompt-content problems observed in competing methods.Textual Inversion struggles with accurate appearance reconstruction, while Dreamix can overfit and omit target objects such as “pizza.”
- Motion customization: DreamVideo captures target motion patterns without appearance coupling, generating temporally consistent and diverse videos.ModelScopeT2V fuses training-video appearance, whereas Tune-A-Video suffers from frame discontinuity.
- User study: Users most preferred DreamVideo across Text Alignment, Subject Fidelity, Motion Fidelity, and Temporal Consistency.The combination study used 50 video groups covering five motion patterns and ten subjects, evaluated by five annotators.
- Ablation studies: Removing textual identity or appearance guidance slightly lowers all metrics, while subject customization improves CLIP-I and DINO-I and the motion adapter improves CLIP-T and Temporal Consistency.These results support distinct contributions from subject and motion components.
5. Conclusion
DreamVideo decouples subject and motion learning through lightweight identity and motion adapters, enabling flexible customization with collected subjects and motion patterns. Its demonstrated scope covers joint and individual customization, but not multiple subjects with multiple motions.
- Conclusion: DreamVideo combines textual inversion and identity-adapter tuning for subjects with a motion adapter and appearance guidance for motions.The method is evaluated using 20 subjects and 30 motion patterns.
- Conclusion: Experiments demonstrate DreamVideo’s efficiency and flexibility for joint and individual customization of subjects and motions.The conclusion reports extensive qualitative and quantitative evaluation.
- Limitations: DreamVideo currently handles one subject with one motion, not customized videos containing multiple subjects with multiple motions.The paper suggests fusion modules or a general customized video model as possible solutions.
A. Experimental Details
The experiments use a fine-tuned video diffusion base and implement each baseline with specified subject, motion, and inference procedures. DreamVideo uses lightweight identity and motion adapters with short training schedules.
- The fine-tuned ModelScopeT2V model serves as the base video diffusion model for most methods after 30k iterations on about 30,000 text-video pairs.
- Subject Customization: DreamVideo optimizes textual identity for ∼3000 iterations, trains an identity adapter for ∼800 iterations, and uses a half-input hidden dimension.Identity-adapter training takes ∼12 minutes on one A100 GPU.
- Subject Customization: Custom Diffusion jointly optimizes cross-attention key and value matrices with the textual identity embedding for 250 iterations at learning rate 4.0 × 10−5.
- Motion Customization: DreamVideo trains its motion adapter for ∼1000 iterations at learning rate 1.0 × 10−5, taking ∼15 minutes for one video and ∼30 minutes for multiple videos on one A100 GPU.
- Inference: During inference, DreamVideo combines the trained identity and motion adapters without additional training and uses a randomly selected training image as appearance guidance.
- Evaluation: The evaluation tables report CLIP-T, CLIP-I, DINO-I, Temporal Consistency, and parameter number for quantitative comparisons.
B. More Results
Additional experiments compare DreamVideo with baselines for subject and motion customization, using qualitative results, human evaluations, and adapter comparisons. Across the reported evaluations, DreamVideo preserves subject identity and motion while outperforming alternatives.
- Qualitative Comparison: DreamVideo preserves both subject identity and motion, whereas AnimateDiff produces little motion and other methods struggle with fusion conflicts.
- Subject Customization: DreamVideo outperforms Custom Diffusion while retaining subject identity, following diverse contextual descriptions, and using fewer parameters.
- Subject Customization: Additional subject results demonstrate favorable generalization across more subjects.
- Motion Customization: DreamVideo shows superior motion customization on both multiple-video and single-video training settings while ignoring training-video appearance information.
- User Study: Human evaluations cover Text Alignment, Subject Fidelity or Motion Fidelity, and Temporal Consistency across generated videos from subjects and motion patterns.The motion study uses 120 videos from 20 motion patterns and six text prompts.
- Adapter Comparison: Quantitative comparisons report superior video-customization performance for the adapter compared with LoRA, with Temporal Consistency and parameter number included.
- User Study: DreamVideo consistently outperforms other methods on all reported human-evaluation metrics for motion customization.
C.1. More Qualitative Results
Ablation and adapter studies examine appearance guidance, parameter matching, and adapter placement. The results favor appearance guidance, adapters over LoRA, and parallel adapters across all layers.
- Component Ablation: Without appearance guidance, generated videos may learn noise, artifacts, backgrounds, and other subject-unrelated information from training videos.
- Adapter Versus LoRA: The adapter is evaluated against LoRA using a reduced hidden dimension of 32 and without textual identity or appearance guidance for comparable parameter counts.
- Adapter Versus LoRA: Adapters alleviate fusion conflicts and produce a more harmonious combination when spatial and temporal components are merged.
- Adapter Versus LoRA: The quantitative comparison reports superior video-customization performance for the adapter compared with LoRA.
- Adapter Design: Among serial and parallel adapter combinations across self-attention and feed-forward layers, parallel adapters on all layers achieve the best performance.
D. Social Impact and Discussions
DreamVideo aims to make video customization more flexible by combining user-specified subjects and motions while requiring only a few images or videos. The discussion identifies boundaries involving base-model capabilities, fine single-video motion, and multiple-object combinations.
- Social Impact: DreamVideo supports arbitrary combinations of customized subjects and motions, as well as separate subject or motion customization.
- Limitations: The method is limited by the inherent capabilities of the base model, such as failing to generate a wolf riding a bicycle.
- Limitations: For fine single-video motion, DreamVideo may learn only a similar rough motion pattern rather than frame-by-frame correspondence.
- Limitations: Difficult multi-object combinations remain challenging because the method can confuse object identities and transfer one subject’s characteristics to another.
- Supplementary Results: Supplementary figures present additional subject, motion, joint-customization, ablation, adapter, and failure-case results.