Source-linked AI summary
MotionDirector: Motion Customization of Text-to-Video Diffusion Models
Rui Zhao, Yuchao Gu, Jay Zhangjie Wu, David Junhao Zhang, Jiawei Liu, Weijia Wu, Jussi Keppo, Mike Zheng Shou
TL;DR
Motion customization seeks to adapt text-to-video diffusion models to specified motions without coupling those motions to limited reference appearances. MotionDirector uses dual-path LoRAs and an appearance-debiased temporal loss to separate appearance from motion, and experiments show diverse-appearance motion generation with downstream video-mixing and image-animation applications.
Problem
Existing motion-customization adaptations can couple learned motion with limited training-video appearances, making generalization to other appearances difficult.
Method
MotionDirector decouples appearance and motion using a dual-path LoRAs architecture and an appearance-debiased temporal loss.
Results
Across two benchmarks, MotionDirector generates diverse appearances for customized motions and outperforms controllable-generation and tuning-based methods; on UCF Sports Action, raters preferred its motion fidelity at least 75% of the time versus 25% for base models.
Takeaways & Limitations
The decoupled representation supports mixing appearance from one video with motion from another and animating a single image with learned motions.
Takeaways & Limitations
MotionDirector remains limited in learning complex motions involving multiple subjects, such as groups of boys playing soccer.
Abstract
from arXiv · showhide
Large-scale pre-trained diffusion models have exhibited remarkable capabilities in diverse video generations. Given a set of video clips of the same motion concept, the task of Motion Customization is to adapt existing text-to-video diffusion models to generate videos with this motion. For example, generating a video with a car moving in a prescribed manner under specific camera movements to make a movie, or a video illustrating how a bear would lift weights to inspire creators. Adaptation methods have been developed for customizing appearance like subject or style, yet unexplored for motion. It is straightforward to extend mainstream adaption methods for motion customization, including full model tuning, parameter-efficient tuning of additional layers, and Low-Rank Adaptions (LoRAs). However, the motion concept learned by these methods is often coupled with the limited appearances in the training videos, making it difficult to generalize the customized motion to other appearances. To overcome this challenge, we propose MotionDirector, with a dual-path LoRAs architecture to decouple the learning of appearance and motion. Further, we design a novel appearance-debiased temporal loss to mitigate the influence of appearance on the temporal training objective. Experimental results show the proposed method can generate videos of diverse appearances for the customized motions. Our method also supports various downstream applications, such as the mixing of different videos with their appearance and motion respectively, and animating a single image with customized motions. Our code and model weights will be released.
1 INTRODUCTION
Motion Customization adapts text-to-video diffusion models to generate specified motions while generalizing beyond the limited appearances in reference videos. MotionDirector addresses this challenge by decoupling appearance and motion with dual-path LoRAs and an appearance-debiased temporal loss.
- Motion Customization adapts pre-trained text-to-video diffusion models to generate videos exhibiting motions represented by reference videos.
- Existing tuning strategies can couple learned motion with reference-video appearances, limiting generalization to other appearances.
- MotionDirector uses dual-path LoRAs to learn appearance and motion separately while keeping pre-trained model parameters fixed.
- MotionDirector generalizes customized motions to diverse appearances, mixes appearance and motion from different videos, and animates single images.
- On UCF Sports Action, human raters preferred MotionDirector for higher motion fidelity at least 75% of the time versus 25% for base models.
- An appearance-debiased temporal loss reduces appearance influence on the temporal training objective.
- Experiments on two benchmarks show MotionDirector works across foundation models and outperforms controllable-generation and tuning-based methods.
2 RELATED WORK
Related work spans text-to-video generation, customization of pre-trained generation models, and controllable video generation. These lines situate MotionDirector among diffusion-based generation and adaptation approaches.
- Text-to-Video Generation: Text-to-video generation has used GANs, autoregressive models, implicit neural representations, and diffusion-based models.
- Text-to-Video Generation: Diffusion-based video methods train conditional 3D U-Nets to denoise randomly sampled Gaussian-noise sequences.
- Generation Model Customization: Generation-model customization adapts pre-trained foundation models to user preferences while retaining their learned generation knowledge.
- Controllable Video Generation: Controllable video generation aligns outputs with explicit signals such as depth maps, human poses, and optical flows.
3 METHODOLOGY
MotionDirector adapts a frozen video diffusion model through separate spatial and temporal LoRAs, using shared spatial weights and appearance-debiased temporal training to isolate motion from appearance. The trained temporal LoRAs are then injected at inference to generate diverse videos exhibiting the learned motion.
- Dual-Path Low-Rank Adaptions: MotionDirector keeps base diffusion weights fixed and uses a dual-path architecture with trainable spatial LoRAs for appearance and trainable temporal LoRAs for motion.The temporal path reuses the spatial-path LoRA weights in its spatial transformers.
- Low-Rank Adaption: LoRA updates frozen pre-trained weights with low-rank factors, requiring fewer computing resources than full-network fine-tuning and supporting plug-and-play deployment.The low-rank factors have rank r much smaller than the original dimensions.
- Video Diffusion Model: A 3D U-Net denoises video latent codes conditioned on text, using spatial transformers for pixel correlations and temporal transformers for frame consistency.The latent code z_t has batch, frame, width, height, and channel dimensions.
- Spatial LoRAs Training: Spatial LoRAs are trained on randomly sampled individual frames, fitting appearance while ignoring motion through a spatial loss.A frame is sampled from the training video at each step.
- Temporal LoRAs Training: Temporal LoRAs update frame-wise correlations while shared spatial LoRAs suppress appearance information, but temporal noise prediction remains partly appearance-dependent.The temporal path initially uses all training frames with the original temporal loss.
- Temporal LoRAs Training: The appearance-debiased temporal loss removes appearance bias from noises and predicted noises while retaining motion-related connectivity structure.The debiasing uses an anchor frame and a strength factor β; the temporal objective combines original and appearance-debiased temporal losses.
- Motion Customization: At inference, trained temporal LoRAs are injected into the pre-trained video diffusion model to generate diverse videos with motions learned from single videos or motion-concept collections.A single video can specify a particular motion, while multiple videos can define concepts such as lifting weights or playing golf.
4 EXPERIMENTS
Experiments evaluate MotionDirector against foundation models, coupled LoRA tuning, Tune-A-Video, and controllable-generation methods using automatic and human measures. Results indicate that dual-path training preserves appearance diversity while improving motion fidelity and generalizing customized motions across appearances.
- Experimental Setup: MotionDirector is compared with vanilla foundation models, coupled LoRA tuning, Tune-A-Video, and controllable-generation methods across two text-to-video foundation models.The experiments use adapted UCF Sports Action data and LOVEU-TGVE benchmark settings.
- Qualitative Results: In golf examples, MotionDirector’s dual-path LoRAs preserve the requested monkey appearance while learning the realistic club-swinging motion.The appearance-debiased temporal loss further enhances learning of the desired motion; similar behavior is reported for riding a panda.
- Quantitative Results: Coupled tuning destroys pre-trained appearance diversity, whereas MotionDirector preserves it and achieves the highest motion fidelity.Automatic and human evaluations are used to assess appearance diversity, temporal consistency, and motion fidelity.
- Single-Video Results: On LOVEU-TGVE, MotionDirector outperforms other methods by a large margin on appearance diversity while achieving competitive motion fidelity.Text alignment uses the original four editing prompts, whereas appearance diversity uses three newly introduced prompts with significant changes.
- Efficiency: MotionDirector adds 9 million spatial-LoRA and 12 million temporal-LoRA parameters, requiring 14 GB VRAM and 20 minutes to converge on multiple videos.For a single reference video, convergence takes 8 minutes; ZeroScope has over 1.8 billion pre-trained parameters.
5 LIMITATIONS AND FUTURE WORKS
MotionDirector can learn motions from one or two subjects, but complex motions involving multiple subjects remain difficult. Future work could decouple different subjects’ motions in latent space and learn them separately.
- Limitations and Future Work: MotionDirector struggles with complex multi-subject motions, such as a group of boys playing soccer.The authors suggest separately decoupling and learning different subjects’ motions in the latent space.
6 CONLCUSION
MotionDirector addresses Motion Customization by decoupling appearance and motion learning with a dual-path architecture and appearance-debiased temporal objective. Evaluations report diverse-appearance generation, motion generalization, and improved appearance diversity and motion fidelity over other methods.
- MotionDirector introduces Motion Customization for adapting pre-trained text-to-video diffusion models to generate desired motions.
- Its dual-path architecture and appearance-debiased temporal training objective decouple the learning of appearance and motion.
- MotionDirector learns desired motion concepts or subject- and camera-specific motions and generalizes them to diverse appearances.
- Automatic and human evaluations on two benchmarks show MotionDirector outperforms other methods in appearance diversity and motion fidelity.
A.1 COMPARISON OF TASKS
Motion Customization learns motion from reference videos so generated videos preserve that motion while allowing varied appearances. This differs from appearance customization, which transfers appearance while permitting different motions, and from controllable generation using control signals.
- Motion Customization adapts pre-trained models to generate videos exhibiting a motion concept represented by reference videos.
- Appearance customization: Appearance customization learns subjects or styles from reference videos and generates videos with those appearances but potentially different motions.
- Controllable generation: Controllable generation targets videos with the same motion represented by control signals, whose constraints can limit the control setup.
A.2 TRAINING AND INFERENCE DETAILS
The implementation trains LoRAs with fixed hyperparameters and combines original temporal and appearance-debiased losses equally. Inference uses 30 diffusion steps, classifier-free guidance, and a DDPM scheduler.
- Training: LoRA training uses Adam with learning rate 1e-4, weight decay 5e-4, dropout 0.1, and rank 32.
- Training: The original temporal loss and appearance-debiased loss each receive coefficient 1.
- Training: Training samples 16 frames at 384 × 384 spatial resolution.
- Inference: Inference uses 30 diffusion steps and classifier-free guidance scale 12 with the DDPM scheduler for training and inference.
A.3 DETAILS OF BENCHMARKS
The paper evaluates motion customization on multiple-video and single-video benchmarks. The former uses filtered UCF Sports Action clips with refined golf-view concepts, while the latter extends a LOVEU-TGVE benchmark with additional appearance-changing prompts.
- Multiple-video benchmark: The multiple-video benchmark uses UCF Sports Action, originally containing 150 videos across 10 human motions.
- Multiple-video benchmark: After removing inconsistent and low-resolution videos, the benchmark splits playing golf into side-, front-, and back-view motion concepts and retains 95 reference videos.
- Single-video benchmark: The single-video benchmark uses 76 LOVEU-TGVE videos, each originally paired with four editing prompts.
- Single-video benchmark: Three additional prompts per video introduce larger appearance changes to test text-guided generation diversity.
- Figure 8 presents examples of the multiple-video and single-video motion-customization benchmarks.
A.4 DETAILS OF HUMAN EVALUATIONS
The human evaluation compares anonymous generated videos on text alignment, temporal consistency, and motion fidelity, using consensus voting to reduce rating noise. Additional figures provide qualitative results and method comparisons for motion customization with single or multiple reference videos.
- Evaluation protocol: Each Amazon MTurk task compares two anonymous generated videos on text alignment, temporal consistency, and motion fidelity.Motion fidelity is judged against a reference video.
- Evaluation protocol: Over 1800 comparison tasks were completed by 5 raters per task, with responses from 421 unique human raters.The statistical results are reported in Tables 1 and 2.
- Evaluation protocol: Consensus voting counts a result as valid only when at least 3 of 5 raters select the same choice.This procedure is used to eliminate noise in the rated results.
- Metric terminology: For multiple-reference evaluation, “appearance diversity” denotes text alignment under prompts with substantial appearance changes, such as changing a human to a panda.For single-reference evaluation, the terminology distinguishes original-prompt text alignment from appearance diversity under newly labeled prompts with significant changes.
- Additional results: Figures 10 and 11 show additional MotionDirector results for multiple and single reference videos, while Figure 12 compares MotionDirector with other controllable-generation and tuning-based methods.These figures extend the qualitative and comparative evaluation beyond the main reported results.