Source-linked AI summary
Follow-Your-Motion: Video Motion Transfer via Efficient Spatial-Temporal Decoupled Finetuning
Yue Ma, Yulong Liu, Qiyuan Zhu, Ayden Yang, Kunyu Feng, Xinhua Zhang, Zexuan Yan, Zhifeng Li, Sirui Han, Chenyang Qi, Qifeng Chen
TL;DR
DiT-based motion transfer remains limited by motion inconsistency from spatial-temporal coupling and inefficient two-stage LoRA tuning. Follow-Your-Motion decouples spatial and temporal adaptation, accelerates temporal tuning with sparse sampling and adaptive RoPE, and evaluates the approach on MotionBench, where it achieves state-of-the-art quality across diverse motions.
Problem
DiT-based motion transfer suffers from motion inconsistency and tuning inefficiency because naive methods couple spatial and temporal adaptation and process excessive video data.
Method
Follow-Your-Motion uses spatial-temporal decoupled LoRA with head-specific adaptation, plus sparse motion sampling and adaptive RoPE for efficient motion transfer.
Results
Follow-Your-Motion achieves state-of-the-art video motion-transfer quality across extensive evaluations on MotionBench and improves motion fidelity by about 48.3% with adaptive RoPE.
Takeaways & Limitations
The framework enables high-fidelity videos that faithfully follow reference motion while adapting powerful video Diffusion Transformers to complex motion.
Abstract
from arXiv · showhide
Recently, breakthroughs in the video diffusion transformer have shown remarkable capabilities in diverse motion generations. As for the motion-transfer task, current methods mainly use two-stage Low-Rank Adaptations (LoRAs) finetuning to obtain better performance. However, existing adaptation-based motion transfer still suffers from motion inconsistency and tuning inefficiency when applied to large video diffusion transformers. Naive two-stage LoRA tuning struggles to maintain motion consistency between generated and input videos due to the inherent spatial-temporal coupling in the 3D attention operator. Additionally, they require time-consuming fine-tuning processes in both stages. To tackle these issues, we propose Follow-Your-Motion, an efficient two-stage video motion transfer framework that finetunes a powerful video diffusion transformer to synthesize complex motion. Specifically, we propose a spatial-temporal decoupled LoRA to decouple the attention architecture for spatial appearance and temporal motion processing. During the second training stage, we design the sparse motion sampling and adaptive RoPE to accelerate the tuning speed. To address the lack of a benchmark for this field, we introduce MotionBench, a comprehensive benchmark comprising diverse motion, including creative camera motion, single object motion, multiple object motion, and complex human motion. We show extensive evaluations on MotionBench to verify the superiority of Follow-Your-Motion.
1 INTRODUCTION
Motion transfer seeks to reapply reference-video dynamics while separating motion from appearance, but DiT-based tuning struggles with spatial-temporal coupling and inefficiency. Follow-Your-Motion addresses these issues with decoupled adaptation, sparse sampling, adaptive RoPE, and MotionBench evaluation.
- Motivation: Motion transfer synthesizes novel videos that replicate reference camera movements and object trajectories while emphasizing motion patterns over low-level appearance.
- Related approaches: Training-free methods manipulate intermediate representations without changing parameters, whereas tuning-based methods optimize parameters to encode complex reference motion.
- Challenges: Naive two-stage LoRA tuning couples spatial appearance with temporal dynamics and can produce motion inconsistency during reconstruction and transfer.
- Challenges: Indiscriminately tuning all DiT attention heads creates parameter redundancy, while processing every reference frame adds computational overhead despite latent temporal interpolation.
- Follow-Your-Motion: Follow-Your-Motion classifies attention heads into spatial and temporal types, updating each type only in its corresponding tuning stage to preserve motion consistency.
- Follow-Your-Motion: Sparse motion sampling and adaptive RoPE improve tuning efficiency and motion interpolation, with sparse sampling achieving a 3.89× speed-up over full-frame tuning.
- Evaluation: MotionBench evaluates single-object, complex human, multi-object, and camera motions across diverse scenes and styles, while reported experiments show state-of-the-art performance.
2 RELATED WORK
Video generation research has progressed from UNet temporal modules to scalable Diffusion Transformers, while motion-transfer methods increasingly use implicit control through training-free or tuning-based paradigms.
- Text-to-video generation: Text-to-video generation aims to produce realistic videos matching both spatial visuals and temporal dynamics specified by an input prompt.
- Text-to-video generation: Diffusion Transformers such as Sora, CogVideoX, HunyuanVideo, and Wan2.1 provide scalable video generation with strong quality and consistency.
- Video motion transfer: Motion-transfer methods use explicit control signals or implicit control, with implicit approaches divided into training-free and tuning-based paradigms.
- Video motion transfer: Explicit-control methods require large control-signal datasets and substantial computational resources, while training-free methods extract motion embeddings during inference.
3 METHOD
Follow-Your-Motion decouples spatial appearance and temporal motion processing in unified 3D attention, then tunes the two branches separately. Sparse motion sampling and adaptive RoPE reduce temporal tuning cost while supporting motion interpolation.
- Motivation: Naive two-stage LoRA tuning causes appearance leakage and remains expensive for DiT-based video diffusion models.After 3,000 optimization steps, motion fidelity remains unsatisfactory; the reported runtime is 3,042s on one H20 GPU.
- Motivation: Unified 3D attention makes temporal-parameter disentanglement difficult, while full-video tuning increases sequence length and computation.The cited example uses 24,276 tokens for 81 frames, and naive LoRA involves 29.5 M parameters.
- Attention classification: Attention heads are classified as spatial or temporal by comparing input attention sparsity with pseudo spatial and temporal attention maps.A head is classified as temporal when Sims < α · Simt, with α = 1.25.
- Dual attention fusion: The method rearranges q, k, v, and o channels into parallel temporal and spatial branches, then fuses their outputs after attention processing.Query, key, and value features from both branches are concatenated along the channel dimension before rotary position embedding and scaled dot-product attention.
- Spatial-temporal tuning: Spatial LoRA is tuned on randomly sampled single frames, while temporal LoRA is tuned after freezing the converged spatial LoRA.Spatial tuning targets spatial-head projections; temporal tuning targets temporal-head projections.
- Efficient temporal tuning: Sparse motion sampling tunes temporal LoRA on 17 frames instead of 81, while adaptive RoPE preserves the original frame-position range during inference.Motion loss uses differences between consecutive frame latents and is combined with video denoising loss for temporal tuning.
- Qualitative examples: The framework targets single-object, multiple-object, complex-human, and camera motion, with galleries showing generated clips that follow reference motion.The examples include single-object and multiple-object motion categories.
4 EXPERIMENTS
Follow-Your-Motion is evaluated on MotionBench through qualitative comparisons, quantitative comparisons, and ablations. The experiments show strong motion-transfer performance across diverse scenarios while the proposed modules improve fidelity, appearance preservation, and tuning efficiency.
- MotionBench: MotionBench contains 200 approximately five-second videos spanning camera, single-object, multiple-object, and complex human motion.Thirty percent are generated by text-to-video models, with the remainder collected from publicly licensed video websites.
- Qualitative comparison: Follow-Your-Motion maintains source motion more effectively than prior methods in single-object, multi-object, and camera-motion cases.The method preserves consistent movement patterns and spatial relationships between multiple moving subjects.
- Quantitative comparison: The quantitative evaluation compares training-free and tuning-based methods on MotionBench using time, motion fidelity, temporal consistency, text similarity, and user studies.Evaluations use 32 frames at 512 × 512 resolution because previous methods support limited video lengths.
- Ablation study: Spatial–temporal decoupled LoRA preserves motion while modifying appearance, improving text similarity compared with jointly tuning spatial and temporal attention heads.Without decoupling, the edited tiger appears unnaturally black because the dog’s appearance is not separated effectively.
- Ablation study: Adaptive RoPE keeps sparsely sampled frames aligned with their original indices and improves motion fidelity by about 48.3%.Removing adaptive RoPE causes disordered tiger motion that fails to match the source dynamics.
- Ablation study: Sparse motion sampling reduces tuning time to 727s while retaining nearly identical motion fidelity to tuning on all frames.The reported motion fidelity is 0.975 with sparse sampling versus 0.971 without it.
5 CONCLUSION
Follow-Your-Motion is a three-stage framework for complex-motion video transfer that improves motion consistency and reduces tuning inefficiency in video diffusion transformers.
- Follow-Your-Motion tunes a video diffusion transformer to synthesize video clips with complex motion.
- Efficient spatial-temporal decoupled LoRA improves motion consistency during transfer.
- Adaptive RoPE and sparse motion sampling accelerate the training process.
- Extensive experiments show state-of-the-art video motion transfer quality against a wide range of previous methods.