Source-linked AI summary
FastVMT: Eliminating Redundancy in Video Motion Transfer
Yue Ma, Zhikai Wang, Tianhao Ren, Mingzhe Zheng, Hongyu Liu, Jiayi Guo, Kunyu Feng, Yuxuan Xue, Zixiang Zhao, Konrad Schindler, Qifeng Chen, Linfeng Zhang
TL;DR
Training-free video motion transfer remains inefficient because existing pipelines recompute redundant attention similarities and gradients. FastVMT uses local sliding-window motion correspondence and step-skipping gradient reuse, achieving a 3.43× average speedup without compromising visual fidelity or temporal consistency.
Problem
Existing training-free motion-transfer pipelines contain considerable motion and gradient redundancy, increasing computation in motion extraction and iterative optimization.
Method
FastVMT combines a sliding-window strategy for local motion correspondence with step-skipping gradient computation that reuses recent gradients.
Results
3.43× average speedup is achieved without compromising visual fidelity or temporal consistency.
Takeaways & Limitations
FastVMT enables high-fidelity video generation with camera trajectories and object motions following the source video.
Takeaways & Limitations
Performance remains bounded by the pretrained video backbone, making highly complex out-of-distribution motions difficult to handle.
Abstract
from arXiv · showhide
Video motion transfer aims to synthesize videos by generating visual content according to a text prompt while transferring the motion pattern observed in a reference video. Recent methods predominantly use the Diffusion Transformer (DiT) architecture. To achieve satisfactory runtime, several methods attempt to accelerate the computations in the DiT, but fail to address structural sources of inefficiency. In this work, we identify and remove two types of computational redundancy in earlier work: motion redundancy arises because the generic DiT architecture does not reflect the fact that frame-to-frame motion is small and smooth; gradient redundancy occurs if one ignores that gradients change slowly along the diffusion trajectory. To mitigate motion redundancy, we mask the corresponding attention layers to a local neighborhood such that interaction weights are not computed unnecessarily distant image regions. To exploit gradient redundancy, we design an optimization scheme that reuses gradients from previous diffusion steps and skips unwarranted gradient computations. On average, FastVMT achieves a 3.43x speedup without degrading the visual fidelity or the temporal consistency of the generated videos.
FASTVMT
FastVMT is presented by Yue Ma and Zhikai Wang among its listed authors.
- Yue Ma and Zhikai Wang are listed as authors of FastVMT.
- The author list also includes Tianhao Ren, Mingzhe Zheng, Hongyu Liu, and Jiayi Guo.
- Kunyu Feng, Yuxuan Xue, Zixiang Zhao, Konrad Schindler, Qifeng Chen, and Linfeng Zhang are also listed.
1 EPIC Lab, Shanghai Jiao Tong University, 2 HKUST, 3 THU, 4 Meta, 5 ETH Zürich
The listed affiliations span EPIC Lab at Shanghai Jiao Tong University, HKUST, THU, Meta, and ETH Zürich.
- Figure 1 describes FastVMT as supporting single-object and multi-object motion, camera ego-motion, and complex articulations.
- The paper marks some authors with equal-contribution and corresponding-author symbols.
1 INTRODUCTION
The introduction motivates training-free motion transfer by identifying redundancy in attention-based motion extraction and iterative gradient computation, then presents FastVMT’s two efficiency strategies and reported speedup.
- Motion transfer generates novel videos from a target prompt while preserving camera and object motion from a reference sequence.
- Training-free approaches avoid per-video fine-tuning and use iterative denoising to extract reference motion before synthesizing the output.
- Existing pipelines contain motion and gradient redundancy because video motion is locally smooth and consecutive gradient updates are similar.
- FastVMT uses a sliding-window strategy with window loss for local motion correspondence instead of token-by-token extraction.
- FastVMT recalculates gradients only at selected iterations and reuses recent values between them.
- 3.43× average speedup is reported without compromising visual fidelity or temporal consistency.
2 RELATED WORK
Related work covers text-to-video generation and video motion transfer, emphasizing evolving generative architectures and distinct motion-control strategies.
- Text-to-video generation: Text-to-video generation seeks realistic videos matching visual content and motion dynamics specified by an input prompt.
- Video motion transfer: Video motion transfer generates novel videos by transferring reference motion while decoupling spatial appearance from temporal motion.
- Video motion transfer: Early motion-transfer approaches use explicit control signals such as poses, depths, and bounding boxes.
3 METHOD
FastVMT is a training-free DiT-based framework that transfers reference-video motion while preserving target content, addressing redundancy in motion extraction and gradient optimization. It combines sliding-window attention motion extraction, corresponding-window loss, and step-skipping gradient reuse to improve efficiency and motion consistency.
- FastVMT generates a prompt-conditioned video that preserves the reference video's camera pose changes and object motion.
- The method targets motion redundancy from global token matching and gradient redundancy from costly, similar gradients across consecutive optimization steps.
- Sliding-window motion extraction: Sliding-window motion extraction restricts query-key interactions to local spatial and temporal neighborhoods, reducing temporal complexity from O(F^2) to O(F).Representative queries estimate window centers, which guide window-constrained Attention Motion Flow computation.
- Corresponding-window loss: The weighted AMF loss aligns reference and generated displacement matrices, while the corresponding-window loss penalizes inconsistent key representations across adjacent frames.Together, these losses target motion-transfer accuracy and temporal stability.
- Step-skipping gradient optimization: Step-skipping gradient optimization reuses cached gradients between selected computations because consecutive latent gradients are highly similar.This reduces computational overhead while maintaining motion-transfer quality.
4 EXPERIMENTS
Experiments evaluate FastVMT with WAN-2.1 on motion-transfer videos using quantitative, qualitative, and ablation comparisons. The results indicate faster motion transfer with preserved motion fidelity, temporal consistency, and video quality.
- Experimental Setup: WAN-2.1 is used as the base model with 50 denoising steps and default outputs of 480 × 832 resolution and 81 frames.Latent tiling and sliding-window AMF guidance are enabled in the stated implementation configuration.
- Comparison with Baselines: FastVMT is compared with state-of-the-art training-free and tuning-based methods on 50 high-quality DAVIS videos using runtime, motion fidelity, and temporal consistency.The comparison uses WAN-2.1 as a common backbone; prior methods are evaluated on 32 frames at 830 × 480 resolution.
- Comparison with Baselines: FastVMT is reported as the fastest method while delivering better performance than training-free methods.Runtime includes inference-time optimization, and the evaluation also measures motion fidelity and temporal consistency.
- Comparison with Baselines: FastVMT significantly outperforms all baselines across subject consistency, motion smoothness, aesthetic quality, and background consistency.These four VBench metrics are evaluated on 40 real-world and 40 generated videos, each paired with five prompts.
- Ablation Study: Removing sliding-window motion extraction degrades multiple metrics and increases computational overhead and inference time.Qualitative results also show reduced motion fidelity and temporal consistency without sliding windows.
- Ablation Study: Excluding corresponding-window loss substantially degrades motion fidelity, whereas step-skipping gradient reuse saves time with negligible impact on motion fidelity and temporal consistency.The corresponding-window loss adds less than 1% processing time, while gradient reuse preserves video generation quality.
5 CONCLUSION
FastVMT addresses motion and gradient redundancy in diffusion-transformer video motion transfer. Its strategies achieve a 3.43× average speedup without compromising visual fidelity or temporal consistency.
- Conclusion: FastVMT is a training-free framework targeting motion redundancy in diffusion transformers and gradient redundancy along the diffusion trajectory.The framework combines sliding-window motion extraction, corresponding-window loss, and step-skipping gradient computation.
- Conclusion: Middle DiT attention layers exhibit stronger token correspondence performance than other examined layers.This observation supports the reported analysis of attention-based motion extraction.
- Conclusion: 3.43× average speedup is achieved without compromising visual fidelity or temporal consistency.The conclusion presents this as the main efficiency result of the proposed strategies.
REPRODUCIBILITY STATEMENT
The work states that its quantitative tables, qualitative images, and video results are reproducible raw model outputs with only minimal format conversion and compression.
- Reproducibility Statement: Results are produced without manual editing or post-hoc alteration, except for minimal format conversion and compression.The authors plan to release inference scripts, example data, and example videos after review.
ETHICS STATEMENT
The paper studies motion-transfer video editing and addresses representational coverage, licensing, release safeguards, and misuse risks. It limits dataset release until watermarking and an ethics/content-safety audit are completed.
- Ethics Statement: The work studies motion-transfer video editing involving people, vehicles, and landscape camera motions.Examples span different races, genders, and styles to mitigate representational bias in demonstrations.
- Ethics Statement: Illustrative videos come from publicly available web content and are used for research while respecting original licenses and terms of service.The statement does not describe public release of the underlying dataset at this stage.
- Ethics Statement: The dataset will not be publicly released before AI-generated watermarking and an ethics/content-safety audit are completed.Future code releases will use visible or machine-detectable watermarking during inference.
- Ethics Statement: The paper prohibits harmful or deceptive uses, including deepfake attacks and other malicious generative behaviors.Future releases will forbid impersonation and harassment and may remove or restrict privacy-, legal-, or safety-sensitive content.
B.1 IMPLEMENTATION DETAILS
FastVMT combines sliding-window motion extraction, corresponding-window guidance, and step-skipping gradient optimization within a DiT-based video motion-transfer pipeline. Experiments include quantitative, qualitative, user-study, ablation, and challenging-motion evaluations.
- Motion extraction: The sliding-window AMF uses a 3×4 token grid, temporal span sf = 5, local search window l = 21, and temperature τ = 1.0.Within the inner loop, Q/K and tile-AMF are recomputed every 3 steps, with cached gradients reused between recomputations.
- Pipeline: FastVMT optimizes the generating latent through attention features, displacement computation, loss evaluation, backpropagation, and denoising over diffusion iterations.The algorithm aligns noise levels, extracts self-attention features, computes displacement matrices, optimizes the latent, and then denoises.
- Window loss: The corresponding-window formulation compares fixed-size target-frame regions across temporal neighborhoods for each spatial tile.Per-tile temporal features are stacked, and the tracking loss penalizes first-order temporal variations of window means across adjacent target frames.
- Qualitative evaluation: FastVMT achieves superior motion consistency over previous works in qualitative comparisons across additional visual results and varied motion types.The reported comparisons include complex object motion, complex camera motion, and other challenging cases.
F.1 LIMITATION
The framework remains constrained by the pretrained video backbone and incurs additional memory use from its pairwise AMF design. The paper identifies long-video scaling and out-of-distribution motion as important boundaries.
- Scope boundary: FastVMT remains bounded by the pretrained video backbone, making out-of-distribution prompts or motions difficult to handle.Highly complex human actions and semantic conflicts between generated content and conditioning prompts can produce degraded or unstable motion transfer.
- Resource limitation: Pairwise AMF increases memory consumption relative to prior methods, which may create practical challenges for long-video generation.The paper notes that this overhead does not critically affect short-video synthesis and suggests engineering optimization, compression, or hierarchical generation as future mitigations.
- Future direction: The paper proposes extending motion transfer toward an agent-based architecture that separately handles motion decomposition, structural alignment, identity preservation, and temporal stabilization.This is presented as a future direction rather than a current capability of FastVMT.