Source-linked AI summary
RoPECraft: Training-Free Motion Transfer with Trajectory-Guided RoPE Optimization on Diffusion Transformers
Ahmet Berke Gokmen, Yigit Ekin, Bahri Batuhan Bilecen, Aysegul Dundar
TL;DR
Text conditioning in diffusion transformers often leaves detailed video dynamics underspecified. RoPECraft transfers motion by modifying rotary positional embeddings and optimizing them during generation with flow matching and phase regularization. It reports strong benchmark performance, including the highest MF score and lowest CD-FVD among evaluated methods.
Problem
Text conditioning in diffusion transformers is often too ambiguous to specify detailed dynamics such as body movement, camera motion, or interactions.
Source: §1 · p. 2Method
RoPECraft warps rotary positional embeddings with optical-flow displacements and applies flow-matching optimization and phase regularization during generation, without backbone fine-tuning.
Source: §1 · p. 2Results
RoPECraft achieves the highest MF score (0.5816) and lowest CD-FVD (1284.58), surpassing GWTF by +0.0103 MF and -200.6 CD-FVD.
Source: §5.4 · p. 8Takeaways & Limitations
The paper reports high-quality motion-transfer performance across multiple benchmarks using motion-augmented RoPE, flow-matching optimization, and phase-based regularization.
Source: §6 · p. 10The framework's future development includes handling motion with extreme occlusion and improving high-frequency details in generated videos.
Source: §6 · p. 10
Abstract
from arXiv · showhide
We propose RoPECraft, a training-free video motion transfer method for diffusion transformers that operates solely by modifying their rotary positional embeddings (RoPE). We first extract dense optical flow from a reference video, and utilize the resulting motion offsets to warp the complex-exponential tensors of RoPE, effectively encoding motion into the generation process. These embeddings are then further optimized during denoising time steps via trajectory alignment between the predicted and target velocities using a flow-matching objective. To keep the output faithful to the text prompt and prevent duplicate generations, we incorporate a regularization term based on the phase components of the reference video's Fourier transform, projecting the phase angles onto a smooth manifold to suppress high-frequency artifacts. Experiments on benchmarks reveal that RoPECraft outperforms all recently published methods, both qualitatively and quantitatively.
1 Introduction
Text conditioning in diffusion transformers often leaves detailed motion underspecified, while existing motion-control approaches face domain-shift, latent-distribution, or computational challenges. RoPECraft modifies positional embeddings and combines motion guidance with optimization and regularization, without backbone fine-tuning.
Motivation
- Text conditioning often cannot specify detailed dynamics such as body movement, camera motion, or interactions.The paper identifies demand for more precise and controllable motion synthesis as generative quality improves.Source: §1 · p. 2
Prior limitations
- GWTF requires around 40 GPU days of fine-tuning, while DiTFlow incurs high computational costs from full-size attention-based feature computation.The paper also notes that directly warping noise disrupts the pretrained DiT's intended latent distribution.Source: §1 · p. 2
Contributions
- RoPECraft combines optical-flow-warped rotary positional embeddings, flow-matching optimization, and phase regularization for training-free motion transfer.The method updates positional embeddings rather than the latent space and requires no backbone fine-tuning.Source: §1 · p. 2
- Fréchet Trajectory Distance (FTD) is introduced to quantify motion alignment between generated and reference videos.The paper presents it as an evaluation metric alongside its motion-transfer method.Source: §1 · p. 2
Motivation
- Figure 2 contrasts failed latent warping without fine-tuning and inadequate latent optimization with RoPECraft's successful transfer while keeping the latent space intact.The caption identifies the outcomes in Columns 2, 3, and 4, respectively.Source: Figure 2 · p. 2
2 Related Work
Related work progresses from explicit structural guidance and learned motion representations to methods using signals from generative-model features, trajectories, or latents. RoPECraft builds on test-time positional-embedding updates while keeping the DiT backbone frozen.
Text-to-video models
- DiT pipelines have gained attention for temporal modeling and video quality, following earlier U-Net approaches with temporal modules or inflated convolutions.The paper adopts a DiT backbone.Source: §2 · p. 3
Motion transfer
- Motion-transfer methods have used explicit structures, learned motion embeddings, backbone motion signals, reference trajectories, or warped initial noise.GWTF warps initial noise using reference optical flow and fine-tunes the DiT; DiTFlow updates latents or positional embeddings at test time.Source: §2 · p. 3
Positional embeddings in vision transformers
- RoPE encodes relative spatial or temporal relationships by rotating query and key values according to patch positions.The paper situates its on-the-fly RoPE updates within positional-embedding work for vision transformers.Source: §2 · p. 3
3 Preliminaries
The preliminaries introduce flow matching as a velocity-field approach and RoPE as a positional encoding based on rotations of query and key values. These concepts underpin the paper's generation-time motion guidance.
3.1 Flow matching
- Flow matching uses a deterministic, time-dependent velocity field to transform a base distribution into a target distribution.It minimizes discrepancy between model and target velocities; the paper describes transport along a predefined probability path as mass-preserving.Source: §3.1 · p. 3
3.2 Rotary position embeddings (RoPE)
- RoPE encodes position by rotating query and key values in the complex plane, placing positional information in the phase of frequency components.The paper connects attention patterns to motion control and uses RoPE for motion transfer.Source: §3.2 · p. 3
4 Methodology
RoPECraft modifies spatial RoPE using optical-flow displacements, refines the embeddings with flow-matching optimization, and adds Fourier-phase regularization. The method’s qualitative results indicate that phase constraints fix duplicate generations and artifacts.
4 Methodology
- The methodology combines optical-flow-augmented RoPE, generation-time optimization, and additional constraints to guide motion transfer.The overall architecture is shown in Fig. 3.Source: §4 · p. 4
4.1 Motion-augmented RoPE
- Optical-flow-derived motion signals create row- and column-specific spatial RoPE embeddings, while the temporal component remains unchanged.The resulting embeddings transfer coarse input motion, but using them alone can leave subjects facing the wrong direction or missing challenging trajectories.Source: §4.1 · pp. 4–5
4.2 Optimization with flow-matching
- Early-step flow-matching optimization aligns predicted and target velocities; initialized with motion-augmented RoPE, it reduces artifacts and corrects subject orientation and trajectories.Optimization without that dedicated initial condition improves subject placement but can miss motion direction and produce visual artifacts.Source: §4.2 · p. 5
4.3 Phase constraints
- Phase constraints fix duplicate generations and artifacts in the qualitative results.The method adds a phase constraint to the flow-matching objective using Fourier-transformed target and generated velocities.Source: Figure 7 · p. 6, §4.3 · p. 6
5 Experimental Results
RoPECraft leads the quantitative comparison on MF and CD-FVD, while ranking second on CLIP similarity and both FTD variants. The section introduces FTD to measure trajectory geometry, then compares methods qualitatively and quantitatively.
5.1 Metrics and baselines
- The evaluation uses DAVIS videos with four generated prompts per video and assesses fidelity, prompt similarity, and motion alignment against recent motion-transfer baselines.Motion metrics use Co-Tracker3 trajectories; FTD samples foreground points for object motion and foreground plus background points for object and camera motion.Source: §5.1 · pp. 6–7
5.2 Fréchet Trajectory Distance
- FTD tracks foreground and background points with occlusion-aware handling, then computes the RMS discrete Fréchet distance between normalized generated and reference trajectories.It drops tracks whose points never reappear and is resolution-invariant after coordinate normalization.Source: §5.2 · p. 7, Figure 8 · p. 7
5.3 Qualitative evaluation
- Across diverse prompts, RoPECraft generally captures reference motion and subject orientation, while competing methods show motion-direction, prompt-alignment, or artifact failures.The comparisons include failures to preserve stationary subjects and cases where GWTF sacrifices prompt alignment.Source: §5.3 · p. 8
5.4 Quantitative evaluation
- RoPECraft achieves the highest MF score (0.5816) and lowest CD-FVD (1284.58), but ranks second on CLIP similarity and both FTD variants.Against GWTF, its MF score is higher by +0.0103 (approximately +1.8%) and its CD-FVD lower by -200.6 (approximately -13.5%).Source: §5.4 · p. 8
- The ablations support motion-augmented RoPE, flow-matching optimization, and phase constraints; optimization uses (t, s) = (10, 5), since s = 10 significantly decreases visual quality.The authors report Table 2 for component ablations and Table 3 for optimization-step choices.Source: §5.4 · p. 8
5.5 User study
A user study assessed motion judgments and overall visual preference using 20 DAVIS prompt-reference pairs. Participants favored RoPECraft consistently, and FTD aligned with human motion judgments better than MF.
5.5 User study
- The study sampled 20 prompt-reference pairs from DAVIS and generated outputs for all competing methods.The survey separately elicited motion-match selections and overall visual-preference rankings.Source: §5.5 · p. 10
- Participants selected their top three videos for reference-motion match and ranked their top three for overall visual preference.The respective results are presented in Tables 4 and 5.Source: §5.5 · p. 10, Table 4 · p. 10, Table 5 · p. 10
- The user study finds that FTD correlates more strongly with human motion judgments than MF, while RoPECraft is consistently preferred across evaluated visual-quality dimensions.Participants selected top-three videos for motion match and ranked top-three videos for overall visual preference.Source: §5.5 · p. 10
6 Conclusion and Discussion
RoPECraft modifies diffusion-transformer rotary positional embeddings for training-free motion transfer, combining motion-augmented embeddings, flow-matching optimization, and phase-based regularization. The authors identify extreme occlusion and high-frequency detail as cases for future extension.
6 Conclusion and Discussion
- RoPECraft combines motion-augmented RoPE, flow-matching optimization, and phase-based regularization in a training-free diffusion-transformer motion-transfer method.The paper reports high-quality performance across multiple benchmarks and motion-transfer results.Source: §6 · p. 10
- Future work could extend the framework to extreme occlusion, improve high-frequency video details, and support controllable video editing.The authors place discussion of limitations and broader impacts in the supplementary material.Source: §6 · p. 10
A.1 Ablation on Fourier Features
The Fourier-feature ablation finds the phase-based constraint more effective than the magnitude-based constraint. The authors set λ to 1.0 because sweeping it produced no significant changes.
A.1 Ablation on Fourier Features
- The phase-based constraint is more effective than the magnitude-based constraint, with λ set to 1.0 across experiments because sweeping it yielded no significant changes.The comparison uses two Fourier regularizers added to the flow-matching objective; the figure shows examples for prompts P1 and P2.Source: §A.1 · p. 14, Figure 11 · p. 14
A.2 Prompts
The appendix describes how prompts are generated to vary the subject or setting, or paraphrase a scene while preserving its action. It also lists example prompts used in the main-paper figures.
A.2 Prompts
- Prompts are extracted from DAVIS videos using ShareGPT4V, then LLAMA 3.2 generates varied prompts under the three system prompts.A paraphrased prompt is also provided for reconstruction.Source: §A.2 · p. 14
- Three system prompts respectively replace the main subject, relocate the scene, or paraphrase the prompt while preserving specified scene details.Subject replacement keeps the action, environment, camera movements, and style; setting changes retain the subject and actions.Source: §A.2 · p. 14
- The listed motion prompts include changed subjects and settings, such as a helicopter at a beach promenade, a sports car in a cliffside town, and a woman in a bookstore.Other examples include a motorcycle in a garage and a man sprinting on a beach.Source: §A.2 · p. 15
- Additional examples describe a motorcycle rider, backpacker, van, and goose, illustrating the range of source prompts listed.The examples span coastal, mountainous, urban, and riverside settings.Source: §A.2 · p. 15
A.3 Hyperparameters and Computational Requirements
This appendix reports implementation settings, outlines the Fréchet Trajectory Distance implementation, and discusses limitations and further evaluations. On RealCamVid, the authors report robustness on more complex videos than DAVIS.
A.3 Hyperparameters and Computational Requirements
- The experiments use Wan2.1-1.3B and re-implement several baselines on that backbone, while GWTF uses its original CogVideoX weights and a 2B checkpoint.ConMo uses ground-truth DAVIS masks; GWTF's full Wan2.1 training was infeasible given computational resources.Source: §A.3 · p. 15
- The reported hyperparameter search uses Adam with default β parameters and specifies settings for DiTFlow, MOFT, SMM, ConMo, and RoPECraft.The settings include learning rate, block index where applicable, optimization steps, early denoising steps, and method-specific parameters.Source: §A.3 · p. 15
A.4 Fréchet Trajectory Distance
- The appendix provides Fréchet Trajectory Distance pseudocode, including trajectory-consistency comparison and track filling and dropping.The text identifies frechetdist and CoTracker3 as dependencies.Source: §A.4 · pp. 15–16
A.5 Limitations and Broader Impacts
- The method's limitations may be mitigated with a heavier DiT video-generation model or a higher-quality motion extractor; examples include intermittent boat appearance and a distorted final-frame figure.The figure caption attributes the latter to the source video's corresponding frame lacking the person, and notes optical-flow difficulty with occluded or missing subjects.Source: §A.5 · p. 16, Figure 12 · p. 17
- The authors note that realistic video motion may benefit animation, virtual production, education, and accessibility, while synthetic media also poses deepfake and deception risks.They identify responsible use and detection and verification research as relevant concerns.Source: §A.5 · p. 16
A.6 More results on Challenging Videos
- On a randomly sampled RealCamVid subsample containing camera motion and multiple objects, the method is reported as robust on more complex data than DAVIS.Table 6 compares motion-transfer methods on RealCamVid.Source: Table 6 · p. 17, §A.6 · p. 17, Figure 13 · p. 18