Source-linked AI summary
OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer
Pengze Zhang, Yanze Wu, Mengtian Li, Xu Bai, Songtao Zhao, Fulong Ye, Chong Mou, Xinghui Li, Zhuowei Chen, Qian He, Mingyuan Gao
TL;DR
Existing video customization methods do not fully exploit videos’ spatio-temporal information, often relying on reference images or task-specific temporal priors. OmniTransfer unifies spatio-temporal video transfer through task-aware positional bias, decoupled causal branches, and multimodal alignment, outperforming existing methods across appearance and temporal transfer while matching pose-guided motion transfer without pose.
Problem
Existing methods incompletely exploit spatio-temporal video information, relying on reference images for appearance or task-specific priors and procedures for temporal transfer.
Method
OmniTransfer unifies video transfer using Task-aware Positional Bias, Reference-decoupled Causal Learning, and Task-adaptive Multimodal Alignment.
Results
OmniTransfer outperforms existing methods in ID, style, camera movement, and effect transfer, matches pose-guided motion transfer without pose, and reduces runtime by 20% versus the basemodel architecture.
Takeaways & Limitations
The framework supports flexible, high-fidelity video generation across diverse spatio-temporal transfer tasks.
Takeaways & Limitations
The work relies on self-collected Internet datasets because public datasets containing reference video pairs are unavailable.
Abstract
from arXiv · showhide
Videos convey richer information than images or text, capturing both spatial and temporal dynamics. However, most existing video customization methods rely on reference images or task-specific temporal priors, failing to fully exploit the rich spatio-temporal information inherent in videos, thereby limiting flexibility and generalization in video generation. To address these limitations, we propose OmniTransfer, a unified framework for spatio-temporal video transfer. It leverages multi-view information across frames to enhance appearance consistency and exploits temporal cues to enable fine-grained temporal control. To unify various video transfer tasks, OmniTransfer incorporates three key designs: Task-aware Positional Bias that adaptively leverages reference video information to improve temporal alignment or appearance consistency; Reference-decoupled Causal Learning separating reference and target branches to enable precise reference transfer while improving efficiency; and Task-adaptive Multimodal Alignment using multimodal semantic guidance to dynamically distinguish and tackle different tasks. Extensive experiments show that OmniTransfer outperforms existing methods in appearance (ID and style) and temporal transfer (camera movement and video effects), while matching pose-guided methods in motion transfer without using pose, establishing a new paradigm for flexible, high-fidelity video generation.
1 Introduction
OmniTransfer addresses the limited use of video reference information by unifying spatial appearance and temporal transfer in one framework. It combines multi-frame reference modeling with task-specific designs to support flexible transfer across identity, style, effects, camera movement, and motion.
- Reference videos provide spatial identity and style cues plus temporal camera movement, motion, and effects for more coherent video synthesis.
- Existing methods often rely on single reference images for appearance or predefined priors, inversion, and test-time fine-tuning for temporal control.
- OmniTransfer unifies identity, style, effect, camera movement, and motion transfer while generalizing to compositional combinations.
- Task-aware Positional Bias, Reference-decoupled Causal Learning, and Task-adaptive Multimodal Alignment provide the framework’s three core designs for flexible spatio-temporal transfer.
- OmniTransfer outperforms existing methods on ID, style, camera movement, and effects, matches pose-guided motion transfer without pose, and reduces runtime by 20% versus the basemodel architecture.
2 Related Work
Related work covers appearance transfer through identity and style methods, while temporal customization addresses motion, camera movement, and effects. Existing approaches commonly depend on single images, pose priors, inversion, or test-time fine-tuning.
- Appearance reference: Video identity-transfer methods mostly use single reference images, while some subject-to-video approaches and temporal feature concatenation seek broader appearance preservation.
- Temporal customization: Temporal customization includes motion generation, camera movement, and effect synthesis across GAN-based warping and diffusion-based approaches.
- Temporal customization: Pose-based motion methods can lose appearance cues and hinder extension to multi-person scenarios because of skeleton alignment.
- Temporal customization: More general motion-control methods often rely on diffusion inversion or test-time fine-tuning.
3 Preliminary
The framework builds on Wan2.1 I2V 14B and represents video inputs as concatenated noise, condition, and mask latents. Its DiT blocks use self-attention with 3D RoPE and cross-attention with prompt-derived features.
- OmniTransfer uses Wan2.1 I2V 14B as its underlying diffusion model.
- The input latent concatenates noisy video features, condition latents from an image and zero-filled frames, and binary mask latents.The mask marks preserved frames with 1 and generated frames with 0.
- The framework combines Task-aware Positional Bias, Reference-decoupled Causal Learning, and Task-adaptive Multimodal Alignment to unify video transfer tasks.
- Wan2.1 DiT blocks contain self-attention and cross-attention layers, with self-attention using 3D Rotary Positional Embedding.
- Cross-attention integrates textual features through keys and values derived from the prompt.
4 Method
OmniTransfer unifies appearance and temporal video-reference tasks through task-aware latent construction, positional bias, causal branch decoupling, and multimodal alignment. These designs adapt reference use to task semantics while preserving reference information and reducing inference cost.
- Reference Latent Construction: OmniTransfer constructs separate target and reference latents to support videos with different resolutions and task-specific reference handling.The reference mask uses distinct flags for temporal, ID, and style tasks, while its latent remains noise-free to preserve information.
- Task-aware Positional Bias: Task-aware positional bias shifts reference-video RoPE along width for temporal tasks and along time for appearance tasks.The temporal offset equals the target width, while the appearance offset equals the target frame count.
- Reference-decoupled Causal Learning: Reference-decoupled causal learning prevents bidirectional reference-target interaction that can produce copy-paste artifacts and concatenates reference keys and values into target attention.The reference branch performs intra-branch attention, while the target branch uses its own and reference features.
- Reference-decoupled Causal Learning: Fixing the reference branch at t = 0 makes it time-invariant during inference and reduces generation time by 20% versus the standard architecture.The reference branch is therefore independent of the target video's noise level.
- Task-adaptive Multimodal Alignment: Task-adaptive multimodal alignment uses MLLM features to distinguish temporal and appearance semantics from reference, target, template, and prompt inputs.Task-specific learnable tokens aggregate temporal cues for temporal tasks or fuse identity/style with prompt semantics for appearance tasks; aligned features are injected only into the target branch.
5 Experiment
The training process uses sequential objectives and a custom data collection because public reference-video-pair datasets are unavailable.
- Data: The authors collect their own Internet-sourced datasets because public datasets containing reference video pairs are unavailable.This collection supports spatio-temporal video-transfer training.
5.2 Evaluation Details
Evaluation uses dedicated test sets covering identity, style, effects, camera movement, and motion transfer, with unseen or professionally sourced examples across subtasks.
- Appearance Transfer: The ID test set contains 50 diverse-individual videos paired with two prompts each, while the style set contains 20 unseen styles with two prompts each.These sets assess identity consistency and stylistic variation.
- Temporal Transfer: The effect, camera-movement, and motion sets each contain 50 examples drawn from visual-effects websites, professional footage, and popular dance videos.The camera set covers complex trajectories, while the motion set covers diverse and fine-grained body motions.
5.3 Comparison
OmniTransfer is evaluated across identity, style, effect, camera movement, and motion transfer tasks, achieving strong appearance and temporal transfer while avoiding pose guidance for motion.
- ID Transfer: OmniTransfer preserves facial identity across frames and diverse poses, including frontal, profile, and tilted views, while maintaining natural motion.Multi-view facial information helps preserve fine details such as acne that are difficult for image-reference methods to capture.
- Style Transfer: OmniTransfer outperforms competing methods on all three style-transfer metrics and captures video style more effectively than previous image-based approaches.The evaluation uses CLIP-T, Aesthetics Score, and video CSD Score.
- Effect Transfer: OmniTransfer achieves the highest scores on effect fidelity, first-frame consistency, and overall visual quality, reproducing reference effects more accurately than Seedance and Wan I2V.The comparison is based on a 20-volunteer user study using five-point ratings.
- Camera Movement Transfer: OmniTransfer outperforms other camera-movement methods across camera fidelity, image consistency, and overall quality, including cinematic and complex professional tracking shots.Competing methods produce fixed-resolution outputs that require resizing or cropping.
- Motion Transfer: Without additional pose input, OmniTransfer achieves the highest image consistency while matching WanAnimate in motion consistency and quality despite using a smaller model.It uses the 14B Wan 2.1 model, whereas WanAnimate relies on the larger 28B Wan 2.2 model.
5.4 Ablation Study
The ablation study incrementally adds TPB, RCL, and TMA to a vanilla in-context baseline, showing distinct improvements in motion transfer, reference handling, efficiency, and semantic understanding.
- Baseline: The baseline concatenates reference features along the temporal dimension and applies full attention in a vanilla in-context learning setup.This baseline is the starting point for incremental addition of TPB, RCL, and TMA.
- TPB: Adding TPB enables fine-grained motion transfer through spatial context and alleviates task confusion using different RoPE biases.Without TPB, subtle motion transfer is ineffective and appearance cues can leak into temporal transfer.
- RCL: Replacing full attention with RCL alleviates copy–paste artifacts, improves facial naturalness, and increases inference speed by 20%.The ablation reports incomplete identity copying in some examples and a more natural face in another.
- TMA: Adding TMA forms the full model and substantially enhances semantic understanding of task-specific reference content.Examples include recognizing an unchanged scene and generating money rather than simply copying reference content.
6 Conclusion
OmniTransfer unifies spatio-temporal video transfer by combining multi-view and temporal reference information across diverse tasks. Experiments support flexible, high-fidelity generation with superior appearance and temporal transfer.
- Conclusion: OmniTransfer integrates TPB, RCL, and TMA to leverage multi-view and temporal information for fine-grained, consistent video generation across diverse tasks.The framework is presented as a unified approach to spatio-temporal video transfer.
A Additional Comparison Results
The appendix provides additional qualitative comparisons covering identity, style, effect, camera motion, and motion transfer.
- Additional Results: Additional comparisons are provided for identity and style transfer, followed by effect, camera motion, and motion transfer results.Figures 7–16 extend the qualitative evaluation across all reported transfer categories.
B Video Transfer Combination
OmniTransfer combines multiple video transfer operations by concatenating reference-video and multimodal-language-model tokens. Results on unseen task combinations demonstrate strong generalization, alongside comparisons spanning identity, style, effects, camera movement, and motion transfer.
- Concatenating reference video tokens and MLLM tokens across tasks enables seamless combinations of multiple video transfer operations.
- Figures 17 and 18 evaluate video transfer combinations on entirely unseen task combinations and report superior handling by OmniTransfer.
- Appearance transfer: The additional comparisons cover identity transfer against ConsisID, Phantom, and Stand-in, and style transfer against StyleCrafter and StyleMaster.
- Temporal transfer: Additional evaluations compare effect transfer with Wan2.1 I2V and Seedance 1.0 I2V, and camera movement transfer with MotionClone and CamCloneMaster.
- Motion transfer: Additional motion-transfer comparisons use MimicMotion and WanAnimate; unavailable results indicate failures in pose extraction or pose alignment.