Source-linked AI summary
Wan-Animate: Unified Character Animation and Replacement with Holistic Replication
Gang Cheng, Xin Gao, Li Hu, Siqi Hu, Mingyang Huang, Chaonan Ji, Ju Li, Dechao Meng, Jinwei Qi, Penchong Qiao, Zhen Shen, Yafei Song, Ke Sun, Linrui Tian, Feng Wang, Guangyuan Wang, Qi Wang, Zhongjian Wang, Jiayu Xiao, Sheng Xu, Bang Zhang, Peng Zhang, Xindi Zhang, Zhe Zhang, Jingren Zhou, Lian Zhuo
TL;DR
Wan-Animate addresses the gap in holistic, high-fidelity character animation that jointly controls motion, expression, and environmental interaction. Built on Wan-I2V, it unifies inputs, uses skeleton and implicit facial controls, and adds Relighting LoRA for replacement; experiments report versatile, competitive performance and planned open release.
Problem
Existing open-source methods do not holistically control motion, expression, and environment interaction with high fidelity, and replacement methods can lack consistency and usability.
Method
Wan-Animate adapts Wan-I2V with a unified input paradigm, spatially aligned skeleton control, implicit facial features, and an auxiliary Relighting LoRA for character replacement.
Results
Wan-Animate precisely reenacts facial expressions and body motions, supports animation and replacement, and demonstrates versatile, robust, and competitive quality across scenarios.
Takeaways & Limitations
The framework provides a unified open-source pipeline for expressive character animation and environmentally integrated character replacement, with model weights and the complete pipeline slated for release.
Abstract
from arXiv · showhide
We introduce Wan-Animate, a unified framework for character animation and replacement. Given a character image and a reference video, Wan-Animate can animate the character by precisely replicating the expressions and movements of the character in the video to generate high-fidelity character videos. Alternatively, it can integrate the animated character into the reference video to replace the original character, replicating the scene's lighting and color tone to achieve seamless environmental integration. Wan-Animate is built upon the Wan model. To adapt it for character animation tasks, we employ a modified input paradigm to differentiate between reference conditions and regions for generation. This design unifies multiple tasks into a common symbolic representation. We use spatially-aligned skeleton signals to replicate body motion and implicit facial features extracted from source images to reenact expressions, enabling the generation of character videos with high controllability and expressiveness. Furthermore, to enhance environmental integration during character replacement, we develop an auxiliary Relighting LoRA. This module preserves the character's appearance consistency while applying the appropriate environmental lighting and color tone. Experimental results demonstrate that Wan-Animate achieves state-of-the-art performance. We are committed to open-sourcing the model weights and its source code.
1 INTRODUCTION
Wan-Animate addresses the lack of an open-source framework that jointly controls motion, expression, and environment interaction. It unifies character animation and replacement through a modified input paradigm, achieving high-fidelity generation and broad reported capability.
- Motivation: Existing open-source methods often lag state of the art and inadequately combine expressive facial dynamics with body-motion control.Dedicated open-source approaches for character replacement remain scarce, while available methods can suffer from consistency and usability issues.
- Contribution: Wan-Animate supports Animation and Replacement modes from a character image and reference video.Animation reenacts reference motion and expression; Replacement substitutes the reference character while integrating the source identity into the environment.
- Capabilities: Wan-Animate replicates facial expressions and body movements while matching lighting and color tone for environmental integration.The framework targets realistic character videos and seamless fusion during replacement.
- Contribution: Its modified input paradigm distinguishes reference conditions from generation regions and unifies reference-image injection, temporal guidance, and mode selection.The design is built on Wan-I2V and adapts its input formulation for character-animation requirements.
- Results: Experiments report expressive animation, robustness across portraits, half-body, and full-body shots, and competitive quality against closed-source commercial products.The authors also commit to releasing model weights and the complete pipeline.
2 RELATED WORKS
Related work spans diffusion-based character animation, environment-aware generation, facial control, and increasingly dominant DiT-based video generation. Wan-Animate builds on these directions while combining their relevant control signals and environmental adaptation.
- Character Image Animation: Diffusion-based architectures have improved character-animation performance, while earlier work used warping-based representations and GANs.Recent methods focus on stronger consistency and controllability.
- Environment-Aware Generation: Environment-aware methods extend animation toward specified scenes, character replacement, and human-object interaction synthesis.Examples use background images or masks to distinguish environments and interactive objects.
- Facial Animation: Facial-animation methods have progressed from landmark controls toward implicit representations extracted directly from source images.Implicit representations are reported to improve expressiveness and generality.
- Wan-Animate: Wan-Animate combines spatially aligned skeleton control, implicit facial features, and environmental information within a Wan-based framework.Its overview also includes an auxiliary Relighting LoRA for replacement.
- Video Generation: DiT-based video generation has increasingly supplanted UNet-based approaches as a mainstream route, aided by Transformer scaling and expanded training data.Video tokenization enables multimodal feature fusion within Transformer architectures.
3 MODEL DESIGN AND ARCHITECTURE
Wan-Animate adapts Wan-I2V with a unified input formulation, separate body and face controls, and a replacement-specific relighting module. Staged training and inference-time pose retargeting support controllable animation across two generative modes.
- Modes and Inputs: Animation Mode preserves the source-image background, whereas Replacement Mode inserts the source character into the reference-video environment.The two modes correspond approximately to I2V and V2V synthesis tasks.
- Input Formulation: The model modifies Wan-I2V inputs to inject appearance references, guide subsequent temporal segments, and distinguish preserved from generated regions.Wan-I2V uses noise latents, conditional latents, and a binary mask; Wan-Animate repurposes this formulation for arbitrary-length animation.
- Control Signals: Body motion uses spatially aligned skeleton signals because they offer generality for non-humanoid characters without embedding shape cues into the motion representation.The design accepts trade-offs in complex inter-limb relationships and extremity precision.
- Control Signals: Face control extracts implicit features from face images, temporally aligns their latents with noise latents, and injects them through cross-attention.The Face Adapter uses causal convolutions for temporal downsampling and Linear Motion Decomposition for expression disentanglement.
- Relighting: The Relighting LoRA adjusts character lighting and color tone during Replacement Mode to match the new environment without breaking perceptual identity.It is trained with IC-Light-generated character-background pairs and applied within DiT attention layers.
- Training: Training proceeds from body control to face control, joint control, dual-mode adaptation, and exclusive relighting training.Portrait data and higher loss weights for head, eye, and mouth regions support expression fidelity during face-control training.
- Inference: Inference-time pose retargeting adjusts reference skeleton bone lengths to match the source character’s proportions.The procedure addresses identity and proportion differences between the source image and reference video.
4 IMPLEMENTATION
The implementation uses human-centric video data and distributed training strategies, while adapting inference resolution to aspect ratio and video-token budgets.
- Data Construction: The dataset covers speaking, facial expressions, and body movements in single-character videos filtered for quality and identity consistency.Skeletons support motion annotation, behavior-based filtering, character tracking, and mask extraction for replacement.
- Data Construction: Skeleton information serves both as motion-signal annotation and as a criterion for filtering videos by character behavior.For replacement, annotated skeletons track characters before SAM2 extracts corresponding masks.
- Parallel Strategy: DiT and T5 use Fully Sharded Data Parallelism, while the remaining models use standard Data Parallelism.The DiT model additionally uses Context Parallelism combining RingAttention and Ulysses.
- Inference Resolution: Wan-Animate supports arbitrary output resolutions, with aspect ratios inherited from the character image in Animation Mode and the reference video in Replacement Mode.Inference resolution is selected according to the total number of video tokens after patchification.
5 EXPERIMENTS
Experiments evaluate Wan-Animate through quantitative reconstruction, human preference, qualitative comparisons, ablations, and application examples across animation and replacement modes.
- Quantitative Evaluation: The quantitative evaluation uses self-reconstruction videos spanning human subjects, character scales, and actions, with the first frame as the reference image.Subsequent-frame motion signals drive video reconstruction for comparison with open-source frameworks.
- Human Evaluation: Human evaluation compares anonymous Wan-Animate and competitor videos using generation quality, identity consistency, motion accuracy, and expression accuracy.Participants preferred Wan-Animate in the reported study.
- Qualitative Evaluation: Wan-Animate shows more comprehensive and stable Animation Mode performance than Animate Anyone, VACE, Runway Act-two, and DreamActor-M1.The cited comparison reports lower quality, instability, complex-motion difficulty, or lower local fidelity among alternatives.
- Qualitative Evaluation: In Replacement Mode, Wan-Animate performs better and is more user-friendly than Animate Anyone 2 and VACE.The comparison highlights VACE’s identity-consistency issues and dependence on parameter tuning.
- Ablation Study: Progressive training from body control to facial expressions and joint training benefits Face Adapter convergence.The authors validate this scheme with an ablation study against jointly training the control module on all data.
- Ablation Study: Relighting LoRA makes character-environment fusion more realistic and harmonious without breaking perceptual identity.It adds lighting and color adaptability beyond strict appearance consistency in Replacement Mode.
- More Qualitative Results: Application examples include performance reenactment, cross-style transfer, complex advertisement production, and character replacement.Character replacement supports re-imagining film and television scenes and editing characters in commercial photography and advertising.
6 CONCLUSION
The conclusion presents Wan-Animate as a unified system for character animation and replacement, combining motion and expression control with environment-aware integration.
- Conclusion: Wan-Animate drives a character image with reference-video motion for animation or replaces the original subject in the reference video.Its modified input paradigm unifies these input forms and supports both facial-expression and body-motion reenactment.
- Conclusion: Motion signals enter through spatially aligned fusion, while expression signals use attention over implicit facial features.An auxiliary LoRA improves lighting and color-tone consistency during character replacement.
- Conclusion: The paper reports performance surpassing current open-source and closed-source algorithms.
7 CONTRIBUTORS
The contributors are listed alphabetically by last name.
- Contributors: All contributors are listed in alphabetical order by their last names.
- Contributors: The listed contributors include Gang Cheng, Xin Gao, Li Hu, Siqi Hu, Mingyang Huang, Chaonan Ji, and additional authors.