PaperlayerView exact revision on arXiv

Source-linked AI summary

Wan-Animate: Unified Character Animation and Replacement with Holistic Replication

Gang Cheng, Xin Gao, Li Hu, Siqi Hu, Mingyang Huang, Chaonan Ji, Ju Li, Dechao Meng, Jinwei Qi, Penchong Qiao, Zhen Shen, Yafei Song, Ke Sun, Linrui Tian, Feng Wang, Guangyuan Wang, Qi Wang, Zhongjian Wang, Jiayu Xiao, Sheng Xu, Bang Zhang, Peng Zhang, Xindi Zhang, Zhe Zhang, Jingren Zhou, Lian Zhuo

arXiv:2509.14055v1cs.CV

TL;DR

Existing open-source frameworks lack a holistic solution for character animation that combines motion, expression, and environmental integration. Wan-Animate unifies animation and replacement, and reports versatile performance across character types and shot scales, with competitive quality against commercial products.

  • Problem

    Existing frameworks lack a holistic solution for high-fidelity character animation that unifies motion, expression, and environmental interaction, particularly in open-source character replacement.

    Source: §1 · pp. 1–2

  • Method

    Wan-Animate unifies animation and replacement, using a shared input paradigm to replicate reference motion and expressions and a Relighting LoRA to adapt replacement lighting and color tone.

    Source: §3.1 · p. 4, §3.4 · p. 7, §6 · p. 13

  • Results

    Wan-Animate reports versatile capabilities across portrait, half-body, and full-body shots, with competitive quality against several closed-source commercial products.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors present releasing the model weights and complete pipeline publicly as a contribution to developing character image animation.

    Source: §1 · p. 3

Abstract

from arXiv · show

We introduce Wan-Animate, a unified framework for character animation and replacement. Given a character image and a reference video, Wan-Animate can animate the character by precisely replicating the expressions and movements of the character in the video to generate high-fidelity character videos. Alternatively, it can integrate the animated character into the reference video to replace the original character, replicating the scene's lighting and color tone to achieve seamless environmental integration. Wan-Animate is built upon the Wan model. To adapt it for character animation tasks, we employ a modified input paradigm to differentiate between reference conditions and regions for generation. This design unifies multiple tasks into a common symbolic representation. We use spatially-aligned skeleton signals to replicate body motion and implicit facial features extracted from source images to reenact expressions, enabling the generation of character videos with high controllability and expressiveness. Furthermore, to enhance environmental integration during character replacement, we develop an auxiliary Relighting LoRA. This module preserves the character's appearance consistency while applying the appropriate environmental lighting and color tone. Experimental results demonstrate that Wan-Animate achieves state-of-the-art performance. We are committed to open-sourcing the model weights and its source code.

1 INTRODUCTION

Wan-Animate addresses the stated gap in holistic character animation by combining reference-driven motion and expression reenactment with character replacement that adapts to the video environment. The introduction reports robustness across different shot types and competitive quality against several closed-source commercial products.

Unified animation and replacement

  • Given a character image and reference video, Wan-Animate reenacts expressions and body movements to animate the source identity or replace the video's character while matching its lighting and color tone.Figure 1 depicts these two functions as Animation and Replacement.Source: §1 · p. 2, Figure 1 · p. 2

Control representation

  • Built on Wan-I2V, a modified input definition distinguishes reference conditions from generation regions and unifies image injection, temporal guidance, and animation or replacement modes while preserving the original input structure.The design aims to minimize distributional shift during post-training.Source: §1 · p. 2

Motion and expression control

  • Body motion is controlled by spatially aligned skeleton signals added to initial noise latents, while temporally compressed face-image latents drive expression through cross-attention.The face images serve as the expression signal while disentangling expression information from identity attributes.Source: §1 · p. 2

Environmental integration

  • An auxiliary Relighting LoRA applies environmental lighting and color tones after the base model transfers character appearance, promoting seamless integration during replacement.The module is intended to improve consistency between the character and its environment.Source: §1 · pp. 2–3

Results and release

  • Wan-Animate reports expressive videos, generalization across portraits, half-body and full-body shots, and quality competitive with several closed-source commercial products.The authors also commit to publicly releasing the model weights and complete pipeline.Source: §1 · p. 3

2 RELATED WORKS

Related work traces character animation’s shift from warping and GANs to diffusion and video foundation models, alongside advances in environmental integration and facial control. DiT-based video generation and implicit facial representations provide relevant foundations for Wan-Animate.

Character Image Animation

  • Image animation moved from warping and GAN architectures to diffusion methods, then video foundation models; Wan-Animate builds on Wan2.1’s pretrained knowledge.Animate Anyone uses ReferenceNet for appearance consistency and temporal layers inspired by AnimateDiff, while video foundation models bring inter-frame consistency and continuity from pretraining.Source: §2 · p. 3
  • Environmental character generation adds background inputs or masks for scenes and interactive objects, while human-object interaction methods generate synchronized, physically consistent co-moving objects.The passage notes these environmental approaches could be adapted to video character replacement.Source: §2 · p. 3

Facial Animation

  • Facial animation moved from landmark-driven expression control, which can lose detail and challenge cross-identity retargeting, toward implicit source-image representations that improve expressiveness and generality.Landmark-based methods follow pose-guided human animation; later methods use raw source images to extract implicit control representations.Source: §2 · pp. 3–4

Video Generation

  • DiT-based video generation uses 3D VAE compression and patchifying to tokenize video, and open-source models such as HunyuanVideo and Wan have spurred further applications.The passage describes DiT-based approaches as supplanting UNet-based ones and links video foundation model advances to improving character animation.Source: §2 · p. 4

3 MODEL DESIGN AND ARCHITECTURE

Wan-Animate unifies character animation and replacement in one jointly trained model, adapting Wan-I2V inputs to support reference appearance, motion and expression control, and mode-specific backgrounds. It adds a Replacement-only Relighting LoRA and staged training and inference procedures to support coherent, environment-matched videos.

3.1 TASK DEFINITION

  • Animation Mode preserves the source image background, while Replacement Mode places the driven character in the reference video environment; both replicate reference motion and expressions in one model.The modes correspond respectively to I2V and V2V, and differ in the source of the final background; the Relighting LoRA is specific to Replacement Mode.Source: §3.1 · p. 4

3.2 INPUTS FORMULATION

  • Wan-Animate modifies Wan-I2V inputs to treat the character image as an appearance reference, support temporal continuity, and represent both output modes compatibly.It encodes the character image as a reference latent and concatenates it with conditional latents; randomly selected early target latents can provide temporal guidance. Animation Mode uses zero-filled target conditions, while Replacement Mode conditions on segmented-background frames so generation is confined to the subject region.Source: §3.2 · p. 5

3.3 CONTROL SIGNALS

  • For body control, Wan-Animate uses spatially aligned 2D skeletons, which generalize to non-human characters, and injects pose latents into target—not reference—latents.VitPose extracts target poses, which are compressed with Wan-VAE, projected, and added to patchified noise latents; omitting pose information from the reference latent distinguishes it temporally from targets. Skeletons are less suited to complex motion with spatial ambiguity or missing keypoints than SMPL, while SMPL can encode shape cues that complicate identity consistency.Source: §3.3 · pp. 5–6
  • For facial control, Wan-Animate drives expression from raw face images compressed into latents, avoiding the fine-detail loss and identity-consistency risks of facial landmarks.Spatial compression and scaling, color-jitter, and noise augmentations discourage identity leakage; temporally downsampled face features are injected by aligned cross-attention into selected Transformer layers, as illustrated in Figure 3.Source: §3.3 · p. 6, Figure 3 · p. 6

3.4 RELIGHTING LORA

  • The Replacement-only Relighting LoRA adjusts character lighting and color tone to match the new environment while preserving identity.It is trained on character crops composited onto random backgrounds with IC-Light and applied to self-attention and cross-attention layers in the DiT blocks.Source: §3.4 · p. 7

3.5 TRAINING

  • Training proceeds from body-controlled Animation Mode, to facial control and joint control, then to both modes and finally Replacement-mode relighting.Portrait data and higher loss weights on head, eye, and mouth regions support expression learning in the face-control stage; joint control uses the full dataset, and the final stage trains the Relighting LoRA.Source: §3.5 · p. 7

3.6 INFERENCE

  • At inference, pose retargeting adjusts reference-video skeleton proportions and position for Animation Mode, but is not recommended for Replacement Mode because it can disrupt character–environment interactions.T-pose image editing can improve bone-length scaling; skipping retargeting in Replacement Mode can cause deformation when character body shapes differ substantially.Source: §3.6 · p. 8
  • Long videos are generated iteratively, using the previous segment’s last frames as temporal guidance for each subsequent segment.Typically one or two latents provide guidance, corresponding to 1 or 5 frames; reference and guidance portions are discarded before generated segments are concatenated.Source: §3.6 · p. 8

4 IMPLEMENTATION

Wan-Animate’s implementation combines curated human-video data, distributed training strategies, and mode-specific inference settings. Motion signals are the dominant control, while text prompts are a non-core feature.

4.1 DATA CONSTRUCTION

  • Training data comprise quality-filtered human-centric clips with one consistent character, skeleton annotations for motion and filtering, SAM2 masks for replacement, and QwenVL2.5-72B captions.Text control is non-core compared with motion signals, so a default text prompt is recommended.Source: §4.1 · p. 8

4.2 PARALLEL STRATEGY

  • DiT and T5 use FSDP, other models use DP, and DiT additionally uses RingAttention–Ulysses context parallelism to reduce memory and accelerate training.Within each Ulysses group, Face Adapter facial frames from a video are batched for parallel feature extraction.Source: §4.2 · p. 8

4.3 DETAILS

  • Inference resolution follows the input-image aspect ratio in Animation Mode and the reference-video aspect ratio in Replacement Mode, with token count matched to a standard-resolution target.Segments contain 78 frames, including one reserved character-image frame; later segments use 1 or 5 temporal reference frames from the preceding segment. CFG is off by default but can be enabled for face conditioning when finer expression control is desired.Source: §4.3 · p. 9

5 EXPERIMENTS

Wan-Animate outperforms open-source character animation frameworks in benchmark comparisons and user preference, while qualitative evaluations and ablations show stable animation, improved replacement integration, and applications across diverse scenarios.

5.1 QUANTITATIVE EVALUATION

  • Wan-Animate outperforms current open-source character animation works in a self-reconstruction benchmark and is described as the most comprehensive, highest-performing open-source model to date.The evaluation uses the first video frame as the reference, subsequent-frame motion signals, SSIM, LPIPS, and FVD, with a separate portrait-only comparison against facial animation methods.Source: §5.1 · p. 9

5.2 HUMAN EVALUATION

  • In a cross-ID user study with 20 participants, Wan-Animate achieved a superior outcome to closed-source competitors across video quality, identity consistency, motion accuracy, and expression accuracy.Participants anonymously compared generated videos from Wan-Animate and a competing method side by side and chose their preferred result.Source: §5.2 · pp. 9–10, Figure 5 · p. 10

5.3 QUALITATIVE EVALUATION

  • In Animation Mode, Wan-Animate performs more comprehensively and stably than Animate Anyone, VACE, Runway Act-two, and DreamActor-M1 in qualitative comparisons.The comparisons report lower quality for Animate Anyone, instability for VACE, difficulty capturing complex motions for Act-two, and lower local detail and image fidelity for DreamActor-M1.Source: Figure 6 · p. 10, §5.3 · p. 11
  • In Replacement Mode, Wan-Animate performs better and is more user-friendly than Animate Anyone 2 and VACE, which shows identity inconsistency and requires substantial parameter tuning.Animate Anyone 2 also exhibits insufficient generation quality in the comparison.Source: §5.3 · p. 11, Figure 7 · p. 11

5.4 ABLATION STUDY

  • Progressive body-first, then expression, then joint training helps the face adapter converge; joint training from the start produced inaccurate expression driving and poor convergence.Training the expression module on portrait data, where faces occupy more of the frame, significantly accelerates convergence.Source: §5.4 · p. 11, Figure 8 · p. 12
  • Relighting LoRA makes character-environment fusion more realistic and harmonious without breaking perceptual identity, while adapting lighting and color tone to the new environment.Without LoRA, the generated character retains the reference image’s lighting and color tone, which can appear incongruous in the new environment.Source: §5.4 · pp. 11–12, Figure 9 · p. 12

5.5 MORE QUALITATIVE RESULTS

  • Wan-Animate demonstrates applications in performance reenactment, cross-style transfer, complex motion synthesis, dynamic-camera actions, and character replacement.The described uses include recreating performance scenes, transferring real-person performance to varied characters, generating dances and special actions, and editing characters in film, television, and commercial imagery.Source: §5.5 · pp. 12–13, Figure 10 · p. 13

6 CONCLUSION

Wan-Animate unifies character animation and replacement, precisely reenacting facial expressions and body motion. The paper reports performance surpassing current open- and closed-source algorithms and commits to open-sourcing the model.

6 CONCLUSION

  • Wan-Animate’s performance surpasses current open-source and closed-source algorithms.Source: §6 · p. 13
  • The framework supports both animating a character image with motion from a reference video and replacing the video’s character with that image.Source: §6 · p. 13
  • A modified input paradigm unifies the different input forms and makes training more efficient.Source: §6 · p. 13
  • Motion is fused with input noise latents using spatial alignment, while facial-image implicit features inject expression signals through attention.Source: §6 · p. 13
  • An auxiliary LoRA supports lighting and color-tone consistency with the new environment during character replacement; the authors commit to open-sourcing Wan-Animate.Source: §6 · p. 13

Related papers

Loading 2509.14055v1…