PaperlayerView exact revision on arXiv

Source-linked AI summary

StableAnimator: High-Quality Identity-Preserving Human Image Animation

Shuyuan Tu, Zhen Xing, Xintong Han, Zhi-Qi Cheng, Qi Dai, Chong Luo, Zuxuan Wu

arXiv:2411.17697v2cs.CVcs.AI

TL;DR

Human image animation models can distort faces and lose identity, and temporal modeling makes it challenging to preserve identity without compromising video fidelity. StableAnimator combines a content-aware Face Encoder and distribution-aware ID Adapter with HJB-guided face optimization during denoising. On TikTok, it outperforms ControlNeXt by 47.1% in CSIM and achieves the best FVD result (140.62).

  • Problem

    Human image animation can distort facial regions and lose identity under substantial pose variation, while preserving identity alongside video fidelity remains challenging with temporal modeling.

    Source: §1 · pp. 1–2

  • Method

    StableAnimator refines reference face embeddings with a global content-aware Face Encoder, aligns image and face-conditioned features using a distribution-aware ID Adapter, and applies HJB-based face optimization during inference denoising.

    Source: §1 · p. 2

  • Results

    On TikTok, StableAnimator outperforms ControlNeXt by 47.1% in CSIM and achieves the best FVD result (140.62).

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments across benchmark datasets demonstrate StableAnimator's effectiveness in producing high-quality, ID-preserving human animations without third-party face-swapping tools.

    Source: §1 · p. 2, §5 · p. 9

    When a hand obscures most of the reference face, StableAnimator struggles to fill the obscured regions, degrading synthesized-face quality.

    Source: §A.12 · p. 14

Abstract

from arXiv · show

Current diffusion models for human image animation struggle to ensure identity (ID) consistency. This paper presents StableAnimator, the first end-to-end ID-preserving video diffusion framework, which synthesizes high-quality videos without any post-processing, conditioned on a reference image and a sequence of poses. Building upon a video diffusion model, StableAnimator contains carefully designed modules for both training and inference striving for identity consistency. In particular, StableAnimator begins by computing image and face embeddings with off-the-shelf extractors, respectively and face embeddings are further refined by interacting with image embeddings using a global content-aware Face Encoder. Then, StableAnimator introduces a novel distribution-aware ID Adapter that prevents interference caused by temporal layers while preserving ID via alignment. During inference, we propose a novel Hamilton-Jacobi-Bellman (HJB) equation-based optimization to further enhance the face quality. We demonstrate that solving the HJB equation can be integrated into the diffusion denoising process, and the resulting solution constrains the denoising path and thus benefits ID preservation. Experiments on multiple benchmarks show the effectiveness of StableAnimator both qualitatively and quantitatively.

1. Introduction

Human image animation can distort faces and lose identity under substantial pose variation, while temporal modeling complicates preserving identity without sacrificing video fidelity. StableAnimator combines identity-focused training modules with inference-time face optimization; it reports stronger identity and video-quality results than ControlNeXt on TikTok.

1. Introduction

  • Under significant pose variation, existing human image animation approaches can produce distortions and inconsistencies, especially in facial regions, damaging identity information.The paper also identifies a challenge in preserving identity while maintaining video fidelity when temporal layers interfere with stable spatial priors.Source: §1 · pp. 1–2
  • StableAnimator uses a global content-aware Face Encoder and distribution-aware ID Adapter to incorporate reference embeddings while maintaining video fidelity and ID consistency.The adapter aligns outputs from separate cross-attention with refined face and image embeddings, progressively bringing their distributions closer to mitigate temporal-layer interference.Source: §1 · p. 2
  • At inference, HJB-based face optimization updates latents during denoising to constrain the denoising path toward ID consistency and reduce reliance on face-swapping tools.The method is active during inference without training diffusion components.Source: §1 · p. 2
  • The authors present StableAnimator as an end-to-end ID-preserving video diffusion framework and report qualitative improvement over ControlNeXt, even when it uses face swapping or restoration tools.The paper reports experiments on benchmark datasets as evidence of the model's superiority.Source: §1 · p. 2
  • On TikTok, StableAnimator outperforms ControlNeXt by 47.1% in CSIM and achieves the best FVD result (140.62).The paper describes CSIM as face similarity between an animated frame and the reference.Source: §1 · p. 2

2. Related Work

Prior work includes video diffusion models built from image-generation models and numerous methods for image-domain identity preservation, but temporal layers can disrupt spatial distributions. Human animation systems also use pose and appearance conditioning, with some relying on face-swapping post-processing; StableAnimator uses a video-diffusion backbone and an ID Adapter to integrate identity information.

Diffusion for Video Generation

  • Video-generation systems commonly add temporal layers to pretrained image diffusion models, while some replace the U-Net with transformers; StableAnimator uses Stable Video Diffusion as its backbone.Video generation requires temporal smoothness and consistency in addition to image-generation capabilities.Source: §2 · p. 2

ID Consistency Image Generation

  • Image-domain ID-preservation methods use techniques such as customized training, separated cross-attention, hybrid mechanisms, facial prompts, and alignment losses, but temporal layers can shift latent distributions and hinder direct integration into video diffusion.StableAnimator's distribution-aware ID Adapter is presented as a way to integrate ID information while addressing this conflict.Source: §2 · p. 2

Pose-guided Human Image Animation

  • StableAnimator injects image embeddings and Face Encoder-refined face embeddings into U-Net blocks through the ID Adapter.Figure 2 identifies panels (a) and (b) as the Face Encoder and a U-Net block, respectively.Source: Figure 2 · p. 3
  • Pose-guided animation methods transfer motion from a pose sequence to a reference image, using approaches such as reference and pose networks, 3D signals, Mamba, regional losses, or convolution-based pose networks.MimicMotion and ControlNeXt use FaceFusion post-processing to address face distortion, which can degrade overall video quality; StableAnimator is described as avoiding post-processing.Source: §2 · p. 3

3. Method

StableAnimator builds on SVD, combining reference-image pathways with pose features in a video diffusion model. Its training modules align face information with diffusion features, while inference integrates face optimization into denoising.

3. Method

  • A reference image enters through VAE, CLIP, and ArcFace pathways, while PoseNet features from the pose sequence are added to noisy latents.At inference, the original video-frame inputs are replaced with random noise.Source: §3 · p. 3

3.1. ID-preserving During Training

  • The Global Content-Aware Face Encoder lets face embeddings interact with image embeddings, supplying reference-layout context before face information enters the U-Net.This addresses the lack of global layout and background awareness when face embeddings are fed directly to the U-Net.Source: §3.1 · pp. 3–4
  • The Distribution-aware ID Adapter aligns refined face embeddings with diffusion latents before temporal modeling to avoid feature distortion from distribution shifts.The paper links such distortion to conflict between video fidelity and ID consistency.Source: §3.1 · p. 4

3.2. ID-preserving During Inference

  • At each denoising step, face-similarity optimization refines the predicted sample; integrating the HJB solution into denoising constrains the path and reduces detail distortions.The optimization uses ArcFace similarity to the reference, and the paper describes the HJB solution and diffusion SDE as structurally compatible.Source: §3.2 · pp. 4–5

3.3. Training

  • Training uses reconstruction loss with a trainable U-Net, FaceEncoder, and PoseNet, and ArcFace-derived face masks enhance modeling of face regions.The loss compares diffusion latents with denoised latents.Source: §3.3 · p. 5

4. Experiments

Experiments compare StableAnimator with human image animation models, ablate its components and face-enhancement strategies, and evaluate long, multi-person animation and user preferences.

4.1. Implementation Details

  • The model is trained on 3K internet videos; evaluation includes TikTok and 100 unseen internet videos, with competitors trained on the authors’ dataset for Unseen100 comparisons.Training uses 20 epochs on four NVIDIA A100 80G GPUs.Source: §4.1 · pp. 5–6, §4.2 · p. 6

4.2. Comparison with State-of-the-Art Methods

  • StableAnimator outperforms leading competitor Unianimate by 36.9% and 45.8% in CSIM across two datasets, without sacrificing video fidelity and single-frame quality.The comparison covers TikTok and Unseen100; Table 1 reports quantitative comparisons on both datasets.Source: Table 1 · p. 6, §4.2 · p. 6

4.3. Ablation Study

  • Removing core components significantly degrades performance, particularly in face-related CSIM, while the study reports effects on video fidelity and single-frame quality.The quantitative ablations use Unseen100.Source: §4.3 · p. 6, Table 2 · p. 7, Figure 4 · p. 7
  • IP-Adapter improves ID consistency but degrades video fidelity and single-frame quality; FaceFusion refines faces but relatively degrades video fidelity.StableAnimator refines face quality while maintaining high video fidelity in the same-domain end-to-end pipeline.Source: §4.3 · p. 6, Table 3 · p. 7
  • Addition and Norm fail to eliminate spatial-feature distortion after temporal modeling, while distribution alignment mitigates its impact.The alignment integrates mean and standard deviation from cross-attention features.Source: Table 4 · p. 7, §4.3 · p. 7
  • Face optimization boosts MagicAnimate overall, mitigates fidelity deterioration from IP-Adapter, and remains effective across diffusion backbones.The reported comparisons include MagicAnimate with optimization and a different diffusion-based backbone.Source: Table 5 · p. 8, §4.3 · p. 8

4.4. Applications and User Study

  • On pose sequences exceeding 300 frames with complex motion, StableAnimator handles long animations in high fidelity while preserving identities; it also animates multiple people.The long-animation comparison is qualitative and uses intricate appearance and background references.Source: §4.4 · p. 8
  • In a user study of 30 selected videos, participants preferred StableAnimator over competitors on subjective evaluation.Participants compared motion, appearance, background, and ID alignment with the reference.Source: Table 6 · p. 8, §4.4 · p. 8

5. Conclusion

StableAnimator combines training and inference modules to generate high-quality, identity-preserving human animations, with HJB-based face optimization integrated into denoising. The conclusion reports results across datasets and describes the project’s research-only status and data-filtering measures.

5. Conclusion

  • StableAnimator combines a Face Encoder and aligned ID-Adapter with HJB-based face optimization in a video diffusion pipeline.The Face Encoder refines face embeddings; the ID-Adapter mitigates interference from temporal modeling, while HJB optimization runs alongside diffusion denoising.Source: §5 · pp. 8–9
  • Experimental results across various datasets demonstrated the model’s superiority in producing high-quality, identity-preserving human animations.Source: §5 · p. 9
  • StableAnimator is a research project with no current plans for product integration or public access, and its training data is filtered for inappropriate content.Source: §5 · p. 9

A.1. Evaluation Metrics

The evaluation compares StableAnimator with current state-of-the-art animation models using pixel, image-similarity, perceptual, video-fidelity, and face-similarity metrics.

A.1. Evaluation Metrics

  • L1 measures average absolute pixel differences, while PSNR compares original-image signal power with corrupting-noise power.Higher PSNR indicates better quality.Source: §A.1 · p. 12
  • SSIM measures image similarity through luminance, contrast, and structural information.Source: §A.1 · p. 12
  • LPIPS assesses image similarity through patch feature representations intended to reflect human visual perception.Source: §A.1 · p. 12
  • FVD measures real-versus-generated video feature-distribution disparity across spatial and temporal dimensions, and is used to assess video fidelity.Source: §A.1 · p. 12
  • CSIM is the cosine similarity between facial embeddings extracted by ArcFace.Source: §A.1 · p. 12

A.2. Preliminaries

The preliminaries describe diffusion as a forward process that adds Gaussian noise and a reverse process that predicts noise to recover the data. They also characterize denoising continuously through a reverse SDE.

A.2. Preliminaries

  • The forward diffusion process progressively adds Gaussian noise to a data sample, ultimately converting x_0 into Gaussian noise x_T after T steps.α_t is a constant noise schedule.Source: §A.2 · p. 12
  • In reverse diffusion, ε_θ(x_t, t) predicts noise from the current sample and timestep to recover x_0, and an MSE loss trains the predictor.Source: §A.2 · p. 12
  • Continuous denoising is represented as a reverse SDE, with the diffusion model approximating its score function.The reverse-SDE description identifies Brownian motion, score function, drift, and volatility.Source: §A.2 · p. 12

A.3. Details of Testing Dataset

Unseen100 is a test dataset assembled from 100 unseen online videos, with varied people, appearances, settings, and motion.

A.3. Details of Testing Dataset

  • Unseen100 contains 100 internet-sourced videos, each 10–20 seconds long, selected for testing.Source: §A.3 · p. 12
  • The videos span ethnicities, genders, full-body to close-up shots, and varied indoor and outdoor settings.Source: §A.3 · p. 12
  • Compared with the TikTok dataset, Unseen100 features more complicated motion and intricate appearances, including changing positions and facial expressions.Some videos include head shaking, making identity consistency more challenging.Source: §A.3 · p. 12

A.4. Long Animation

StableAnimator produces long, complex animations with less body and identity distortion than competing models. Its HJB-based inference procedure refines face consistency while continuing diffusion denoising.

A.4. Long Animation

  • Videos exceeding 300 frames show StableAnimator generating long animations while competing models suffer dramatic body and identity distortion.The examples feature complex appearances, motion sequences, and backgrounds.Source: §A.4 · p. 12
  • The HJB optimization starts from a diffusion model’s denoised prediction and updates it using ArcFace similarity between the decoded face and the reference.The algorithm optimizes the predicted sample for ten steps, then uses the result in the denoising update.Source: §A.4 · p. 13
  • After optimization, the procedure advances the latent with an Euler step and applies a second-order correction when the next timestep is nonzero.The algorithm also temporarily increases the noise level before predicting and optimizing the denoised sample.Source: §A.4 · p. 13

A.5. Multiple Person Animation

The supplementary results show StableAnimator preserving identity across multiple people, complex pose changes, and extended animation examples, while ablations examine its components. The paper also reports a face-occlusion failure and flags potential misuse of generated videos.

A.6. Optimization Details

  • The authors’ optimization algorithm closely follows the EDM algorithm structure; its γ1 expression is derived from the transversality condition at the terminal time.The expression is derived from Equations 4 and 5.Source: §A.6 · p. 13

A.8. Identity-Preserving Loss

  • Training uses face masks in the reconstruction MSE loss rather than ArcFace ID loss, avoiding the resource-intensive decoder backpropagation that the latter requires.The paper notes that temporal layers in the SVD VAE decoder make this backpropagation especially demanding.Source: §A.8 · pp. 13–14

A.5. Multiple Person Animation

  • With multiple protagonists, StableAnimator can preserve the original identities while maintaining high video fidelity.The paper demonstrates this capability on a particular video.Source: §A.5 · p. 13, Figure 11 · p. 18

A.7. Additional Face Discussion

  • StableAnimator outperforms facial-restoration competitors in identity preservation; GFP-GAN and CodeFormer exhibit facial distortion and over-sharpening.The authors attribute this limitation of post-processing enhancement to the baseline’s imprecise facial layout.Source: §A.7 · p. 13, Figure 12 · p. 18

A.9. Additional Comparison Results

  • Across complex pose sequences and frontal or profile references, StableAnimator preserves identity while competitors show body distortion and lost facial detail.The examples are shown in Figures 13 and 14.Source: §A.9 · p. 14, Figure 13 · p. 19, Figure 14 · p. 20

A.10. Animation Results

  • StableAnimator maintains appearance, background, and identity across varied animations, including closed eyes, head shaking, and body rotation.The examples include continuously changing head orientation and camera-facing angle.Source: §A.10 · p. 14, Figure 15 · p. 21, Figure 16 · p. 21

A.11. Additional Ablation Study

  • Ablations across SVD- and SD-based backbones show the proposed components facilitating performance, especially in facial regions; HJB optimization also improves animations without face encoders or adapters.The authors describe per-denoising-step updates based on face-embedding similarity as the reason for this improvement.Source: §A.11 · p. 14, Figure 19 · p. 24

A.12. Limitation and Future Work

  • When a hand covers most of the face, StableAnimator struggles to fill the obscured regions; face-aware inpainting is proposed as future work.The paper identifies this as one failure case.Source: §A.12 · p. 14, Figure 20 · p. 24
Loading 2411.17697v2…