Source-linked AI summary
MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model
Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, Mike Zheng Shou
TL;DR
Human image animation must generate motion-following videos while preserving reference identity, but existing approaches often lack temporal consistency and detailed appearance preservation. MagicAnimate uses video diffusion with temporal modeling, dense appearance encoding, joint image-video training, and video fusion for long animations. It reports state-of-the-art performance across two benchmarks, including more than 38% higher video quality than the strongest baseline on TikTok dancing.
Problem
Existing animation methods can lack temporal modeling and detailed reference-appearance preservation, causing frame-level flicker and weaker identity fidelity.
Method
MagicAnimate combines temporal attention in a video diffusion model, dense appearance encoding, image-video joint training, and sliding-window video fusion.
Results
MagicAnimate achieves state-of-the-art single-frame and video quality on two benchmarks, surpassing the strongest baseline by more than 38% in video quality on TikTok dancing.
Takeaways & Limitations
The framework supports temporally consistent human animation of arbitrary length, cross-identity animation, unseen domains, and multi-person scenarios.
Takeaways & Limitations
Temporal attention improves temporal modeling but can degrade per-frame quality, while segment-by-segment long-video processing can produce unnatural transitions across segments.
Abstract
from arXiv · showhide
This paper studies the human image animation task, which aims to generate a video of a certain reference identity following a particular motion sequence. Existing animation works typically employ the frame-warping technique to animate the reference image towards the target motion. Despite achieving reasonable results, these approaches face challenges in maintaining temporal consistency throughout the animation due to the lack of temporal modeling and poor preservation of reference identity. In this work, we introduce MagicAnimate, a diffusion-based framework that aims at enhancing temporal consistency, preserving reference image faithfully, and improving animation fidelity. To achieve this, we first develop a video diffusion model to encode temporal information. Second, to maintain the appearance coherence across frames, we introduce a novel appearance encoder to retain the intricate details of the reference image. Leveraging these two innovations, we further employ a simple video fusion technique to encourage smooth transitions for long video animation. Empirical results demonstrate the superiority of our method over baseline approaches on two benchmarks. Notably, our approach outperforms the strongest baseline by over 38% in terms of video fidelity on the challenging TikTok dancing dataset. Code and model will be made available.
1. Introduction
Human image animation seeks to generate videos that follow motion signals while preserving a reference identity, but existing methods struggle with temporal consistency, generalization, and appearance detail. MagicAnimate addresses these limitations with temporal modeling, dense appearance encoding, image-video joint training, and video fusion.
- Human image animation brings static reference images to life according to motion signals such as video, depth, or pose.
- GAN-based methods warp reference images and extrapolate missing regions, but have limited motion transfer, occlusion quality, and cross-identity generalization.
- Diffusion-based methods process long videos frame by frame, neglecting temporal consistency and often producing flickering results.
- MagicAnimate uses temporal attention in a video diffusion model and a dense appearance encoder to preserve identity, background, clothes, and other details.
- Image-video joint training augments single-frame data for richer visual cues, while simple video fusion enables smooth transitions in long animations.
- 38%+ improvement in video quality over the strongest baseline is reported on the challenging TikTok dancing dataset, alongside state-of-the-art results on two benchmarks.
2. Related Work
Related work spans implicit and explicit animation pipelines and includes GAN- and diffusion-based approaches for human-centric image and video animation. These methods use motion signals, deformation or latent manipulation, and diffusion-based conditioning to animate reference images.
- Prior human animation work uses motion representations including keypoints, semantic parsing, and statistical parametric models.
- Implicit and explicit animation: Implicit animation transforms reference images toward target motion by deforming them in subexpression space or manipulating a generative model’s latent space.
- Diffusion advances have supported controllable image generation, video generation, and human-centric video animation.
- Some diffusion-based animation methods generate 2D optical flow before animating the reference image with frame warping, while others use Stable Diffusion-based frameworks.
3. Method
MagicAnimate animates a reference identity according to a target motion sequence using temporal video diffusion, detailed appearance conditioning, pose guidance, and video fusion for long sequences. Its pipeline is designed to maintain temporal and appearance consistency while preserving per-frame quality.
- Objective: MagicAnimate synthesizes a continuous video that preserves the reference image's appearance while following a provided motion sequence.The motion sequence can contain N frames, producing a corresponding video of N synthesized frames.
- Temporal Modeling: Temporal attention blocks extend the diffusion backbone into a video model that aggregates information across neighboring frames.The model reshapes features across the frame dimension, adds positional encoding, and computes cross-frame attention.
- Appearance Encoding: A trainable appearance encoder extracts detailed identity and background features from the reference image for conditioning during denoising.Unlike compact CLIP features, the resulting appearance features are injected through hybrid spatial attention to preserve clothing, accessories, identity, and background.
- Motion Transfer: DensePose conditioning guides the video diffusion model to transfer the reference human identity to the target motions.DensePose is used instead of sparse OpenPose keypoints because major keypoints may be less robust for motions such as rotation.
- Long Video Animation: Sliding-window video fusion overlaps segments during inference to reduce discontinuities in arbitrarily long animations.The method partitions the motion and noise sequences into overlapping K-frame segments and averages overlap frames during fusion.
- Training: Joint image-video training balances temporal modeling with single-frame fidelity after temporal attention improves coherence but degrades per-frame quality.Image data are sampled probabilistically during training, including reference-image reconstruction in the first stage.
4. Experiments
MagicAnimate is evaluated against GAN- and diffusion-based baselines on TikTok and TED-talks, using quantitative comparisons, qualitative videos, ablations, and application scenarios. Results show stronger video and identity fidelity, while ablations support the roles of temporal modeling, appearance encoding, and video fusion.
- Quantitative comparisons: MagicAnimate achieves the best TED-talks FID-VID of 19.00 and FVD of 131.51, while also obtaining the best single-frame FID of 22.78.It improves FVD by 28.1% over MRAA and FID by 17.2% over DisCo.
- Quantitative comparisons: MagicAnimate achieves the best AKD, MKR, and AED results on TED-talks, providing evidence of stronger identity preservation and animation precision.
- Qualitative comparisons: On TikTok, GAN-based MRAA and TPS struggle with substantial pose differences, whereas diffusion-based baselines provide better single-frame quality but weaker temporal consistency.IPA+CtrlN and DisCo generate frames independently, causing clothing color changes and inconsistent occluded backgrounds; IPA+CtrlN-V is more temporally consistent but has weaker reference conditioning.
- Qualitative comparisons: MagicAnimate produces temporally consistent animations with high-fidelity background, clothing, face, and hand details, and shows superior identity- and background-preserving ability on TED-talks.On TED-talks, its higher overall L1 error is attributed to missing background information in DensePose control signals, although foreground-human L1 error is comparable to MRAA.
- Ablation studies: Removing temporal attention decreases single-frame and video-fidelity metrics, while replacing the appearance encoder with CLIP or IP-Adapter reduces reference-image preservation.Without explicit temporal modeling, the model fails to maintain temporal coherence for humans and backgrounds.
- Ablation studies: Skipping video fusion or using different initial noises across segments reduces appearance and video quality, while MagicAnimate generalizes to unseen styles, DALL·E3-generated images, and multiple people.
5. Conclusion
MagicAnimate combines temporal modeling, appearance encoding, and video frame fusion to improve human avatar animation quality and coherence. It achieves state-of-the-art single-frame and video quality while generalizing to unseen domains and multi-person scenarios.
- MagicAnimate models temporal information to enhance the coherence of animation results.
- Its appearance encoder improves single-frame quality and contributes to temporal consistency.
- Video frame fusion enables seamless transitions across the animation video.
- MagicAnimate achieves state-of-the-art performance in both single-frame and video quality.
- The framework generalizes to unseen domains and multi-person animation scenarios.