Source-linked AI summary
Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance
Shenhao Zhu, Junming Leo Chen, Zuozhuo Dai, Qingkun Su, Yinghui Xu, Xun Cao, Yao Yao, Hao Zhu, Siyu Zhu
TL;DR
Human image animation needs motion guidance that preserves a reference person’s shape and appearance across varied movements and identities. The paper integrates SMPL-based shape and pose alignment with latent diffusion, rendered geometric conditions, skeleton guidance, and multilevel feature fusion. Experiments report improved animation quality and generalization across benchmark and diverse real-world datasets, while SMPL–DWpose inconsistency remains a potential error source.
Problem
Skeleton or dense-pose guidance can limit pose alignment and motion guidance, while motion transfer may produce artifacts and temporal inconsistencies under identity or scene variation.
Method
The method uses SMPL within latent diffusion to align reference shape with driving poses and condition generation using rendered depth, normal, semantic, and skeleton maps.
Results
The method improves human-animation quality on TikTok and UBC fashion datasets and demonstrates robust generalization on a diverse real-world dataset.
Takeaways & Limitations
Unified SMPL shape-and-pose guidance, complemented by skeleton input and self-attention, supports animations that more accurately reflect human anatomy and movement.
Takeaways & Limitations
SMPL models faces and hands less effectively than feature-based methods, and independently solved SMPL and DWpose signals may be inconsistent.
Abstract
from arXiv · showhide
In this study, we introduce a methodology for human image animation by leveraging a 3D human parametric model within a latent diffusion framework to enhance shape alignment and motion guidance in curernt human generative techniques. The methodology utilizes the SMPL(Skinned Multi-Person Linear) model as the 3D human parametric model to establish a unified representation of body shape and pose. This facilitates the accurate capture of intricate human geometry and motion characteristics from source videos. Specifically, we incorporate rendered depth images, normal maps, and semantic maps obtained from SMPL sequences, alongside skeleton-based motion guidance, to enrich the conditions to the latent diffusion model with comprehensive 3D shape and detailed pose attributes. A multi-layer motion fusion module, integrating self-attention mechanisms, is employed to fuse the shape and motion latent representations in the spatial domain. By representing the 3D human parametric model as the motion guidance, we can perform parametric shape alignment of the human body between the reference image and the source video motion. Experimental evaluations conducted on benchmark datasets demonstrate the methodology's superior ability to generate high-quality human animations that accurately capture both pose and shape variations. Furthermore, our approach also exhibits superior generalization capabilities on the proposed in-the-wild dataset. Project page: https://fudan-generative-vision.github.io/champ.
1 Introduction
Human image animation methods use reference images with motion guidance, but substantial identity and scene variation can produce motion-transfer artifacts, temporal inconsistency, and limited shape alignment. The proposed approach uses SMPL within latent diffusion, combining shape and pose guidance to improve animation quality and generalization.
- Motivation: GAN-based methods struggle to transfer motion when human identity or scene dynamics differ substantially between the reference image and source video.These challenges manifest as unrealistic visual artifacts and temporal inconsistencies in synthesized content.
- Motivation: Reference images combined with skeleton or dense-pose guidance may provide limited pose alignment and motion guidance.The limitation motivates using a 3D parametric human model rather than relying only on sparse or pose-focused representations.
- Proposed approach: SMPL provides a unified low-dimensional representation of body shape and pose, including geometry, occlusions, contours, and surface deformations.Its parametric structure also supports geometric correspondence between the reference human and source-video motion sequences.
- Proposed approach: Rendered depth, normal, and semantic maps enrich latent-diffusion conditioning with 3D structure, surface orientation, and body-part information.A skeleton is retained as an auxiliary input for intricate movements such as facial expressions and finger movements, while self-attention integrates the guidance features.
- Evaluation: Experiments on TikTok and UBC fashion datasets evaluate the method’s human-animation quality, while a diverse real-world dataset tests robust generalization.The paper reports comparative analysis against state-of-the-art approaches on the proposed in-the-wild dataset.
2 Related Work
Prior human-image animation methods use diffusion, GANs, and pose representations such as skeletons or DensePose. This work extends diffusion-based animation with SMPL-based geometric reconstruction and motion modeling for improved appearance alignment and pose guidance.
- Diffusion-based animation: Diffusion models have been applied to human-image animation by conditioning generation on reference-image appearance and motion information.Examples include DreamPose, DisCo, Animate Anyone, and MagicAnimate, which use combinations of CLIP, VAE, adapters, ControlNets, ReferenceNet, and temporal layers.
- Positioning: The proposed method introduces a 3D parametric model to reconstruct reference-image geometry and model motion in source-video sequences.SMPL is used to reconstruct both poses and shapes, providing more complete conditions for appearance alignment and pose guidance.
- Pose guidance: DWpose offers more accurate and expressive skeletons than OpenPose, while DensePose establishes dense correspondences between RGB images and surface representations.These representations provide pose guidance but differ in the density and type of human information they encode.
3 Method
The method aligns a reference person’s SMPL shape with a driving video’s pose sequence, renders multiple motion conditions, and integrates them into a latent-diffusion video pipeline. Guidance encoding and fusion combine these conditions while preserving identity and temporal coherence.
- Pipeline: Given a human image and driving video, the pipeline extracts SMPL representations and uses the aligned sequence as 3D motion guidance.The overall objective is to generate a temporally coherent video in which the image subject follows the driving video’s actions.
- Latent diffusion: Latent diffusion encodes images into a low-dimensional latent space, predicts noise during denoising, and decodes the denoised latent back into image space.The denoising network predicts noise at each timestep, conditioned on the timestep and conditioning information.
- Multi-Layer Motion Condition: SMPL sequences are rendered into depth, normal, and semantic maps that encode distance, surface orientation, and pixel-level body-part information.These maps provide complementary geometric and semantic information for human animation conditioning.
- Parametric Shape Alignment: Parametric Shape Alignment replaces the driving sequence’s shape with the reference human’s shape while retaining each driving pose.The aligned model is formulated as H_trans^i = SMPL(β_ref, θ_m^i), then rendered into motion conditions.
- Multi-Layer Motion Guidance: A skeleton supplements SMPL-derived maps for intricate facial and finger movements, while self-attention integrates multilevel guidance features.Guidance networks encode each condition, aggregate their embeddings by summation, and combine the result with the noisy latent representation.
4 Experiments
Experiments evaluate CHAMP on benchmark and unseen in-the-wild data, with qualitative comparisons, quantitative metrics, and ablations of its guidance components. Results support improved animation quality, shape alignment, motion guidance, and multi-view consistency, while noting limitations for faces and hands.
- 4.2 Comparisons: On TikTok, the original and fine-tuned variants perform strongly across L1, PSNR, SSIM, LPIPS, FID-VID, and FVD, with the fine-tuned model achieving the best overall results.The reported pattern is lower L1, LPIPS, FID-VID, and FVD, together with higher PSNR and SSIM.
- 4.1 Implementations: The proposed method is evaluated on TikTok and UBC fashion datasets, alongside comparisons on a diverse unseen dataset and several state-of-the-art baselines.The unseen evaluation uses 100 authentic human videos with substantial variation in shape, pose, and appearance.
- 4.2 Comparisons: Qualitative and statistical comparisons on the unseen dataset demonstrate the method’s effectiveness and robust generalization beyond domain-proximate benchmark datasets.The dataset contains videos gathered from diverse real-world scenarios and is designed to test robustness under broader variation.
- 4.3 Ablation Studies: The full SMPL-based guidance configuration outperforms ablated variants, while SMPL provides larger PSNR and SSIM gains than skeleton-only guidance.Reported gains are 1.27 versus 0.48 for PSNR and 0.10 versus 0.05 for SSIM; skeleton guidance further helps facial and hand motion.
- 4.3 Ablation Studies: Guidance self-attention improves performance across all evaluated metrics compared with omitting the attention mechanism.Additional qualitative results are provided for the guidance self-attention ablation.
- 4.3 Ablation Studies: Parametric shape alignment produces the most consistent shape and figure alignment in the reported extreme-figure reference-image comparison.The multi-view experiment uses sequential SMPL guidance and achieves satisfactory results, though it may not match direct 3D rendering for consistent novel views.
5 Conclusion
The paper presents a human image animation approach that combines SMPL with latent diffusion to improve pose alignment, motion guidance, and realistic shape and movement capture.
- The method integrates the SMPL 3D parametric human model with latent diffusion for human image animation.
- SMPL provides a unified representation of shape and pose variations for capturing realistic human movements and body shapes.
- Depth, normal, and semantic maps, together with skeleton-based motion guidance, enrich the animation conditions with 3D shape and pose information.
- Self-attention mechanisms integrate feature maps to refine the animation process and better reflect human anatomy and movement.
- Experiments across various datasets confirm the approach produces high-quality human animations and supports detailed, realistic human representations.