MagicPose: Realistic Human Poses and Facial Expressions Retargeting with Identity-aware Diffusion

1. Introduction

• Earlier motion-transfer methods commonly used task-specific GANs and warping, which struggle to interpolate occluded body parts and generalize beyond training data.The paper motivates diffusion priors as a way to address the limited quality, identity preservation, and generalizability of recent diffusion-based approaches.Source: §1 · pp. 1–2
• MagicPose separates identity/appearance control from pose/motion control using an Appearance Control Model and a Pose ControlNet, trained with a multi-stage strategy.The Appearance Control Model provides reference-image guidance through a Multi-Source Attention Module; the Pose ControlNet provides pose and expression guidance.Source: §1 · p. 2
• MagicPose is presented as a pose- and expression-retargeting plug-in for Stable Diffusion, with a Multi-Source Attention Module for appearance guidance.The stated contributions also include a two-stage training strategy for disentangling appearance and pose control.Source: §1 · p. 2
• Out-of-domain experiments report generalization to diverse image styles and human poses.The paper also reports generalization to unseen identities and motions without fine-tuning.Source: §1 · p. 2
• TikTok experiments report superior pose-retargeting performance for MagicPose.The introduction identifies comprehensive experiments on the TikTok dataset but provides no metric values here.Source: §1 · p. 2

2.1. Human Motion/Expression Transfer

• GAN-based work advanced human motion and expression transfer, while DreamPose and DisCo applied Stable Diffusion to fashion-video synthesis and controllable human-dance generation.Earlier manual and 3D approaches required substantial time and labor.Source: §2.1 · p. 2

2.2. Image/Video Diffusion Models

• MagicPose’s pipeline trains appearance control and its Multi-Source Self-Attention Module first, then jointly fine-tunes appearance control with Pose ControlNet.An optional motion module can subsequently be integrated and fine-tuned for sequential output generation.Source: Figure 2 · p. 3
• Diffusion models support controllable image generation, and ControlNet adds conditioning such as pose keypoints or segmentation maps to Stable Diffusion.The related-work overview also describes extending latent diffusion models with a temporal dimension for video generation.Source: §2.2 · pp. 2–3

3. Preliminary

• Latent diffusion models run the diffusion process in an autoencoder’s lower-dimensional latent space rather than pixel space.The passage identifies Stable Diffusion as an example of a latent diffusion model.Source: Latent Diffusion Models (Rombach et… · p. 3
• Stable Diffusion encodes an image as z0, adds noise through diffusion steps, and learns to denoise the resulting latent using text conditioning.The UNet predicts noise at timestep t; its modules include residual blocks and self- and cross-attention mechanisms.Source: Latent Diffusion Models (Rombach et… · p. 3
• ControlNet controls Stable Diffusion’s generated layout by learning feature residuals from a replicated encoder, without modifying the original model parameters.The passage notes its application to pose-conditioned human image generation.Source: Latent Diffusion Models (Rombach et… · p. 3

4. MagicPose

• MagicPose takes a reference image and target pose skeleton and facial landmarks, aiming to transfer the person’s and background’s appearance while controlling pose and expression.The design inherits as much structure and parameters as possible from pre-trained Stable Diffusion models.Source: §4 · pp. 3–4

4.1. Exploration of Appearance Control Mechanism

• Connected self-attention produces appearance resemblance without fine-tuning, unlike vanilla ControlNet, but the zero-shot results are unstable.The authors suggest that UNet self-attention transmits appearance information spatially; this is presented as a plausible explanation.Source: §4.1 · p. 4

4.2. Appearance Control Pretraining

• The Appearance Control Model extends zero-shot attention with task-specific parameters for stability, using a UNet branch that shares key-value pairs with the Stable Diffusion UNet.During pretraining, the original UNet is fixed and only the Appearance Control module is trained; the pose ControlNet is not included.Source: §4.2 · p. 4

4.3. Appearance-disentangled Pose Control

• Directly combining pre-trained OpenPose ControlNet with the Appearance Control Model yields severe pose errors, so MagicPose jointly fine-tunes them to separate pose control from appearance guidance.The Pose ControlNet modulates pose attributes while the Appearance Control Model focuses on appearance.Source: §4.3 · pp. 4–5

5.1. Datasets

• The TikTok experiments use 335 videos for training and 10 TikTok-style videos of different people for testing; additional datasets probe in-the-wild appearance and full-body motion.The self-collected out-of-domain images test in-the-wild appearance, while Everybody Dance Now tests generalization to in-the-wild full-body motion.Source: §5.1 · p. 5

5.2. Implementation Details

• Training pretrains appearance control for 10k steps, then jointly fine-tunes appearance and pose control for 20K steps, with Stable Diffusion U-Net weights frozen.Both stages use eight NVIDIA A100 GPUs; evaluation uses center cropping instead of the random training crops.Source: §5.2 · p. 5

5.3. Qualitative and Quantitative Comparison

• MagicPose reaches a Face-Cos score of ∼0.426, +0.260 over Disco, and the authors report improvements across all evaluated image-quality metrics.Face-Cos measures cosine similarity between AdaFace features extracted from generated and ground-truth faces.Source: §5.3 · p. 5
• In the user study, participants chose MagicPose most often for identity preservation among the compared methods.The study collected votes from 100 participants for eight test subjects.Source: Table 2 · p. 7, §5.3 · p. 7

5.4. Ablation Analysis

• Appearance Control Pretraining improves Face-Cos by +944.73% and SSIM by +149.82%; Appearance-disentangled Pose Control adds +7.30% and +3.43%, respectively.Randomly masking facial landmarks and hand poses improves Face-Cos by +2.20% and SSIM by +0.13%; Image-CFG improves them by +56.62% and +14.11%.Source: §5.4 · p. 7, Table 3 · p. 8

5.5. Generalization Ability

• MagicPose generalizes to out-of-domain styles and unseen real-human examples, and performs on full-body images despite training on upper-body TikTok data.The authors report better generation quality after fine-tuning on specific datasets.Source: §5.5 · pp. 7–8

6. Conclusion

• MagicPose combines human pose and facial-expression transfer and generates consistent in-the-wild animations without further fine-tuning.The conclusion describes generalization over diverse human identities and complex motion sequences.Source: §6 · p. 8
• The authors present MagicPose as a plug-in module or extension compatible with models such as Stable Diffusion.Source: §6 · p. 8

Impact Statement

• MagicPose may support avatars and digital characters in virtual meetings, online classrooms, social networks, and entertainment media.The statement presents these as potential applications, including more lifelike characters in films, games, and animation.Source: Impact Statement · p. 9
• The experiments report generalization across real-human ethnicities and ages and to cartoon, painting, and AI-generated images.Source: Impact Statement · p. 9
• The method could be misused to create fake animated videos for fraud; the statement proposes watermarking, detection, legal measures, and media-literacy education as safeguards.Source: Impact Statement · p. 9

A. Detailed User Study

• One hundred English-speaking Prolific participants evaluated outputs for eight subjects, choosing the method that best preserved identity and appearance.Methods were anonymized and their presentation order randomized for each subject.Source: A. Detailed User Study · p. 12
• The authors report that participants preferred MagicPose over prior methods for identity preservation.Source: A. Detailed User Study · p. 12
• The chi-square test reports 116.02 with p-value 1.11 × 10^-12 and 28 degrees of freedom, indicating an association between subject and method choice.Source: A. Detailed User Study · p. 12

B.1. TikTok

• Figures 10–14 show additional TikTok test-set retargeting examples, while Figure 9 examines styles unlike the TikTok training set.Source: §B.1 · p. 12, Figure 9 · p. 12

B.2. EverybodyDanceNow

• Figures 15–16 provide zero-shot generation examples on the Everybody Dance Now dataset.Source: §B.2 · p. 12
• MagicPose receives the highest identity-preservation vote share for all eight subjects in the 100-participant user study.The accompanying visualizations cover subjects 1–8.Source: Table 5 · p. 13, Figure 10 · p. 13, Figure 11 · p. 14

B.3.1. OUT-OF-DOMAIN IMAGES

• Figures 9 and 17–19 show zero-shot generations for out-of-domain images.Source: §B.3.1 · p. 13

B.3.2. COMBINE WITH T2I MODEL

• MagicPose can edit images generated by a text-to-image model, with Figure 20 illustrating identity-preserving generation without further fine-tuning.The model was trained on real-human dance videos.Source: §B.3.2 · p. 13

C. Sequence Generation with Motion Module

• An optional motion module improves temporal consistency, but AnimateDiff required fine-tuning for complex dance motion.The motion module was fine-tuned for 30k steps; the method also uses a special DDIM sampling strategy for smoother, more consistent video generation.Source: C. Sequence Generation with Motion… · pp. 13–14

• OpenPose detection can be incomplete under rapid movement, occlusion, or partial visibility, affecting pose control and generated-image quality and temporal consistency.The paper suggests adopting a more advanced pose detector for better image-editing quality.Source: D. Limitations · p. 14

• MagicPose separates preservation of reference appearance and identity from editing geometry and structure under pose-skeleton guidance.Source: E. Discussion on motivation and future… · p. 14
• The proposed attention design is suggested for object novel-view synthesis, geometry-guided scene manipulation, and animal motion transfer.Source: E. Discussion on motivation and future… · pp. 14, 17
• TikTok examples show vivid, realistic motion and expressions under diverse pose-skeleton and face-landmark inputs while maintaining reference identity.Source: Figure 13 · p. 16

Comparison with ControlNet

• Unlike ControlNet’s geometry and structure control, MagicPose’s Appearance Control Model supplies identity and appearance information regardless of text input.It replicates a whole UNet and controls Stable Diffusion through attention layers, allowing information exchange among distant pixels.Source: F. Comparison to prior works · p. 17

Comparison with MasaCtrl and Reference Only ControlNet

• MasaCtrl and Reference Only ControlNet are described as inference-only and text-guided, whereas MagicPose can be fine-tuned on custom data and generate consistently without text prompts.MagicPose does not need additional text information for appearance guidance; appearance comes from the reference image.Source: F. Comparison to prior works · pp. 17, 19

• Supplementary figures cover zero-shot human-motion transfer, animation, cartoon generation, and text-to-image edits; the paper reports out-of-domain generation without further fine-tuning after real-human dance-video training.Source: Figure 15 · p. 18, Figure 17 · p. 20, F. Comparison to prior works · p. 20