Source-linked AI summary
HunyuanPortrait: Implicit Condition Control for Enhanced Portrait Animation
Zunnan Xu, Zhentao Yu, Zixiang Zhou, Jun Zhou, Xiaoyu Jin, Fa-Ting Hong, Xiaozhong Ji, Junwei Zhu, Chengfei Cai, Shiyu Tang, Qin Lin, Xiu Li, Qinglin Lu
TL;DR
Portrait animation must control facial expression and head pose while preserving identity across facial geometries and image styles. HunyuanPortrait uses implicit motion representations with stable video diffusion and attention-based adapters, achieving reported improvements in temporal consistency, controllability, identity preservation, and video quality, while remaining limited for full-body animation and unknown viewpoints.
Problem
Existing portrait-animation methods have limited control, temporal consistency, and identity preservation when facial geometries and motions vary.
Method
HunyuanPortrait decouples identity and motion with pretrained encoders, encodes motion implicitly, and injects appearance and motion controls into stable video diffusion through attention-based adapters.
Results
The method demonstrates improved temporal consistency, controllability, identity preservation, video quality, and generalization, with state-of-the-art subjective and objective performance reported by the authors.
Takeaways & Limitations
Implicit conditioning supports lifelike portrait animations that preserve identity while capturing detailed facial dynamics and head movement across challenging reenactment settings.
Takeaways & Limitations
The method is restricted to head-and-shoulder portraits, while full-body generation with hands can produce deformities and blurriness because hand regions are inadequately represented in the dataset.
Abstract
from arXiv · showhide
We introduce HunyuanPortrait, a diffusion-based condition control method that employs implicit representations for highly controllable and lifelike portrait animation. Given a single portrait image as an appearance reference and video clips as driving templates, HunyuanPortrait can animate the character in the reference image by the facial expression and head pose of the driving videos. In our framework, we utilize pre-trained encoders to achieve the decoupling of portrait motion information and identity in videos. To do so, implicit representation is adopted to encode motion information and is employed as control signals in the animation phase. By leveraging the power of stable video diffusion as the main building block, we carefully design adapter layers to inject control signals into the denoising unet through attention mechanisms. These bring spatial richness of details and temporal consistency. HunyuanPortrait also exhibits strong generalization performance, which can effectively disentangle appearance and motion under different image styles. Our framework outperforms existing methods, demonstrating superior temporal consistency and controllability. Our project is available at https://kkakkkka.github.io/HunyuanPortrait.
1. Introduction
HunyuanPortrait targets controllable, identity-consistent portrait animation despite diverse facial geometries and complex motion. It combines implicit conditional control with stable video diffusion and reports high-fidelity, temporally consistent animations.
- Portrait animation must preserve reference identity and background while following driving-video facial expressions and head poses.
- Earlier GAN-based methods suffer artifacts, motion distortion, background jitter, and blurred portraits when driving and source facial shapes differ substantially.
- Diffusion-based methods improve background handling and generalization but remain vulnerable to temporal defects and explicit-keypoint control limitations.
- HunyuanPortrait decouples identity and motion with a pretrained motion encoder, stable video diffusion, appearance and motion attention, and a frozen-parameter ID-aware Multi-scale Adapter.
- The authors report state-of-the-art subjective and objective performance, with improved temporal consistency, fidelity, precise control, and identity consistency.
2. Related Work
Related portrait-animation methods use predetermined motion representations, warping, or explicit keypoints, but can fail under large appearance differences and complex movements. HunyuanPortrait instead investigates implicit expression descriptors to adapt across face shapes and capture subtle dynamics.
- Human portrait animation uses driving sources such as facial landmarks or head-containing frames to bring still images to life.
- Non-diffusion methods can produce artifacts under large driving-source disparities and struggle with substantial head and body movements.
- Diffusion methods commonly fine-tune Stable Diffusion and use explicit keypoints, but may alter facial identity and miss subtle facial dynamics.
- This study investigates implicit expression descriptors to adapt to varied face shapes while capturing facial dynamics for speech and expressions.
3. Methodology
The methodology builds portrait animation on stable video diffusion with separate appearance and motion extraction. Implicit motion conditioning, intensity-aware features, memory-based refinement, and multi-scale identity adapters inject controllable signals while preserving portrait details.
- 3.1. Preliminaries: The diffusion backbone performs denoising in latent space, where a VAE encodes images and a UNet predicts noise conditioned on embeddings.
- 3.2. Overall Architecture: Stable video diffusion supplies appearance and motion extractors for controlling portrait identity, background, facial dynamics, and fine-grained visual details.
- 3.3. Intensity-Aware Motion Extractor: Implicit expression descriptors encode identity-agnostic dynamics such as lip synchronization, micro-expressions, eye gaze, and blinking from cropped central-face regions.
- 3.3. Intensity-Aware Motion Extractor: Expression and head-pose intensities are discretized into 64 levels, embedded, concatenated, and used to adjust features according to motion intensity.
- 3.3. Intensity-Aware Motion Extractor: Motion features are refined with adaptive normalization, multi-head cross-attention, and learnable memory before entering the denoising U-Net through cross-attention.
- 3.4. Appearance Extractor: The ID-aware Multi-scale Adapter downsamples appearance features, processes them with multi-scale convolutions and cross-attention, and merges identity-aware features back into patch features.
- 3.4. Appearance Extractor: During training, identity information comes from a random video frame; during inference, it comes from the reference image.
- 3.5. Style Adaptation: AnimeGANv3 style transfer augments the training data with anime-stylized portraits for adaptation to varied image styles.
4. Experiments
HunyuanPortrait is evaluated through quantitative metrics, qualitative comparisons, and component ablations for portrait video quality, motion accuracy, identity preservation, and temporal smoothness.
- Experimental Setting: The evaluation measures self-reenactment with video quality, fidelity, perceptual, and landmark metrics, while cross-reenactment uses expression, pose, and identity metrics.Self-reenactment uses FID, FVD, PSNR, SSIM, LPIPS, and LMD; cross-reenactment uses AED, APD, and ArcFace identity similarity.
- Quantitative Analysis: HunyuanPortrait attains the lowest FID-VID and FVD in Table 1, with a significant advantage over the previously top-performing method.The reported comparison emphasizes generated-video quality.
- Quantitative Analysis: In cross-reenactment, HunyuanPortrait substantially improves all metrics and demonstrates stronger generalization in the more challenging setting.The authors characterize cross-reenactment as closer to real-life application scenarios than self-reenactment.
- Visualization: Qualitatively, HunyuanPortrait captures eye gaze, eye rotation, lip synchronization, and overall head rotation while preserving portrait identity and facial shape.The comparison highlights detailed facial dynamics and head movement that previous methods struggle to reproduce consistently.
- Ablation Study: The motion memory bank improves FID-VID, FVD, facial movements, and smoothness by enriching motion features with context from preceding and subsequent frames.Ablation evidence also links component removal to less accurate micro-expressions, including gaze direction.
5. Conclusion
The framework uses implicit representations with stable video diffusion to produce temporally consistent, precisely controlled, natural portrait animations.
- Implicit representations and stable video diffusion support temporal consistency and precise control over facial dynamics.
A. Benchmark Metrics Details
The evaluation uses qualitative and quantitative analyses to assess generated-video quality and motion accuracy, including three self-reenactment metrics.
- Self-reenactment is evaluated with PSNR, SSIM, and LPIPS metrics.LPIPS uses an AlexNet-based perceptual similarity measure.
B. Discussions, Limitations and Future work
The discussion compares identity-preservation designs and identifies limitations in unknown-view identity consistency, full-body generation, and computational efficiency.
- Discussions: The fine-grained appearance extractor and IMAdapter model identity and background information from the reference image for high-fidelity preservation.
- Discussions: X-Portrait extracts appearance and background features for expressive animation, while DiffPortrait3D injects appearance context for 3D-consistent novel views.
- Limitations and Future work: Identity consistency is limited from unknown perspectives after significant head rotation, and DiffPortrait3D remains constrained to specific angles.
- Limitations and Future work: The method is restricted to head-and-shoulder portraits because hand regions are inadequately represented, causing occasional hand deformities and blurriness in full-body attempts.
- Limitations and Future work: Diffusion-model computational costs impede real-time applicability, motivating future acceleration through model distillation.
C. More Implementation Details
Implementation uses a frozen DiNOv2-Large appearance extractor, multi-scale convolution and cross-attention in the IMAdapter, and a fixed pretrained motion encoder.
- The appearance extractor resizes reference images to 256x256 and uses frozen DiNOv2-Large weights with four register tokens.
- The IMAdapter uses 1 × 1, 3 × 3, and 5 × 5 convolutions, fuses ID features through eight-head cross-attention, and reduces features to 384 dimensions.
- The motion extractor has six blocks, 768-dimensional latent features, eight attention heads, and SiLU activations.
- The motion encoder is upgraded from ResNet-18 to ResNet-50, pretrained, then fixed while fine-tuning stable video diffusion.
D. More Visualizations
Figure 6 presents additional self-reenactment and cross-reenactment visualizations to demonstrate HunyuanPortrait’s appearance–motion decoupling. The broader demonstrations also target generalization across varied image and video styles.
- The visualizations cover both self-reenactment and cross-reenactment settings.
- These results are intended to demonstrate decoupling between appearance and motion.
- The examples use images and videos from varied style domains, including Civitai, Bilibili, and VFHQ.
E.1. User Study Details
The user study evaluates generated videos through participant ratings of facial movement, video quality, and temporal smoothness. It involved 120 experienced participants using five-point rating scales, with example videos provided to standardize criteria.
- 120 experienced participants took part in the user study.
- Participants rated Facial Movement, Video Quality, and Temporal Smoothness on five-point scales.
- Example videos illustrated generation quality and helped ensure consistent rating criteria, but were excluded from the ratings.
E.2. Societal Impacts and Responsible AI
The paper frames HunyuanPortrait as a tool for beneficial virtual-avatar applications while acknowledging risks from misuse and impersonation. It also notes that generated videos retain identifiable traces of the source individuals.
- The stated goal is to advance virtual AI avatars for beneficial applications.
- The authors oppose misuse that creates deceptive or harmful content through impersonation of real individuals.
- Proposed positive applications include educational fairness, communication assistance, companionship, and therapeutic aid.
- Generated videos retain identifiable traces of the actual individuals on which they are based.