Source-linked AI summary
HunyuanVideo-Avatar: High-Fidelity Audio-Driven Human Animation for Multiple Characters
Yi Chen, Sen Liang, Zixiang Zhou, Ziyao Huang, Yifeng Ma, Junshu Tang, Qin Lin, Yuan Zhou, Qinglin Lu
TL;DR
Audio-driven human animation still faces challenges in dynamic yet consistent motion, emotion alignment, and multi-character control. HunyuanVideo-Avatar addresses these gaps with three MM-DiT-based modules, and author-reported results show improvements across motion, consistency, emotion alignment, lip synchronization, and multi-character scenarios. Its current workflow remains limited by emotion-reference-image dependence and slow inference.
Problem
Audio-driven human animation remains limited by dynamic-motion and character-consistency trade-offs, emotion-audio misalignment, and multi-character scene control.
Method
HunyuanVideo-Avatar uses character image injection, an Audio Emotion Module, and a Face-Aware Audio Adapter within an MM-DiT-based framework.
Results
HunyuanVideo-Avatar outperforms existing methods in video dynamism, subject consistency, lip-sync accuracy, audio-emotion-expression alignment, and multi-character scenarios.
Takeaways & Limitations
The framework supports dynamic single- or multi-character talking videos with localized audio control and emotion-controllable facial expressions.
Takeaways & Limitations
The method relies on emotion reference images, limiting dynamic emotional changes within one audio segment, and generating a 10s 720×1216 video takes approximately 60 minutes with 50 inference steps.
Abstract
from arXiv · showhide
Recent years have witnessed significant progress in audio-driven human animation. However, critical challenges remain in (i) generating highly dynamic videos while preserving character consistency, (ii) achieving precise emotion alignment between characters and audio, and (iii) enabling multi-character audio-driven animation. To address these challenges, we propose HunyuanVideo-Avatar, a multimodal diffusion transformer (MM-DiT)-based model capable of simultaneously generating dynamic, emotion-controllable, and multi-character dialogue videos. Concretely, HunyuanVideo-Avatar introduces three key innovations: (i) A character image injection module is designed to replace the conventional addition-based character conditioning scheme, eliminating the inherent condition mismatch between training and inference. This ensures the dynamic motion and strong character consistency; (ii) An Audio Emotion Module (AEM) is introduced to extract and transfer the emotional cues from an emotion reference image to the target generated video, enabling fine-grained and accurate emotion style control; (iii) A Face-Aware Audio Adapter (FAA) is proposed to isolate the audio-driven character with latent-level face mask, enabling independent audio injection via cross-attention for multi-character scenarios. These innovations empower HunyuanVideo-Avatar to surpass state-of-the-art methods on benchmark datasets and a newly proposed wild dataset, generating realistic avatars in dynamic, immersive scenarios.
1 Introduction
HunyuanVideo-Avatar addresses persistent limitations in audio-driven human animation by combining dynamic, consistent motion, emotion alignment, and localized multi-character audio control. Its three modules target character conditioning, affective expression, and face-specific audio driving.
- Existing methods struggle with unnatural full-body motion, audio-emotion misalignment, and multi-character audio driving.
- The character image injection module improves dynamic motion and character consistency by replacing conventional reference-image conditioning.It transforms image features into representations injected along the channel dimension.
- The Audio Emotion Module transfers affective cues from reference images so facial expressions better align with audio emotions.
- The Face-Aware Audio Adapter masks latent face regions to independently drive different characters with distinct audio inputs.This supports realistic multi-character dialogue generation.
- Extensive experiments improve dynamism and consistency in generated multi-person audio-driven scenarios.
2 Related work
Related work spans audio-conditioned portrait and full-body animation, with full-body methods adding gestures and structural priors to improve synchronized video synthesis. These approaches motivate broader solutions for dynamic human animation.
- Audio-conditioned portrait animation: SadTalker generates audio-driven 3D motion coefficients for head pose and expression, targeting natural movement, expression fidelity, and identity preservation.
- Audio-conditioned full-body animation: DiffTED produces synchronized talking-head animations with natural co-speech gestures from a single image using keypoint-guided motion modeling.
- Audio-conditioned full-body animation: CyberHost combines region attention, identity-specific local features, and human structural priors to improve key-area synthesis and motion stability.
3 Methods
HunyuanVideo-Avatar uses an MM-DiT video-generation framework with character-image injection, face-aware audio conditioning, emotion alignment, and long-video fusion. These components target dynamic, consistent, emotionally aligned, and multi-character audio-driven videos.
- Framework: The framework combines Character Image Injection, Audio Emotion, and Face-Aware Audio Adapter modules for consistency, emotion alignment, and multi-character animation.The method uses HunyuanVideo as its MM-DiT backbone and conditions generation on a reference image, driving audio, and facial mask.
- Character Image Injection Module: Character image injection combines replicated reference-image features with noisy video features and concatenates the resulting tokens with reference-image tokens.The reference image is encoded through a pretrained 3D VAE and tokenizers before forming the model input.
- Character Image Injection Module: The injection design addresses the dynamism-consistency trade-off associated with padding frames and direct reference conditioning during inference.The explored mechanisms compare token concatenation, channel-wise injection, and direct latent addition; the selected approach avoids training-inference condition mismatch described for conventional conditioning.
- Face-Aware Audio Adapter: Face-aware audio conditioning temporally and spatially aligns audio features and face masks with compressed video latents before cross-attention audio injection.The audio influence is scaled by αA, and masking localizes audio-driven changes to targeted face regions.
- Audio Emotion Module: The Audio Emotion Module injects emotional reference-image features through an FC layer and spatial cross-attention, with the Double Block better supporting emotion expression than the Single Block.The emotional reference supplies Key and Value features while the video latent supplies the Query; γE controls their influence.
- Long Video Generation: Timeaware Position Shift Fusion extends HunyuanVideo-13B beyond its 129-frame output limit without extra inference or training costs and reduces jitter and abrupt transitions.The method uses a starting offset that shifts forward by α=3-7 steps to connect successive audio-conditioned segments.
4 Experiment
Experiments evaluate HunyuanVideo-Avatar on portrait and full-body animation using objective metrics, user studies, qualitative comparisons, and module ablations. The results report strong video quality, motion, synchronization, consistency, emotion alignment, and multi-character driving, with noted deficiencies against OmniHuman-1 on facial and full-body naturalness.
- Experiment Settings: 500,000 audio-character training samples totaling approximately 1,250 hours support evaluation on CelebV-HQ, HDTF, and a wild full-body test set.The data pipeline filters audio-visual asynchrony, low brightness, and low aesthetics before training.
- Qualitative Results: Qualitative comparisons report higher video quality, more natural facial expressions, and more aesthetically pleasing effects on HDTF.The comparison includes Sonic, EchoMimic, EchoMimicV2, and Hallo-3.
- Qualitative Results: Wild full-body comparisons show more natural foreground, background, and character motion, with more accurate lip synchronization and better character consistency.The comparison includes Hallo3, FantasyTalking, and OmniHuman-1.
- Quantitative Results: The method achieves the best performance in FID, FVD, IQA, ASE, and Sync-C for portrait animation, and performs best on most metrics for full-body animation.These evaluations use CelebV-HQ, HDTF, and the proposed test set.
- User Study: In a 30-participant user study, HunyuanVideo-Avatar outperforms baselines on Identity Preservation and lip synchronization but trails OmniHuman-1 on Facial Naturalness and Full-body Naturalness.Participants rated lip synchronization, identity preservation, full-body naturalness, and facial naturalness on a 1-to-5 scale.
- Ablation Study: Ablations indicate that character image injection improves video dynamics and character consistency, while AEM and FAA improve emotion transfer and localized multi-character audio driving.AEM transfers emotional information to facial expressions, and FAA uses face masks to drive one selected character at a time.
5 Conclusion
HunyuanVideo-Avatar combines character consistency, dynamic motion, emotion alignment, and multi-character audio-driven animation. The reported results indicate stronger performance across video dynamism, subject consistency, and synchronization-related capabilities.
- HunyuanVideo-Avatar is designed to achieve high character consistency together with dynamic motion in audio-driven human animation.
- The Audio Emotion Module transfers affective cues from emotion reference images to target animations, aligning character expressions with audio.
- The Face-Aware Audio Adapter localizes audio-driven animation to targeted face regions, enabling independent control of multiple characters.
- HunyuanVideo-Avatar outperforms existing methods in video dynamism, subject consistency, lip-sync accuracy, audio-emotion-expression alignment, and multi-character scenarios.
6 Appendix
The appendix presents visual evidence across multi-character dialogue, realistic and diverse character styles, and emotion control, while documenting limitations in emotional input and inference speed.
- More Visualization Results: Figure 8 covers multi-character scenarios including crosstalk, singing, and walking conversations.
- More Visualization Results: Figure 9 shows realistic human images while maintaining character consistency and enhancing dynamics.
- More Visualization Results: Figure 10 demonstrates generalization across LEGO, Chinese painting, anime, and pencil-sketch character styles.
- More Visualization Results: Figure 11 illustrates control of happiness, sadness, excitement, and anger to align generated animation with audio emotions.
- Summary: The authors report that the method offers multi-character and emotion-control animation while outperforming prior methods in character consistency and video dynamics.
- Limitations: The method relies on emotion reference images, which complicates operation and cannot represent dynamic emotional changes within one audio segment.
- Limitations: Generating a 10s video at 720×1216 resolution with 50 inference steps takes approximately 60 minutes, preventing real-time application requirements.
7 Contributors
The listed contributors include project leaders, core contributors, and contributors. The supplied appendix figure references additionally point to multi-character, realistic-scenario, diverse-style, and emotion-control visualizations.
- Contributors: Qinglin Lu, Qin Lin, and Yuan Zhou are listed as project leaders.
- Contributors: Yi Chen, Sen Liang, Zixiang Zhou, Ziyao Huang, and Yifeng Ma are among the listed core contributors.
- Contributors: The contributor list also includes Zhentao Yu, Zhengguang Zhou, Teng Hu, Zhiyao Sun, and additional named contributors.
- Appendix Visualizations: The referenced figures cover multi-character animation, realistic scenarios, diverse character styles, and emotion control.