HunyuanVideo-Avatar: High-Fidelity Audio-Driven Human Animation for Multiple Characters

Introduction

• Full-body animation still faces unnatural movement, emotion-audio misalignment, and difficulty driving multi-character scenes, while portrait methods omit body motion.These limitations motivate the paper’s objectives of dynamic expressiveness with identity preservation, synchronized emotion, and realistic multi-character dialogue.Source: §1 · p. 2
• Character image injection places transformed image features along the channel dimension, avoiding direct latent-space conditioning’s dynamism-consistency trade-off and aligning training with inference.The module is designed to prevent reference expressions and poses from suppressing dynamic, audio-aligned movement.Source: §1 · pp. 2–3
• The Audio Emotion Module uses reference images to map emotion onto generated characters, aligning facial expressions with affective cues in the audio.The paper presents this emotion alignment as a way to improve realism in facial expression.Source: §1 · pp. 2–3
• The Face-Aware Audio Adapter masks latent face regions and fuses them with audio, allowing distinct audio inputs to drive different characters independently.This localized conditioning supports multi-character dialogue generation.Source: §1 · pp. 2–3
• Extensive experiments show that HunyuanVideo-Avatar drives multi-person audio scenarios while improving video dynamism and consistency.The introduction presents these findings as evidence that the framework addresses multi-person generation alongside motion quality and character consistency.Source: §1 · p. 3

Audio-conditioned portrait animation

• Portrait methods target motion and identity consistency (SadTalker [40], Hallo3 [7]), synchronization and control (Hallo [39], V-Express [33]), natural motion (EchoMimic [24], Loopy [16]), and mixed-condition training (OmniHuman-1 [21]).EchoMimic combines audio and facial landmarks, Loopy uses temporal and audio-to-latents modules, and OmniHuman-1 uses hybrid multimodal motion-condition training.Source: §2 · p. 3

Audio-conditioned full-body animation

• Full-body methods use keypoint-guided diffusion for coherent co-speech gestures (DiffTED) and region attention with human-structure priors for stable synthesis (CyberHost [20]).DiffTED uses Thin-Plate Spline motion modeling; CyberHost combines identity-independent latent features with identity-specific local visual features.Source: §2 · p. 3

3.1 Preliminaries

• The model forms its input by concatenating tokenized reference-image features with video latents that combine noise and a repeated reference image.It places the image latent before the first video frame using 3D-RoPE and spatially shifts it to discourage copying the reference directly; training uses Flow Matching with mean-squared error on predicted velocity.Source: §3.1 · pp. 4–5

3.2 Character Image Injection Module

• Mechanism (c), which adds a projected reference-image latent to the video latent, improves motion dynamics while preserving character, background, and foreground consistency.The added image tokenizer is initialized from the backbone tokenizer weights, which the authors report accelerates convergence.Source: §3.2 · p. 5

3.3 Face-aware Audio Adapter

• Audio features and face masks are aligned with compressed video latents, enabling temporally isolated, spatial cross-attention to inject audio into each frame.Whisper extracts audio features, InsightFace detects face boxes, and the audio features are aggregated in groups of four to match the compressed latent sequence.Source: §3.3 · pp. 5–6

3.4 Audio Emotion Module

• The Audio Emotion Module encodes an emotion-reference image and injects its features through spatial cross-attention, with the video latent as Query and reference features as Key and Value.The module works better in a Double Block than a Single Block for learning and expressing emotional cues.Source: §3.4 · p. 6

3.5 Long Video Generation

• To extend generation beyond HunyuanVideo-13B’s 129-frame limit, the method adapts Sonic’s Timeaware Position Shift Fusion, which adds no inference or training cost and reduces jitter and abrupt transitions.At each timestep, audio segments are connected using a starting offset that advances by 3–7 steps.Source: §3.5 · pp. 6–7

4.1 Experiment Settings

• Training uses audio-only data first, then mixed audio and image data at a 1:1.5 ratio to improve motion stability.The model uses HunyuanVideo-I2V [17] as its base; LLaVA and the 3D VAE remain fixed while other learnable parameters are updated. Training uses 160 GPUs, batch size 40, and learning rate 1e-5.Source: §4.1 · pp. 7–8
• The training set contains 500,000 samples totaling about 1,250 hours; testing uses CelebV-HQ, HDTF, and a 250-video full-body set spanning 200 identities.Evaluation uses Q-align IQA and ASE, FID, FVD, VBench smoothness, Sync-C, and 30-user ratings; baselines include Sonic, EchoMimic, EchoMimic-V2, Hallo-3, OmniHuman-1, and FantasyTalking.Source: §4.1 · pp. 8–9

4.2 Comparison with State-of-the-Art Methods

• Qualitative comparisons report higher-quality, more natural portrait expressions and more natural full-body motion, lip synchronization, and character consistency.The full-body comparison is on the wild dataset against Hallo3, FantasyTalking, and OmniHuman-1.Source: Figure 4 · p. 7, Figure 6 · p. 8, §4.2 · p. 9
• Portrait animation achieves the best FID, FVD, IQA, ASE, and Sync-C results; full-body animation performs best on most evaluation metrics.Portrait tests use CelebV-HQ and HDTF; full-body tests use the proposed test set.Source: §4.2 · p. 9, Table 1 · p. 10
• In a 30-person study, HunyuanVideo-Avatar outperforms baselines on IP and LS but has deficiencies versus OmniHuman-1 on FCN and FBN.The paper attributes OmniHuman-1’s subjective visual advantage partly to its online service’s super-resolution, and notes inherited HunyuanVideo issues.Source: §4.2 · p. 9, Table 2 · p. 10

4.3 Ablation Study And Discussion

• The Character Image Injection Module ablation reports better video dynamics and character consistency than the alternatives evaluated.The subjective comparison assesses lip synchronization, video quality, identity preservation, and motion diversity.Source: §4.3 · p. 9, Table 3 · p. 10
• Without the Audio Emotion Module, text-only emotion guidance does not effectively convey emotion to the face; reference-image injection improves emotion transfer and audio-expression alignment.Source: §4.3 · p. 10
• Without a face mask, audio affects both characters; applying and moving the mask directs audio-driven animation to the selected character’s face.This enables audio-driven animation of multiple characters.Source: §4.3 · p. 10

• Character image injection adaptively balances dynamism and consistency, enhancing the naturalness and diversity of generated videos.Source: §5 · p. 10
• The Audio Emotion Module transfers affective cues from reference images, while latent-space face masking localizes audio-driven animation for independent character control.Source: §5 · p. 10
• HunyuanVideo-Avatar outperforms existing methods in video dynamism, subject consistency, lip-sync accuracy, audio-emotion-expression alignment, and multi-character scenarios.Source: §5 · p. 11

6.1 More Visualization Results

• The appendix reports robust multi-character generation, preserved character consistency with stronger dynamics, style generalization, and emotion-aligned control, summarizing practical advantages over prior methods [35] [21] [16] [7] [15].The examples include crosstalk, singing, and walking conversations; LEGO, Chinese painting, anime, and pencil-sketch styles; and happiness, sadness, excitement, and anger.Source: §6.1 · p. 11

6.2 Limitations and Societal Impacts

• Emotion control requires reference images and cannot reflect changing emotions within audio; generating a 10s video at 720×1216 takes approximately 60 minutes, motivating direct audio emotion inference and faster, interactive generation.The authors identify real-time interactive animation with contextual awareness as a future direction.Source: §6.2 · pp. 11–12
• Real-time digital-human deployment is hindered by generative models’ difficulty producing diverse, natural actions and expressions, as well as large models’ slow inference.Source: §6.2 · p. 12