Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer

1. Introduction

• Portrait animation generates facial expressions, lip movements, and head poses from portrait images, with applications including film, games, social media, and online education.Motion signals can include audio, text prompts, facial keypoints, and dense motion flow.Source: §1 · p. 1
• Earlier methods used facial landmarks and 3D morphable models, while GAN and diffusion approaches advanced realistic detail, character generalization, and long-term identity preservation.Explicit intermediate facial representations remain constrained by expression and pose reconstruction accuracy and the richness of resulting expressions.Source: §1 · p. 1
• Existing methods often rely on frontal, centered portraits, struggle with profile, overhead, or low-angle views and moving accessories, and assume static backgrounds.Those assumptions limit authentic effects in dynamic scenes such as foreground campfires or crowded streets.Source: §1 · pp. 1–2
• Figure 1 depicts identity-preserving animation across frontal and varied perspectives, with dynamic foreground and background elements and temporal consistency.The demonstrations use a reference image, audio sequence, and textual prompt.Source: Figure 1 · p. 2
• Hallo3 is the first application of a pretrained DiT video generator to portrait animation, using a 3D VAE and transformer identity network, audio conditioning, and motion frames for long-duration extrapolation.Motion frames use each generated clip’s final frames as inputs for the next clip.Source: §1 · p. 2
• On HTDF and Celeb-V, Hallo3 reports results comparable to methods limited to frontal, centered faces, static backgrounds, and defined expressions.It also generates complex poses and dynamic foregrounds and backgrounds with realistic, smooth motion.Source: §1 · p. 2

2. Related Work

• Audio-driven portrait methods target synchronization, expression, and head motion, while latent-diffusion approaches improve output quality and generalization across identities.Other work emphasizes emotional expressiveness, temporal consistency, efficiency, and long-duration synthesis.Source: §2 · pp. 2–3
• U-Net video-generation methods use pretrained text-to-image models, 3D U-Nets, motion modules, or spatial, temporal, and textual conditions to improve efficiency, coherence, or controllability.Examples include Make-A-Video [27], MagicVideo [40], AnimateDiff [10], and VideoComposer [30].Source: §2 · p. 3
• Recent DiT-based video-generation approaches include CogVideoX [34] and Movie Gen [19]; in the present study, we adopt the DiT diffusion formulation to optimize generalization capabilities of generated video.Existing facial animation techniques still face limitations in handling extreme facial poses, dynamic backgrounds, and camera movements directed by textual prompts.Source: §2 · p. 3

3.1. Baseline Transformer Diffusion Network

• CogVideoX supplies the transformer diffusion backbone, combining video latents and T5 text embeddings with adaptive normalization and 3D rotational positional encoding.The model also conditions on speech audio and identity appearance.Source: §3.1 · p. 3

3.2. Audio-Driven Transformer Diffusion

• Wav2vec speech features are transformed into frame-specific representations, and cross-attention is the best-performing of the three explored audio-fusion strategies.The other strategies are self-attention and adaptive normalization; audio-attention layers are placed after face-attention layers in the denoising network.Source: §3.2 · pp. 3–4, Figure 3 · p. 4

3.3. Identity Consistent Transformer Diffusion

• Face attention combined with the identity reference network performs best; the reference network encodes the reference image with a causal 3D VAE and 42 transformer layers.Reference features are integrated into corresponding denoising layers to reinforce identity consistency across longer sequences.Source: §3.3 · p. 5
• For long-video inference, the method reuses the last n generated frames as motion frames, processes them into latents, and repeatedly conditions subsequent generation on them.The paper states this repeated use achieves temporally consistent long-video inference.Source: §3.3 · p. 5

3.4. Training and Inference

• Training proceeds in two phases: identity-consistent video generation, then audio-driven generation; inference uses reference images, audio, prompts, and the preceding video's last two frames.Each training phase produces 49-frame sequences.Source: §3.4 · p. 5

3.5. Dataset

• The curated training set draws on HDTF, YouTube, and movie videos, which are filtered for speaker, motion, and pose before cropping and feature extraction.The final set contains about 134 hours: 6 from HDTF, 72 from YouTube, and 56 from movies.Source: §3.5 · pp. 5–6

4.1. Experimental Setups

• The experiments include complex poses and dynamic foreground or background scenes; training used two stages of 20,000 steps each, with audio embeddings dropped at probability 0.05 and motion frames masked at 0.25.The reported setup uses 480 x 720 training video and generates 49 frames at a time.Source: §4.1 · pp. 6–7, Figure 7 · p. 7, Figure 8 · p. 7

4.2. Comparison with State-of-the-art

• On the wild set, head and background dynamic degree reached 13.286 and 4.481 with highest lip-synchronization accuracy; on HDTF and Celeb-V, the method achieved best FID and FVD but showed some lip-sync disparity.The authors relate the benchmark lip-sync disparity to training data with greater head and body movement and more varied dynamic scenes.Source: §4.2 · p. 7
• On wild examples, competing methods struggled with side faces and foreground interactions, while Hallo3 produced varied orientations and complex foreground and background scenes.Figure 7 compares methods on wild portraits; Figure 8 shows Hallo3 outputs in complex scenes and poses.Source: §4.2 · pp. 7–8, Figure 7 · p. 7, Figure 8 · p. 7

4.3. Ablation Study and Discussion

• Audio cross-attention improved lip synchronization, particularly local audio-visual alignment around the lips, and also contributed to video quality.The audio-conditioning strategies are compared in Table 4 and Figure 9.Source: §4.3 · p. 8, Table 4 · p. 8, Figure 9 · p. 8
• Without identity conditioning, the model fails to preserve portrait appearance; using face embeddings alone introduces blur and distortion because it focuses solely on facial features and disrupts global visual context.Table 5 and Figure 10 evaluate different identity-conditioning strategies.Source: §4.3 · p. 8, Table 5 · p. 8, Figure 10 · p. 8
• One motion frame achieved the highest Sync-C score (6.889) and lowest Sync-D score (8.695) in the temporal-frame analysis.Table 6 reports the ablation on the number of motion frames.Source: §4.3 · p. 8, Table 6 · p. 8

• Cross-attention audio conditioning captures relationships between audio and facial expressions to support lip synchronization.Source: §5 · p. 8
• An identity reference network is used to preserve facial identity across video sequences.Source: §5 · p. 8
• Motion frames support long-duration video extrapolation, while outputs blend dynamic foreground and background elements across diverse perspectives.Source: §5 · p. 8