Source-linked AI summary
Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation
Mingwang Xu, Hui Li, Qingkun Su, Hanlin Shang, Liwei Zhang, Ce Liu, Jingdong Wang, Yao Yao, Siyu Zhu
TL;DR
Portrait image animation must align audio-driven lip, expression, and pose motion while maintaining high-fidelity, temporally consistent outputs. Hallo uses end-to-end diffusion with hierarchical audio-driven visual synthesis, supported by UNet denoising, temporal alignment, and ReferenceNet. Experiments report superior image and video quality, enhanced lip synchronization, and flexible motion-diversity control.
Problem
Portrait animation requires precise audio synchronization across lips, expressions, and poses while producing visually appealing, high-fidelity, temporally consistent videos.
Method
Hallo combines end-to-end diffusion with hierarchical cross-attention and adaptive weighting, UNet denoising, temporal alignment, and ReferenceNet.
Results
The method reports superior image and video quality, enhanced lip synchronization, and increased motion diversity across evaluations.
Takeaways & Limitations
The approach provides flexible control over expression and pose diversity while supporting high-quality, temporally coherent portrait animation across visual identities and challenging conditions.
Takeaways & Limitations
Future work identifies enhanced visual-audio synchronization and more robust temporal coherence as areas requiring further exploration.
Abstract
from arXiv · showhide
The field of portrait image animation, driven by speech audio input, has experienced significant advancements in the generation of realistic and dynamic portraits. This research delves into the complexities of synchronizing facial movements and creating visually appealing, temporally consistent animations within the framework of diffusion-based methodologies. Moving away from traditional paradigms that rely on parametric models for intermediate facial representations, our innovative approach embraces the end-to-end diffusion paradigm and introduces a hierarchical audio-driven visual synthesis module to enhance the precision of alignment between audio inputs and visual outputs, encompassing lip, expression, and pose motion. Our proposed network architecture seamlessly integrates diffusion-based generative models, a UNet-based denoiser, temporal alignment techniques, and a reference network. The proposed hierarchical audio-driven visual synthesis offers adaptive control over expression and pose diversity, enabling more effective personalization tailored to different identities. Through a comprehensive evaluation that incorporates both qualitative and quantitative analyses, our approach demonstrates obvious enhancements in image and video quality, lip synchronization precision, and motion diversity. Further visualization and access to the source code can be found at: https://fudan-generative-vision.github.io/hallo.
1 Introduction
Portrait image animation seeks speaking portraits from a static image and speech audio, but must synchronize lips, expressions, and poses while preserving visual fidelity and temporal consistency. Hallo addresses these challenges with end-to-end diffusion and hierarchical audio-driven synthesis, reporting improved quality, synchronization, and motion control.
- Portrait image animation generates a speaking portrait from one static image and corresponding speech audio.
- Lip, expression, and pose synchronization, alongside high-fidelity and temporally consistent animation, are the study’s two primary challenges.
- Intermediate facial or decoupled latent representations can limit expressiveness, disentanglement, realism, temporal consistency, and diversity.
- The hierarchical audio-driven visual synthesis module uses cross-attention and adaptive weighting to align audio with lip, expression, and pose features.
- The integrated diffusion architecture combines UNet denoising, temporal alignment, and ReferenceNet for identity-consistent generation with adaptive expression and pose control.
- Evaluations report improved image and video quality, lip synchronization, and motion diversity compared with previous approaches.
2 Related Work
Related work advances portrait animation through diffusion-based video generation and facial representation learning. Prior portrait-animation systems progressively improve synchronization, expressiveness, personalization, and spatial-temporal control across identities and video settings.
- Diffusion-Based Video Generation: Diffusion-based video-generation research extends text-to-image principles to video through space-time U-Nets, cascaded diffusion, and text-to-video systems.
- Facial Representation Learning: Facial representation learning seeks to disentangle identity-related appearance from motion attributes such as lip movements, expressions, and poses.
- Portrait Image Animation: Portrait-animation methods progressed from LipSyncExpert’s speech synchronization toward systems supporting personalized and generalized synthesis across identities.
- Portrait Image Animation: Diffusion-based methods such as DreamTalk and VividTalk emphasize expressive, high-quality animations, improved lip synchronization, and diverse speaking styles.
- Portrait Image Animation: Vlogger and AniPortrait add spatial and temporal controls for accommodating variable video lengths.
3 Methodology
The method uses latent diffusion for audio-driven portrait video generation, combining identity conditioning, temporal alignment, and hierarchical audio-visual attention. Audio guides lip, expression, and pose-related motion while reference features preserve portrait appearance.
- 3.1 Preliminaries: Latent diffusion encodes images into latent representations, denoises them with a conditioned U-Net, and decodes the reconstructed latent into the final image.The denoiser processes noisy latents and conditional embeddings during training, while deterministic sampling supports reconstruction after training.
- 3.1 Preliminaries: Cross attention combines latent representations with conditioning embeddings to focus generation on motion-relevant information.The mechanism dynamically adjusts its focus according to the evolving latent state and specified motion condition.
- 3.2 Hierarchical Audio-Driven Visual Synthesis: The task generates S-frame videos from a face image and corresponding audio, applying audio embeddings to the latent representations.The audio features are extracted with wav2vec and transformed through linear layers for frame-level conditioning.
- 3.2 Hierarchical Audio-Driven Visual Synthesis: Hierarchical audio-visual cross attention learns relationships between audio and lips, expressions, and poses using region masks.MediaPipe landmarks support construction of lip and expression masks, while masked latent representations are merged through an adaptive hierarchical audio-guidance module.
- 3.3 Network Architecture: The network combines a Stable Diffusion backbone, ReferenceNet, temporal alignment, and hierarchical audio-driven synthesis for coherent video generation.ReferenceNet uses existing images to guide portrait and background texture, while temporal alignment incorporates preceding motion frames to improve sequence consistency.
4 Experiment
The experiments evaluate the approach across datasets, portrait and audio styles, alignment mechanisms, personalization, controllability, and efficiency. Results report strong visual quality, synchronization, motion diversity, identity adaptation, and explicit trade-offs between quality and synchronization.
- Quantitative Results: 20.545 FID, 173.497 FVD, and 7.951 E-FID are the lowest reported values on HDTF, while Sync-C reaches 7.750 and Sync-D 7.659.These results are described as indicating high-quality, temporally coherent animations with lip synchronization close to real-video benchmarks.
- Quantitative Results: 44.578 FID, 377.117 FVD, and 78.495 E-FID are the lowest reported values on CelebV, while Sync-C reaches 7.191 and Sync-D 7.984.The results are reported as demonstrating realistic animations with temporal coherence and precise lip synchronization.
- Quantitative Results: 23.266 FID, 239.647 FVD, and 34.731 E-FID are the lowest reported values on the proposed wild dataset, while Sync-C reaches 6.924 and Sync-D 7.969.The Sync-D value is described as closely approximating real-video benchmarks.
- Qualitative Results: The method generates diverse portrait-style and audio-style outputs, including sketching, painting, AI-generated images, sculpture, and varied audio inputs.The qualitative results describe the generated videos as high-fidelity, visually coherent, versatile, and robust across these styles.
- Ablation Study: Hierarchical audio-visual cross attention aligns audio with lip motion and facial expressions, while integrating lip, expression, and pose features yields the strongest overall ablation improvement.The reported alignment also supports more precise motion control during inference.
- Ablation Study: Increasing lip weight improves lip-sync accuracy but slightly reduces fidelity and video quality, whereas increasing expression weight marginally improves quality while slightly affecting synchronization.The study therefore selects weights by balancing fidelity, video quality, and synchronization.
5 Conclusion
The paper presents an end-to-end diffusion method for portrait image animation that improves audio-visual alignment and animation quality. Its hierarchical synthesis module supports flexible expression and pose diversity across visual identities.
- The method uses end-to-end diffusion to synchronize audio-driven facial dynamics while generating high-quality, temporally consistent animations.The targeted dynamics include lip movements, facial expressions, and head poses.
- A hierarchical audio-driven visual synthesis module improves audio-visual alignment through cross-attention mechanisms and adaptive weighting.
- The architecture combines diffusion-based generation, UNet denoising, temporal alignment, and ReferenceNet guidance to improve animation quality and realism.
- The method provides flexible control over expression and pose diversity for diverse visual identities.