Source-linked AI summary
DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation
Shuai Shen, Wenliang Zhao, Zibin Meng, Wanhua Li, Zheng Zhu, Jie Zhou, Jiwen Lu
TL;DR
Talking-head synthesis has struggled to achieve both high generation quality and cross-identity generalization. DiffTalk addresses this with an audio-driven, temporally coherent conditional diffusion process augmented by reference images and landmarks, and reports high-fidelity videos for novel identities without fine-tuning.
Problem
Existing talking-head methods have difficulty addressing generation quality and model generalization simultaneously, although both are important for real-world deployment.
Method
DiffTalk uses a conditional latent diffusion model driven by audio, reference images, and facial landmarks for temporally coherent, identity-aware synthesis.
Results
DiffTalk synthesizes high-fidelity talking videos for novel identities without further fine-tuning and is reported to outperform 2D methods in image quality while surpassing 3D methods in generalization ability.
Takeaways & Limitations
The method provides a generalized talking-head synthesis approach that combines high visual quality with identity transfer across unseen individuals.
Takeaways & Limitations
DiffTalk requires more time to synthesize each frame than most GAN-based approaches because generation uses iterative denoising.
Abstract
from arXiv · showhide
Talking head synthesis is a promising approach for the video production industry. Recently, a lot of effort has been devoted in this research area to improve the generation quality or enhance the model generalization. However, there are few works able to address both issues simultaneously, which is essential for practical applications. To this end, in this paper, we turn attention to the emerging powerful Latent Diffusion Models, and model the Talking head generation as an audio-driven temporally coherent denoising process (DiffTalk). More specifically, instead of employing audio signals as the single driving factor, we investigate the control mechanism of the talking face, and incorporate reference face images and landmarks as conditions for personality-aware generalized synthesis. In this way, the proposed DiffTalk is capable of producing high-quality talking head videos in synchronization with the source audio, and more importantly, it can be naturally generalized across different identities without any further fine-tuning. Additionally, our DiffTalk can be gracefully tailored for higher-resolution synthesis with negligible extra computational cost. Extensive experiments show that the proposed DiffTalk efficiently synthesizes high-fidelity audio-driven talking head videos for generalized novel identities. For more video results, please refer to \url{https://sstzal.github.io/DiffTalk/}.
1. Introduction
DiffTalk targets the gap between generation quality and model generalization in talking-head synthesis by using conditional latent diffusion with audio, reference images, and landmarks. Experiments report high-fidelity videos for novel identities without further fine-tuning, with advantages across 2D- and 3D-based comparisons.
- Research gap: Existing methods often trade off image quality and generalization: GAN-based 2D methods face unstable training and limited resolution, while 3D methods rely on identity-specific training.These limitations make simultaneously improving both factors important for practical deployment.
- Approach: DiffTalk models talking-head synthesis as an audio-driven, temporally coherent denoising process using a conditional latent diffusion model.The design addresses GAN training instability while retaining audio-driven facial motion generation.
- Approach: Reference images and landmarks provide additional conditions for personalized facial modeling and identity-aware synthesis.These controls guide facial identity and head pose beyond the audio signal alone.
- Results: DiffTalk generalizes to novel identities without further fine-tuning while producing high-fidelity talking videos.The introduction reports this outcome from extensive experiments.
- Results: Compared with prior approaches, DiffTalk is reported to improve generated image quality over 2D methods and generalization ability over 3D methods.The comparison is presented as a contribution supported by the reported experiments.
2. Related Work
Prior work spans GAN-based 2D talking-head synthesis and 3D or NeRF-based approaches, while latent diffusion models offer a more efficient generative framework. DiffTalk applies latent diffusion to audio-driven, temporally coherent talking-head generation with additional visual controls.
- 2D-based methods: 2D talking-head methods commonly use GANs to learn audio-to-lip mappings and generally perform competently on model generalization.Related work also includes speaker-aware audio encoding and dedicated lip-synchronization components.
- 3D-based methods: 3D-based methods target higher-quality talking videos but commonly depend on identity-specific training, limiting generalization across people.This identity dependence is a central distinction from the generalized setting pursued by DiffTalk.
- Latent diffusion models: Latent Diffusion Models move diffusion training and inference from pixel space into a compressed lower-dimensional latent space for more efficient computing.The related-work passage places LDMs within broader image and motion generation applications.
- DiffTalk: DiffTalk draws on latent diffusion to model talking-head synthesis as an audio-driven, temporally coherent denoising process.The method extends this framework with visual conditioning for talking-head generation.
3. Methodology
DiffTalk models talking-head synthesis as an audio-driven, temporally coherent latent diffusion process, using audio, reference images, landmarks, and progressive inference to control facial motion, identity, and resolution.
- Latent diffusion foundation: Latent Diffusion Models transfer talking-head learning into a lower-dimensional latent space, where a time-conditional UNet denoises the representation before decoding it into an image.The frozen image encoder and decoder reduce training cost while preserving the original image-space reconstruction process.
- Audio conditioning: Smooth audio features condition the diffusion process to model temporally coherent facial dynamics and bridge audio signals with visual information.The audio encoder uses overlapping 20 ms windows, DeepSpeech features, and learnable temporal filtering over adjacent frames.
- Identity-preserving conditioning: Reference face images and facial landmarks provide additional controls for personalized synthesis and generalization to identities unseen during training.The reference mechanism targets appearance, background, and pose while preserving the source identity.
- Conditional diffusion model: The conditional denoising model combines audio features, reference latents, masked references, and landmarks as its conditioning set.The visual references are concatenated with the noisy latent map and supplied to the network through the conditioning pathway.
- Higher-resolution synthesis: Higher-resolution synthesis reuses a 64 × 64 × 3 latent space by changing the image encoder and decoder downsampling factor from f = 4 to f = 8.The encoder is frozen and transferred to the higher-resolution setting, reducing additional resource pressure.
- Progressive inference: During inference, DDIM sampling and progressive references use the previous synthesized frame to maintain detail consistency and smoother transitions.The progressive strategy is applied only during inference, while frame interpolation further alleviates video jitter.
4. Experiments
Experiments evaluate DiffTalk through ablations, comparisons with 2D and 3D methods, and higher-resolution synthesis. The results support smoother temporal transitions, effective conditioning, strong image quality and synchronization, generalized inference without fine-tuning, and improved quality at 512 × 512 resolution.
- Experimental Settings: The experiments use HDTF data, visual results, image-quality metrics PSNR, SSIM, LPIPS, and the audio-visual synchronization metric SyncNet.The training set contains 100 selected videos from HDTF, while the remaining data form the test set.
- Ablation Study: Audio temporal filtering improves SyncNet performance and produces smoother adjacent-frame transitions by reducing mouth-region jitters.Without filtering, heatmaps show high differences around the mouth; smooth audio yields softer frame differences.
- Ablation Study: The final A+L+M+R conditioning design combines audio, landmarks, masked ground-truth images, and random reference images to improve identity consistency, alignment, and mouth appearance.Audio alone fails to generalize; landmarks address alignment, while the dual reference design supplies identity and lip-appearance information.
- Ablation Study: Progressive inference further boosts SyncNet and LPIPS by using the previous generated frame as the next reference, improving temporal correlation and talking-style coherence.The comparison is between random-reference inference and reference propagation from the generated preceding frame.
- Expand to Higher Resolution: At 512 × 512 resolution, the generated synthesis quality significantly outperforms the 256 × 256 model while both use a compressed latent space to reduce resource pressure.The higher-resolution model uses f = 8 and the lower-resolution model uses f = 4.
5. Conclusion and Discussion
DiffTalk uses a conditional diffusion model to synthesize generalized, high-fidelity talking heads while incorporating audio and reference images to preserve synchronized motion and personalized appearance. It generalizes across identities without fine-tuning and can scale to higher resolutions with negligible extra computational cost, but inference is slower and has specific synchronization and masking limitations.
- Method and generalization: DiffTalk combines audio and reference images as driving factors to generate personalized talking-head videos that generalize across identities without fine-tuning.Audio drives lip motion, while reference images model personalized appearance.
- Resolution: The method can be tailored for higher-resolution synthesis with negligible extra computational cost.
- Limitations: Iterative denoising requires more time per synthesized frame than most GAN-based approaches.The paper identifies this as a common limitation of Latent Diffusion Model-based works.
- Limitations: Cross-identity audio produces slightly inferior audio-lip synchronization than self-driven audio.
- Limitations: Inference is sensitive to the mask shape in zT: the mask must fully cover the mouth region without revealing lip-shape information.