Source-linked AI summary
Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation
Michał Stypułkowski, Konstantinos Vougioukas, Sen He, Maciej Zięba, Stavros Petridis, Maja Pantic
TL;DR
Talking-face generation has struggled to produce natural motion and expressions without additional visual guidance. Diffused Heads uses an autoregressive, frame-based diffusion model conditioned on one identity frame and speech, achieving state-of-the-art results on two datasets while remaining limited in video length and generation speed.
Problem
Existing talking-face methods struggle to create natural expressions and movements while requiring additional supervision or visual guidance.
Method
Diffused Heads autoregressively denoises one frame at a time using one identity frame, speech embeddings, motion frames, and a landmark-based lip sync loss.
Results
State-of-the-art results were achieved on two datasets, with a Turing test involving 140 participants finding outputs indistinguishable from ground-truth videos.
Takeaways & Limitations
The method generates realistic, expressive talking-head videos while preserving identity, background, smooth motion, and plausible lip synchronization from minimal input.
Takeaways & Limitations
Generated quality declines for sequences longer than 8-9 seconds, and diffusion generation is not currently suitable for real-time applications.
Abstract
from arXiv · showhide
Talking face generation has historically struggled to produce head movements and natural facial expressions without guidance from additional reference videos. Recent developments in diffusion-based generative models allow for more realistic and stable data synthesis and their performance on image and video generation has surpassed that of other generative models. In this work, we present an autoregressive diffusion model that requires only one identity image and audio sequence to generate a video of a realistic talking human head. Our solution is capable of hallucinating head movements, facial expressions, such as blinks, and preserving a given background. We evaluate our model on two different datasets, achieving state-of-the-art results on both of them.
1. Introduction
Talking-face synthesis seeks realistic expressions and head movements from speech, but existing methods often require additional guidance and struggle with natural motion. Diffused Heads addresses these issues with a diffusion-based approach using identity, motion, and audio information.
- Existing talking-face methods struggle to maintain natural expressions and movements while requiring additional supervision during generation.
- GAN-based facial animation can require extensive tuning, additional guidance, and remains vulnerable to mode collapse.
- Diffused Heads is a frame-based diffusion model that generates realistic videos from one identity frame and a speech recording.
- The method combines motion frames, pretrained audio embeddings, and a modified loss to support expressive motion, speech consistency, and lip synchronization.
2. Related work
Speech-driven video synthesis progressed from probabilistic sequence models to deep and generative approaches. Diffused Heads extends this line with diffusion models that use one-shot identity input and hallucinate head motion without an actor’s visual guidance.
- Early speech-driven synthesis methods used HMMs and coupled HMMs to model video, speech, and lip-parameter dynamics.
- Deep learning approaches learned mappings from speech to mouth movements, 3D face meshes, or transferred motion.
- GAN-based systems generated talking-head videos from a still image and speech, often using discriminators and intermediate modeling choices.
- Diffusion models have increasingly outperformed GANs in image synthesis and have also been applied to video generation.
- Diffused Heads is presented as the first diffusion-based speech-driven video synthesis method and a one-shot approach that hallucinates diverse head motions without visual driving input.
3. Diffusion models
Diffusion models learn to reverse a fixed noise-adding process, using variational objectives to train denoising predictions. In Diffused Heads, this framework is adapted with landmark-based lip synchronization loss for talking-face generation.
- The forward diffusion process gradually increases noise according to a fixed schedule until the sample approaches an isotropic Gaussian distribution.
- The reverse process learns to denoise Gaussian noise back toward data, conditioning transitions on the original sample to make them tractable.
- The model predicts the Gaussian noise applied during forward diffusion, while an additional output represents variance for the reverse process.
- Landmark-based lip sync loss adds a mouth-region noise-prediction objective alongside the standard L2 noise loss.
- The variational lower bound decomposes into L0, intermediate divergence terms, and LT, with image-specific simplifications for the endpoint terms.
4. Method
Diffused Heads generates talking-head videos autoregressively, conditioning each frame on a fixed identity image, speech embeddings, and motion context. Training adds a mouth-focused lip-sync objective, while sampling uses diffusion denoising with accelerated timestep respacing.
- 4.1. Training: Each frame is generated from a fixed identity frame and speech recording encoded by a pretrained audio encoder.The identity frame remains fixed throughout generation, while audio is divided into frame-aligned chunks and converted into embeddings.
- 4.1. Training: Randomly selecting the identity frame during training exposes the model to more input-frame variation and improves generation robustness.
- 4.2. Motion frames: Motion frames concatenate preceding generated frames to provide temporal information; ablation results identify two motion frames as the best value.During sampling, unavailable preceding frames are handled by padding, and motion-frame inputs are updated with synthesized frames.
- 4.3. Speech conditioning: Audio conditioning shifts and scales UNet hidden states using embeddings that combine past, current, and future audio segments.The authors report this conditioning performs better than alternative scale-only and multihead-attention schemes.
- 4.4. Lip sync loss: A lip-sync loss crops mouth regions using facial landmarks and minimizes noise-prediction error there, directing attention toward synchronization with audio.Sequence-based perceptual losses are not used because the model operates on individual frames and recovering the initial frame from predicted noise is inaccurate or inefficient.
- 4.5. Sampling: Sampling requires only an identity frame and audio embeddings, denoises frames sequentially, and uses timestep respacing to reduce sampling time by a factor of 5.The sequential process replaces the latest motion frame with each synthesized frame after denoising.
- 4.5. Sampling: Sudden head movements can fail because frame-by-frame errors accumulate and generated motion frames may be distorted relative to training data.Converting motion frames to grayscale encourages identity information to come from the identity frame and works well on more complex datasets with many participants.
5. Experiments
Experiments show that Diffused Heads produces realistic, smooth, expressive talking-head videos and generalizes across identities, audio sources, and even synthesized faces, while remaining limited in long-sequence quality and generation speed.
- Qualitative results: Diffused Heads generates realistic videos with natural expressions, eye blinks, smooth motion, preserved identity, few artifacts, and accurate rendering of hair and glasses.The model also performs well for challenging side-view faces.
- Quantitative results: The evaluation compares Diffused Heads with SDA, Wav2Lip, MakeItTalk, PC-AVS, and EAMM on CREMA and LRW using FID, FVD, motion, and synchronization measures.The same test audio is used across models, while driving videos for PC-AVS and EAMM are randomly selected from test sets.
- Quantitative results: Diffused Heads achieves state-of-the-art FVD and best or near-best scores for FID, blinking, smoothness, and AV Offset.Its WER and AV Confidence are worse because lip-sync training lacks expert supervision, although AV Confidence remains comparable to strong methods.
- Quantitative results: A 140-participant Turing test found Diffused Heads more authentic than PC-AVS and even higher-scoring than real LRW videos.The test used 10 videos from each method and 10 real videos.
- Ablation study: Using no motion frames produces almost random facial expressions, while two motion frames concentrate optical flow around the mouth; the model fails with three motion frames.The zero-motion condition has uniformly high optical flow, indicating random movements between consecutive frames.
- Ablation study: A lip-sync loss weight of 0.2 gives very realistic results and the best WER, while values above 0.5 degrade quality.Grayscale motion frames improve quality on LRW, two motion-audio embeddings work best, but RGB helps on less diverse CREMA.
- Generalization: The model performs well when identity frames or audio recordings come from outside the training distribution, including a DALL-E 2 avatar image and custom speech.Generated lip movements and facial expressions remain natural in these tests.
- Limitations: Autoregressive generation without pose or visual guidance fails to preserve initial quality beyond 8–9 seconds and is not currently suitable for real-time applications.Diffusion models also require longer generation times than other generative models.
6. Conclusions
Diffused Heads is a frame-based talking-face generation method that synthesizes realistic speech-driven videos from one identity frame and an audio sequence. It achieves state-of-the-art results on two datasets and produces outputs judged indistinguishable from ground-truth videos by 140 participants.
- The method synthesizes talking-face videos using only one identity frame and an audio sequence containing speech.
- It was evaluated on two datasets with different levels of complexity, achieving state-of-the-art results on both.
- A Turing test with 140 participants found the generated results indistinguishable from ground-truth videos.
A. Temporal metrics
The paper introduces frame-wise Optical Flow Magnitude and Frame-wise Mean Square Error to assess generated-video smoothness. These population-based metrics should match ground-truth values rather than simply approach zero, because repeating a single frame would score zero on both.
- The paper introduces frame-wise Optical Flow Magnitude (OFM) and Frame-wise Mean Square Error (F-MSE) to evaluate video smoothness.
- OFM and F-MSE are defined over a sequence of frames, with OFM using optical-flow magnitude between consecutive frames.
- Reported metric values are averaged over all videos in the test set, with multichannel images receiving additional channel averaging.
- The metrics are population-based and target values close to those computed on corresponding ground-truth sequences.
- A sequence containing one repeating frame scores zero on both metrics, making zero values undesirable as a standalone smoothness objective.