Source-linked AI summary
DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Models
Zhiyao Sun, Tian Lv, Sheng Ye, Matthieu Lin, Jenny Sheng, Yu-Hui Wen, Minjing Yu, Yong-Jin Liu
TL;DR
Speech-driven facial animation must model diverse facial motions and speaking styles from speech, while deterministic mappings and one-hot style labels limit that capability. DiffPoseTalk combines diffusion generation with speech alignment, reference-video style encoding, classifier-free guidance, and head-pose generation. The authors report better results than comparison methods across quantitative metrics and in qualitative evaluation.
Problem
Speech-to-facial-motion generation involves a many-to-many mapping, while deterministic methods can over-smooth motion and one-hot style labels limit adaptation to new styles.
Source: §1 · p. 1Method
DiffPoseTalk uses diffusion to generate facial motion and head pose from speech, with a style encoder extracting style from reference video for classifier-free guidance.
Source: §1 · p. 2Results
DiffPoseTalk outperforms the compared methods across all reported metrics, with the best lip synchronization and head-pose beat alignment.
Source: §4.3 · p. 6Takeaways & Limitations
The approach supports diverse, stylistic speech-driven facial animation that includes head poses and can be guided by reference-video style.
Source: §1 · p. 2Inference has relatively high computational cost because denoising is sequential, and very high noise may yield vague or overly smoothed lip motions.
Source: §5 · p. 8
Abstract
from arXiv · showhide
The generation of stylistic 3D facial animations driven by speech presents a significant challenge as it requires learning a many-to-many mapping between speech, style, and the corresponding natural facial motion. However, existing methods either employ a deterministic model for speech-to-motion mapping or encode the style using a one-hot encoding scheme. Notably, the one-hot encoding approach fails to capture the complexity of the style and thus limits generalization ability. In this paper, we propose DiffPoseTalk, a generative framework based on the diffusion model combined with a style encoder that extracts style embeddings from short reference videos. During inference, we employ classifier-free guidance to guide the generation process based on the speech and style. In particular, our style includes the generation of head poses, thereby enhancing user perception. Additionally, we address the shortage of scanned 3D talking face data by training our model on reconstructed 3DMM parameters from a high-quality, in-the-wild audio-visual dataset. Extensive experiments and user study demonstrate that our approach outperforms state-of-the-art methods. The code and dataset are at https://diffposetalk.github.io .
1 INTRODUCTION
Speech-driven facial animation must handle a many-to-many speech–motion mapping and diverse speaking styles, challenges that deterministic methods and one-hot style labels do not adequately address. DiffPoseTalk combines diffusion-based motion generation with speech alignment, reference-video style control, head-pose generation, and a varied audio-visual dataset.
1 INTRODUCTION
- Deterministic models can suffer regression to the mean and over-smoothed motion, while one-hot style labels limit adaptability to new speaking styles.Facial animation also requires stricter speech-motion timing, phoneme-level speech features, and style representations from arbitrary clips.Source: §1 · pp. 1–2
- DiffPoseTalk uses attention to align facial motion with speech and predicts clean expression signals, alongside head pose, for speech-driven animation.It also uses HuBERT speech features and a corresponding head-pose loss.Source: §1 · p. 2
- A style encoder extracts personalized style from reference video clips, and classifier-free guidance uses that style to control generation.The paper frames this as example-based style control rather than one-hot encoding.Source: §1 · p. 2
- The authors build an audio-visual dataset spanning diverse identities, speaking styles, and head poses to address scarce co-speech 3D facial-animation data.The introduction specifies scarcity of motion-captured co-speech 3D facial-animation data.Source: §1 · p. 2
2 RELATED WORK
Prior speech-driven animation work includes procedural systems and learning-based mappings, while diffusion models offer a way to represent diverse motion distributions. The paper positions DiffPoseTalk as combining diffusion with speech-motion alignment, style control, and head-pose generation.
2.1 Speech-Driven 3D Facial Animation
- Procedural systems map phonemes to visemes with explicit control, but require intricate tuning and do not capture the diversity of real-world speaking styles.Examples include dominance functions and co-articulation models.Source: §2.1 · p. 2
- Learning-based methods map acoustic or pretrained speech features to 3DMM parameters or meshes, but deterministic regression can produce over-smoothed, average motion.The paper reports that performance declines when such methods are applied to its larger dataset, compared with smaller datasets such as VOCA and BIWI.Source: §2.1 · p. 2
- One-hot identity labels limit adaptation to new speakers and fine-grained styles, whereas example-based methods can imitate unseen styles; DiffPoseTalk uses contrastive style features without fine-tuning.Source: §2.1 · p. 2
2.2 Diffusion Probabilistic Models
- Diffusion models add noise and learn to reverse that process, enabling diverse samples and modeling complex many-to-many relationships.Classifier and classifier-free guidance are established approaches for conditional generation across several generative tasks.Source: §2.2 · p. 2
- The denoising network takes speech features, prior clean motion, style, and timestep as inputs and predicts clean motion; a transformer encoder extracts style from motion sequences.Windowing accommodates inputs of arbitrary length.Source: Figure 2 · p. 3
- FaceDiffuser models audio-to-face motion but lacks head-pose control and novel-style conditions; DiffPoseTalk jointly generates facial motion and head poses for arbitrary novel styles.Source: §2.2 · p. 3
3 METHOD
DiffPoseTalk generates speech-driven 3D facial motion from speech, template-face shape, and a style vector, using diffusion with windowed conditioning. Its method combines a transformer denoiser, geometric losses, a video-derived style encoder, and classifier-free guidance.
3.1 Problem Formulation
- The model maps aligned speech features, a template-face shape, and a style vector to a sequence of 3DMM expression and pose parameters.It uses HuBERT speech features and FLAME parameters for the face representation.Source: §3.1 · p. 3
3.2 Facial Animation Diffusion Model
- The diffusion denoiser predicts clean motion samples, using speech-motion alignment and geometric losses to constrain facial and head motion.The alignment mask lets each motion position attend to the corresponding speech feature; losses include vertex, velocity, smoothness, and head-motion terms.Source: §3.2 · p. 4
3.3 Style-Controllable Diffusion Model
- Shape and speaking-style features act as control conditions shared across generation windows.The style vector is extracted from a brief reference video by the speaking style encoder.Source: §3.3 · p. 4, §3.3.1 · p. 4
3.3.1 Speaking Style Encoder.
- A speaking-style encoder maps motion sequences to style embeddings, trained contrastively under the assumption that nearby clips from the same person have similar short-term styles.The encoder uses a transformer and average pooling; contrastive training pairs two halves of a sample as positives and treats other examples as negatives.Source: §3.3.1 · pp. 4–5
3.3.2 Training Strategy.
- Training divides each sample into two windows, using one for initial-window training and the other with preceding motion as context; random truncation improves robustness to speech-length variation.The two windows receive style features derived from their respective motion segments.Source: §3.3.2 · p. 5
3.3.3 Sampling with Incremental Classifier-Free Guidance.
- At sampling, incremental classifier-free guidance steers iterative generation using separate audio and style guidance scales.The denoiser estimates a clean sample and noise is reintroduced at each step.Source: §3.3.3 · p. 5
4 EXPERIMENTS
Experiments report that DiffPoseTalk outperforms the compared methods across quantitative metrics and user ratings of synchronization, style similarity, and naturalness, with qualitative comparisons and ablations examining its behavior.
4.1 Datasets
- TFHP contains 1,052 videos of 588 subjects totaling 26.5 hours, with reconstructed FLAME parameters and speaker-disjoint training, validation, and test splits.The splits contain 460, 64, and 64 speakers, respectively; the dataset includes diverse content such as lectures, courses, interviews, and news.Source: §4.1 · p. 5
4.2 Experiment Setup
- The speaking-style encoder is a four-layer transformer, and the denoising network an eight-layer decoder using a cosine schedule of 500 diffusion steps; comparisons cover 3D facial-animation and 2D talking-face methods.Overall training takes about 12 hours on an Nvidia 3090 GPU.Source: §4.2 · p. 5
4.3 Quantitative Evaluation
- DiffPoseTalk outperforms all compared methods across metrics, with the best lip synchronization and head-pose beat alignment; other methods generally struggle to produce varied outputs from identical inputs.The authors report that most baselines are deterministic; SadTalker samples head poses, while FaceDiffuser produces nearly identical results for the same input.Source: Table 1 · p. 6, §4.3 · p. 6
4.4 Qualitative Evaluation
- Qualitative comparisons report stronger handling of bilabial consonants and rounded vowels, closer speaking-style resemblance, spontaneously generated eye blinks, and head motions aligned with speech stress and rhythm.The authors identify upper-face expressions and mouth-opening size as aspects of speaking style that resemble the ground truth.Source: §4.4 · p. 6
4.5 User Study
- Twenty-six participants rated lip synchronization, speaking-style similarity, and naturalness; the study reports that DiffPoseTalk significantly outperforms existing works on these measures.The comparison separates methods with and without head motion.Source: §4.5 · p. 6, Table 2 · p. 6
4.6 Ablation Study
- Ablations associate removing the style encoder with declines across metrics, removing geometric losses with imprecise animation, and removing the alignment mask with serious synchronization problems.Removing classifier-free guidance improves LVE but slightly worsens FDD and MOD, a mixed metric effect the authors attribute speculatively to its quality–diversity trade-off.Source: §4.6 · p. 7
5 DISCUSSIONS
The discussion motivates FLAME 3DMM parameters as a compact, broadly sourced face representation and identifies inference, lip-motion, inner-mouth, and data-coverage limitations alongside misuse risks.
Choice of Face Representation
- FLAME 3DMM parameters are lower-dimensional than mesh vertices, which the authors say speeds diffusion computation and eases learning; reconstructed parameters from 2D audiovisual data also offer broader identity, phoneme, and style coverage than scanned meshes.The authors additionally cite blendshapes as a prior that simplifies learning and improves generalization.Source: §5 · p. 7
Qualitative Comparison
- Fig. 4 presents qualitative comparisons with head-pose prediction; dashed lines separate identities, and “>” marks speech stress.Source: Figure 4 · p. 8
Choice of Face Representation
- The representation supports downstream use in blendshape-based avatars and facial-expression editing.The cited examples are Qian et al. [2023] and Geng et al. [2019]; Sun et al. [2023].Source: §5 · p. 8
Limitations
- Inference remains relatively costly because denoising is sequential; very high noise can yield vague or overly smoothed lip motion, and the method omits teeth and tongue.The authors also identify collecting real-world 3D talking data covering more identities and styles as a future direction.Source: §5 · p. 8
Ethical Considerations
- Because realistic talking-head sequences could be misused for deepfakes or deliberate manipulation, the authors require generated sequences to be marked as synthetic.They also state that they will make the code available to the deepfake-detection community.Source: §5 · p. 8
6 CONCLUSION
DiffPoseTalk uses diffusion to generate diverse, stylistic 3D facial animations and head poses from speech, addressing the many-to-many relationships among speech, style, and motion.
6 CONCLUSION
- DiffPoseTalk generates diverse, stylistic 3D facial animations and head poses from speech.The method uses diffusion and a speaking style encoder to capture relationships among speech, style, and motion.Source: §6 · p. 8
- The method uses diffusion to address the many-to-many mapping between speech and expression animation.Source: §6 · p. 8
- A speaking style encoder helps the approach capture relationships among speech, style, and motion.Source: §6 · p. 8