FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusion

1 INTRODUCTION

• Existing workflows rely on time-consuming expert keyframing or specialised capture systems, while phoneme-based methods require manually specified co-articulation rules.Deep-learning approaches learn speech-to-animation mappings automatically.Source: §1 · p. 2
• Prior end-to-end deep-learning methods produce the same animation for a given speech input, although non-verbal facial cues are non-deterministic.The paper identifies non-verbal cues throughout the face as non-deterministic in nature.Source: §1 · p. 2
• FaceDiffuser is the first diffusion-based method for speech-driven 3D facial animation, generating animations for both vertex-based and rigged characters.It combines a HuBERT audio encoder with a specialised diffusion model.Source: §1 · p. 2
• The authors report better or comparable results to state-of-the-art methods across their objective and subjective analyses.Their stated contributions include better objective metrics on multiple temporal, high-dimensional 3D vertex datasets.Source: §1 · p. 2

2 RELATED WORK

• Speech-driven facial animation work includes 2D talking heads and 3D synthesis from vertex or blendshape data, while 3D face reconstruction from video is not speech-driven.The paper focuses on 3D speech-driven synthesis and reviews that literature alongside diffusion-based motion synthesis.Source: §2 · p. 2

2.1 Speech-driven 3D Facial Animation

• Procedural phoneme-based methods provide animation control and easy artist-friendly pipeline integration but require explicit co-articulation rules; separate machine-learning approaches learn speech-animation mappings from data, while recent synthesis uses deep-learning models.Most deep-learning approaches use 3D vertex-based datasets and are incompatible with traditional rigged-character pipelines, except a few examples.Source: §2.1 · pp. 2–3
• Existing non-deterministic facial-animation methods use VQ-VAE, but the reviewed methods do not use diffusion to generate varied 3D facial animations from speech.The cited examples include Learning2Listen, CodeTalker [54], and TalkShow.Source: §2.1 · p. 3

2.2 Diffusion for Motion Synthesis

• Diffusion models learn to reverse a gradual noising process and have been used for image generation and motion synthesis.Related motion work includes text-, music-, audio-, and speech-conditioned body animation, while DAE-Talker [12] applies diffusion to talking-head animation.Source: §2.2 · p. 3

3 PROBLEM FORMULATION

• Each animation frame is represented either by vertex positions of size V x 3 or by a vector of C rig-control or blendshape values.N denotes the number of visual frames sampled at the dataset-specific frame rate.Source: §3 · p. 3
• The formulation predicts an animation sequence from audio A, with style S and a same-shaped noise sequence drawn from N (0, 1) also guiding predictions.The style is a one-hot subject vector for vertex-based datasets in these experiments.Source: §3 · p. 3
• Vertex-based models predict displacements relative to neutral face templates, whereas blendshape or facial-control models do not require that step.For rigged faces, the neutral face is not a variable in this setting.Source: §3 · p. 3
• FaceDiffuser maps audio A, the animation sequence after t diffusion steps, and timestep t to a predicted animation sequence.The training diagram shows audio and noised-animation embeddings feeding the decoder, whose output is compared with ground truth.Source: §3 · p. 3, Figure 2 · p. 4

4.1 Training

• The general model supports vertex and blendshape datasets with slight hyperparameter changes; V-FaceDiffuser and B-FaceDiffuser denote the respective configurations.The vertex configuration adds a Noise Encoder that projects high-dimensional vertex data into a low-dimensional latent representation.Source: §4.1 · p. 4
• The model uses pretrained HuBERT audio embeddings and a GRU-based facial decoder followed by a fully connected layer to predict the output sequence.A style embedding can be added through an element-wise product with the decoder hidden states.Source: §4.1 · p. 4
• Training predicts animation data rather than the added noise, which the authors say permits acceptable predictions from early denoising steps and faster sampling.Following MDM [49] and EDGE, the model is conditioned on input audio; completing inference yields the best results.Source: §4.1 · p. 4

4.2 Inference

• Inference iterates backward from timestep T to 1, starting with noise sampled from N (0, 1) and gradually improving predictions.The predicted motion is diffused again and supplied to the next iteration.Source: §4.2 · p. 5, Figure 3 · p. 5

5 EXPERIMENTAL SETUP

• BIWI uses emotional sequences from 14 subjects, with 192 training sentences, 24 validation sentences, and separate seen- and unseen-subject test sets.Source: §5 · p. 5
• Objective analyses use BIWI and Multiface, perceptual studies use BIWI and VOCASET, and qualitative analyses use all datasets.Source: §5 · p. 5
• V-FaceDiffuser projects vertex inputs to 512 dimensions for BIWI and 256 for VOCASET and Multiface, while B-FaceDiffuser needs no projection.Source: Table 1 · p. 5
• BEAT and UUDaMM extend training to lower-dimensional facial controls; prior vertex-focused methods are not directly applicable, so comparison uses a non-diffusion baseline.Source: §5 · p. 5
• Training ran on a shared Linux cluster with an AMD EPYC 7313 CPU, Nvidia A16 GPU, and 1TB RAM.Source: §5 · pp. 5–6
• FaceDiffuser achieves the best results on all four BIWI objective metrics and all Multiface metrics except FDD.Source: Table 2 · p. 6

6.1 Quantitative Evaluation

• FaceDiffuser leads all objective metrics on BIWI and all but FDD on Multiface; on blendshape/controller datasets it has higher frame-level errors but better FDD.The blendshape/controller comparison is against a baseline, not state-of-the-art methods.Source: §6.1 · p. 6, Table 3 · p. 6

6.2 Qualitative Evaluation

• Lip shapes stay close to references while upper-face motions vary across outputs and subject-conditioned BIWI inferences; the authors also report generalization across speakers and audio conditions.Reported conditions include unseen speakers, noisy or overlapping-speaker audio, different languages, and text-to-speech audio.Source: Figure 4 · p. 7, Figure 5 · p. 7, §6.2 · p. 7

6.3 Perception Study

• FaceDiffuser beat VOCA on all aspects for BIWI and VOCASET, but trailed FaceFormer and CodeTalker on BIWI and CodeTalker on VOCASET; ground truth led all.On UUDaMM, users clearly preferred the diffusion model over the baseline without diffusion.Source: §6.3 · p. 8, Table 4 · p. 8

6.4 Ablation Study

• Removing diffusion slightly worsened objective metrics; models with diffusion also produced more facial motion resembling the ground-truth mean motion.The ablation used BIWI and UUDaMM.Source: §6.4 · p. 8
• HuBERT improved over Wav2Vec 2.0 as the audio encoder for speech-driven 3D animation synthesis.Source: §6.4 · p. 8
• The GRU decoder achieved lower error values than the tested RNN and transformer decoders on BIWI and UUDaMM.Source: §6.4 · p. 8

• Objective performance exceeds prior methods, but perceptual evaluation still leaves room for improvement.Source: §7 · p. 9
• Short sequences in datasets [53] limit the model’s ability to capture long-term context and the full range of expressions.Source: §7 · p. 9
• Iterative diffusion sampling makes inference time long and dependent on training timesteps, making the approach unsuitable for real-time applications.Source: §7 · p. 9
• The authors suggest richer repeated-content datasets and emotion information to better assess diversity and non-verbal cues; BIWI offers only two binary-emotion conditions.Source: §7 · p. 9
• Tables 5–6 report BIWI objective-metric ablations of the diffusion process, audio encoders, and facial decoders.Source: Table 5 · p. 9, Table 6 · p. 9

• The model generalizes across high-dimensional temporal vertex data and low-dimensional blendshape data with minimal changes.Source: §8 · p. 9
• Quantitative analysis reports better performance than state-of-the-art methods, while the model produces greater motion diversity across style conditions than competitors.Source: §8 · p. 9
• Animations for rigged characters vary across multiple generations conditioned on the same audio, and ablations support the proposed architecture choices.Source: §8 · p. 9
• The paper notes that face data can be used to generate content that jeopardizes privacy and calls for responsible consideration of privacy and ethics.Source: §8 · p. 10

• The three surveys collected 83 responses: 31 for BIWI, 29 for VOCASET, and 23 for UUDaMM; three more were discarded after failed attention checks.Source: §A.1 · p. 12
• Mean familiarity ratings on a five-point scale were 2.71 for virtual humans, 3.52 for 3D animated movies, and 4.06 for video games.Source: §A.1 · p. 12
• The first two surveys use 12 pairwise comparisons and ask about lip-sync quality, perceived realism, and animation appropriateness.Source: §A.1 · p. 12

• The dataset contains 14 × 40 × 2 paired audio-animation sequences, with each sentence performed once neutrally and once more emotionally.Source: §A.2.1 · p. 12
• Sequences average about five seconds at 25 fps; the front-facing meshes contain 23,370 3D vertices.Source: §A.2.1 · p. 12
• Training uses the emotional subset, with six subjects and 32 spoken sentences per subject, totaling 192 sequences.Source: §A.2.1 · p. 12

A.2.1 BIWI[19].

• BIWI-Test-A uses four held-out sentences from each of six subjects, while BIWI-Test-B uses the last four sentences from eight other subjects.BIWI-Test-A is used for objective metrics; BIWI-Test-B is the primary set for experiments, hyperparameter tuning, and ablation.Source: §A.2.1 · p. 12

A.2.2 VOCASET[8].

• VOCASET uses 8 of 12 subjects for training and 2 each for validation and testing, with 320, 40, and 40 sequences, respectively.The split is used to assess generalizability and conduct the user study.Source: §A.2.2 · p. 13

A.2.3 Multiface[53].

• The public Multiface subset has 13 subjects; its split uses 9 subjects for training and validation, with 4 unseen subjects forming Multiface-Test-B.The dataset includes full-head meshes with 6,172 vertices; Multiface-Test-A and -B each contain 45 sequences.Source: §A.2.3 · p. 13

A.2.4 Utrecht University Dyadic Multimodal Motion Capture Dataset

• UUDaMM captures dyadic conversations across face, audio, gesture, and text; facial performance is retargeted to a rigged character with artist-created blendshapes and controls.The dataset contains 9 hours 41 minutes of conversation, including 6 hours 53 minutes of active speech.Source: §A.2.4 · pp. 13–14

A.2.5 BEAT[31].

• BEAT provides facial animation as blendshape weights; experiments use 16 hours of native-English recordings from four subjects, split into 11,398 ten-second training segments.The full recordings total about 76 hours and include four languages and eight emotions.Source: §A.2.5 · p. 14

A.3 Additional Ablation on Diffusion Steps

• 500 and 750 diffusion steps yield acceptable results, while 100 steps perform worst and increasing steps beyond 750 worsens results.All tested configurations produce correct lip-sync; differences mainly concern animation expressivity.Source: §A.3 · p. 14

A.4 Additional Ablation on Noise Encoder

• The proposed noise encoder performs best among tested configurations; more complex encoders increase facial motion but can worsen ground-truth error metrics.Conv1D approaches outperform the MLP, and max pooling performs slightly better than average pooling.Source: §A.4 · pp. 14–15, Table 10 · p. 14, Figure 9 · p. 14

A.5 Additional Use Cases

• ARKit animation curves can be edited to intensify mouth motion, and animations transfer between characters with matching or mappable blendshapes without additional retargeting.The authors caution that limited ARKit expressivity may produce an uncanny-valley effect on photorealistic human characters.Source: §A.5.1 · p. 15, §A.5.2 · p. 15