DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencoder

1 INTRODUCTION

• Handcrafted landmarks and 3DMM coefficients can insufficiently describe facial movement, while their pretrained extractors may introduce errors and cap generation performance.These representations are designed using human knowledge.Source: §1 · p. 2
• DAE-Talker trains a diffusion autoencoder on talking-face frames, then trains a Conformer-based speech2latent model to predict its data-driven latents.At inference, the decoder reconstructs full frames from predicted latents, supporting speech-aligned natural head movement rather than a predetermined template pose.Source: §1 · p. 2
• A shared Gaussian noise start and a continuous predicted latent sequence let the DDIM decoder generate continuous frames without directly modeling consecutive-frame joint distributions.The DAE is trained on individual frames; the sequence-level speech2latent model supplies continuity across predicted latents.Source: §1 · p. 2
• DAE-Talker outperforms existing popular methods in lip-sync, video fidelity, and pose naturalness.The paper also reports ablations and demonstrates pose controllability.Source: §1 · p. 2

2.1 Diffusion model

• Diffusion advances include DDPM and score-based models, while DDIM uses non-Markovian processes for accelerated sampling and has been used in autoencoders for face representations.Prior diffusion autoencoders have also been used to disentangle face identity and motion.Source: §2.1 · p. 2
• Video diffusion methods model consecutive-frame joint distributions, but this can require large models and slow convergence; DAE-Talker uses an independent-frame DDIM decoder and sequence-level speech2latent for continuity.Other cited video approaches use latent-space diffusion, arbitrary-length generation, or cascaded diffusion and super-resolution models.Source: §2.1 · p. 2

2.2 Speech-driven talking face generation

• Speech-driven face methods span landmark prediction, GANs, pretrained face-video latents, 3D face models, and NeRF; DAE-Talker contrasts with autoregressive Diffused Head by generating frames in parallel with continuity.The cited NeRF discussion notes possible blurry torsos, while Diffused Head's autoregressive frame correlation limits inference speed.Source: §2.2 · pp. 2–3

3 DAE-TALKER

• DAE-Talker trains a diffusion autoencoder on talking-face frames and uses the extracted latents as targets for a Conformer-based speech2latent model.At inference, the model predicts latents from speech and the DAE image decoder generates frames from them.Source: §3 · p. 3

3.1 Latent extraction from diffusion autoencoder

• DDIM decoding is deterministic, allowing image reconstruction to be controlled by the initial noise and latent representation.The decoder reconstructs the image from the DAE encoder's latent vector.Source: §3.1 · pp. 3–4

3.2 Speech2latent: Latent prediction from speech

• Speech2latent predicts monotonically aligned latent sequences from speech using a Conformer architecture.The model uses pretrained wav2vec 2.0 acoustic features.Source: §3.2.2 · p. 4, §3.2 · p. 4, §3.2.1 · p. 4

3.2.2 Local and global context-aware architecture.

• The speech encoder and latent decoder use Conformer blocks that combine self-attention and convolution to capture local and global context.A first-layer stride downsamples the higher-frame-rate speech to match the video frame rate.Source: §3.2.2 · p. 4

3.2.3 Pose modelling.

• Conditioning latent prediction on extracted head poses addresses speech-to-pose ambiguity and allows predicted or user-specified pose control.During training, a pose predictor estimates head poses from speech.Source: §3.2.3 · p. 4

3.2.4 Pseudo-sentence sampling for data augmentation.

• Pseudo-sentence sampling randomly cuts 5–20-second sequences, preserving sequence-level training while increasing the number of samples.This addresses the small number of real sentences available when training data spans only a few minutes.Source: §3.2.4 · p. 4

3.3 Frame-wise conditioned video generation with DDIM image decoder

• A shared Gaussian noise input and continuously changing latent sequence let the frame-wise DDIM decoder generate continuous video frames.The DDIM denoising trajectory is treated as a continuous function of the noise and latent representation.Source: §3.3 · p. 5
• The approach uses a DDIM image decoder rather than a large video diffusion model that explicitly models correlations between consecutive frames.The DAE is trained on individual frames, while the shared noise and continuous latent sequence provide frame continuity.Source: §3.2.4 · p. 4, §3.3 · p. 5

4.1.1 Dataset.

• The 15-minute Obama address is split into 12 minutes for training and 3 minutes for test set A; test set B uses another Obama address, and test set C uses LibriTTS.The video is 256 × 256 at 25 fps, and audio is sampled at 16000 Hz.Source: §4.1.1 · p. 5

4.1.2 Training details.

• The DAE uses a 512-dimensional latent code, 1000 diffusion steps, L1 noise-prediction training, and 100 denoising steps at inference.It is trained for 80k steps on 8 Nvidia V100 GPUs, taking about 3 days.Source: §4.1.2 · p. 5
• Speech2latent uses pretrained wav2vec 2.0 XLSR features and four-block Conformer speech and latent encoders, alongside a convolutional pose predictor.Ground-truth roll, pitch, and yaw are represented as three-dimensional vectors; training uses Adam with a learning rate of 1e-4 and batch size 16.Source: §4.1.2 · p. 6

Baselines

• MakeItTalk predicts facial landmarks, Wav2Lip relies on a lip-sync expert and template-video upper-face motion, and AD-NeRF renders a speech-conditioned neural radiance field.MakeItTalk and Wav2Lip use released checkpoints; AD-NeRF is trained on the 12-minute training set.Source: §4.2 · p. 6

Objective lip-sync evaluation

• DAE-Talker has the best lip-sync performance overall, with the lowest LSE-D and LMD and highest LSE-C scores in most cases.LMD is reported only on test sets A and B because test set C has no ground-truth video.Source: §4.2 · p. 6

Subjective evaluation

• In user studies, DAE-Talker receives the highest video-fidelity scores and outperforms the baselines in lip-sync and head-movement naturalness.Wav2Lip and AD-NeRF poses are more natural than MakeItTalk's but may remain inconsistent with speech; AD-NeRF's torso can appear blurry.Source: §4.2 · p. 7

Shared-noise ablation

• Without shared x_T, generated frames can be discontinuous and faces can jiggle, while lip-sync remains similar to DAE-Talker.This ablation samples different Gaussian noise for each frame.Source: §4.3 · p. 7

Data-augmentation ablation

• Without pseudo-sentence data augmentation, videos sometimes have unnatural head movement and lip-sync performance degrades.The ablated model trains on speech cut into 20-second segments at fixed timestamps.Source: §4.3 · p. 7

Ablation evaluation

• Objective metrics show similar lip-sync for the shared-noise ablation and slightly worse performance without augmentation; preference tests favor DAE-Talker for video fidelity and pose naturalness.The preference test uses 15 raters, assessing video fidelity for experiment 1 and pose naturalness for experiment 2.Source: §4.3 · p. 7

4.4.1 Reconstruction quality from intermediate representation.

• Image reconstruction from diffusion latents outperforms the landmark-based MakeItTalk method on the reported evaluation.The reconstruction study uses Obama test set A and reports PSNR, SSIM, and LPIPS; Table 3 presents the comparison.Source: §4.4.1 · p. 7, Table 3 · p. 7

4.4.2 Alleviating one-to-many mapping with pose modelling.

• With a pose adaptor, latent-prediction mean square error decreases faster and reaches a lower value, supporting pose modelling for the one-to-many speech-to-latent mapping.The comparison plots test-set-A training curves for models with and without the adaptor.Source: §4.4.2 · pp. 7–8, Figure 6 · p. 8

4.4.3 Pose controllability.

• User-specified head poses significantly reduce the distance between desired poses and poses extracted from generated videos, demonstrating pose controllability.The evaluation tests left, right, and straight-ahead poses; Figure 7 shows controlled-pose snapshots.Source: §4.4.3 · p. 8, Table 4 · p. 8, Figure 7 · p. 8

• DAE-Talker outperforms popular methods in lip-sync, video fidelity, and pose naturalness.Source: §5 · p. 8
• A Conformer-based speech2latent model predicts diffusion-autoencoder latents from speech, which the decoder uses to generate full frames and natural head movements aligned with speech content.Unlike a template-video approach, the method explicitly models head pose for controllability.Source: §5 · p. 8
• Ablations support shared noise for video generation and pseudo-sentence sampling for data augmentation; pose modeling alleviates one-to-many latent prediction and provides controllability.The data-driven latent representation also reconstructs better than handcrafted landmarks.Source: §5 · p. 8