PaperlayerView exact revision on arXiv

Source-linked AI summary

SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation

Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, Fei Wang

arXiv:2211.12194v2cs.CV

TL;DR

Audio-driven talking-head generation remains challenging because audio relates differently to facial motions, and existing representations may not separate head pose from expression. SadTalker separately predicts 3DMM expression and pose coefficients to modulate a face renderer, with experiments reporting state-of-the-art motion synchronization and video quality.

  • Problem

    Talking-head generation remains challenging because audio relates differently to facial motions, and landmark- and flow-based methods [39] [40] may not disentangle pose from expression.

    Source: §1 · p. 2

  • Method

    SadTalker learns expression and head-pose 3DMM coefficients separately with ExpNet and PoseVAE, then uses them to modulate its face renderer.

    Source: §3.2 · p. 3, §3.1 · p. 3

  • Results

    Experiments report state-of-the-art performance in motion synchronization and video quality.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The predicted 3D facial coefficients can also be used in other modalities, including personalized 2D visual dubbing and 3D face animation [8] [46].

    Source: §5 · p. 8

    The system models lip motion and eye blinking but not emotion or gaze, so generated videos have fixed emotion and reduced realism.

    Source: §4.5 · p. 8

Abstract

from arXiv · show

Generating talking head videos through a face image and a piece of speech audio still contains many challenges. ie, unnatural head movement, distorted expression, and identity modification. We argue that these issues are mainly because of learning from the coupled 2D motion fields. On the other hand, explicitly using 3D information also suffers problems of stiff expression and incoherent video. We present SadTalker, which generates 3D motion coefficients (head pose, expression) of the 3DMM from audio and implicitly modulates a novel 3D-aware face render for talking head generation. To learn the realistic motion coefficients, we explicitly model the connections between audio and different types of motion coefficients individually. Precisely, we present ExpNet to learn the accurate facial expression from audio by distilling both coefficients and 3D-rendered faces. As for the head pose, we design PoseVAE via a conditional VAE to synthesize head motion in different styles. Finally, the generated 3D motion coefficients are mapped to the unsupervised 3D keypoints space of the proposed face render, and synthesize the final video. We conducted extensive experiments to demonstrate the superiority of our method in terms of motion and video quality.

1. Introduction

SadTalker addresses unnatural talking-head animation by learning audio-driven 3D motion coefficients separately and using them to drive a 3D-aware face renderer. Its introduction highlights the distinct roles of ExpNet, PoseVAE, and the renderer in generating expression, stylized head motion, and video.

1. Introduction

  • Prior talking-face methods produce unnatural videos with issues including restricted poses, blur, identity modification, and distorted faces.The introduction attributes some distortion to representations that do not fully disentangle head motion and expression, despite the 3D facial model's decoupled representation.Source: §1 · p. 2
  • SadTalker uses 3DMM motion coefficients to learn audio-driven expression and head pose individually, aiming to reduce uncertainty between different motion types.ExpNet distills expression coefficients and uses perceptual losses on rendered faces, while PoseVAE models diverse head motion with a conditional VAE.Source: §1 · p. 2
  • A 3D-aware face renderer maps explicit 3DMM coefficients to unsupervised 3D keypoints, which generate warping fields to animate the source image.The renderer is described as semantic-disentangled, and the expression, head-pose, and renderer subnetworks are trained individually but inferred end-to-end.Source: §1 · p. 2
  • The paper reports that SadTalker achieves state-of-the-art performance in motion synchronization and video quality.The introduction also says experiments show advantages on several video and motion metrics.Source: §1 · p. 2

2. Related Work

The related work covers audio-driven single-image talking faces, audio-driven video portraits, and video-driven single-image animation, distinguishing their motion representations, training settings, and limitations.

Audio-driven Single Image Talking Face Generation

  • Early audio-driven single-image methods target lip accuracy [3] [30], while ATVGnet uses landmarks and MakeItTalk separates audio content from speaker information.Later approaches include PC-AVS [51], which separates pose and expression but produces only low-resolution images and requires another video's control signal.Source: §2 · p. 2
  • Audio2Head [39] and Wang et al. [40] adapt video-driven approaches [36], while prior 3DMM methods face inaccurate expressions [33] or obvious artifacts.The passage also describes the resulting head movements as not vivid and the faces as distorted with inaccurate identities.Source: §2 · p. 2

Audio-driven Video Portrait

  • Audio-driven video portrait methods use 3DMM for reconstruction and animation, but personalized-video results do not extend to arbitrary photos or in-the-wild audio [19] [38] [11].The task trains and edits on a specific video; AudioDVP, NVP [38], and AD-NeRF [11] reenact expression to edit mouth shape, while related work also addresses head motion [23] [48] and emotional talking faces [18].Source: §2 · pp. 2–3

Video-Driven Single Image Talking Face Generation

  • Video-driven single-image animation uses landmark, 3DMM, or latent representations, but differs from audio-driven generation by transferring motion within the same domain.SadTalker’s face render draws on unsupervised-landmark [42] and 3DMM [33] methods to map learned coefficients to video; those prior methods do not focus on realistic motion coefficients.Source: §2 · p. 3

3. Method

SadTalker uses 3DMM motion coefficients as an intermediate representation: ExpNet and PoseVAE generate expression and head pose separately, then a 3D-aware face render produces the talking-head video.

3.1. Preliminary of 3D Face Model

  • The method uses predicted 3DMMs as an intermediate representation, modeling expression β and pose (r, t) as identity-irrelevant motion coefficients [33].The generated coefficients implicitly modulate the face render for final video synthesis.Source: §3.1 · p. 3

3.2. Motion Coefficients Generation through Audio

  • Because head pose is weakly related to audio while lip motion is strongly connected, SadTalker generates global pose and local expression with separate models.PoseVAE generates head pose, while ExpNet generates expression.Source: §3.2 · p. 3
  • ExpNet predicts expression from audio conditioned on reference expression β0, using lip-only targets to reduce identity and non-lip motion uncertainty.It encodes 0.2-second mel-spectrograms with a ResNet audio encoder and uses a blinking control signal with eye-landmark loss. Training combines coefficient distillation, rendered-landmark loss, and temporal lip-reading loss.Source: §3.2 · pp. 3–4, Figure 3 · p. 4
  • PoseVAE learns residual head motion from the first-frame pose, conditioning generation on audio and style identity to produce stylized, stable, continuous motion.Its conditional VAE uses KL-divergence, mean-square, and adversarial losses.Source: §3.2 · p. 4, Figure 4 · p. 4

3.3. 3D-aware Face Render

  • MappingNet maps expression and head-pose coefficients into unsupervised 3D keypoints, using a temporal coefficient window to smooth motion.Training first fits face-vid2vid self-supervised, then freezes its appearance encoder, canonical keypoints estimator, and image generator while training MappingNet with keypoint and generated-video supervision.Source: §3.3 · pp. 4–5, Figure 5 · p. 5

4. Experiments

SadTalker is evaluated on HDTF against talking-head and audio-to-expression methods, with experiments and ablations assessing video quality, motion, and component contributions. User studies favor its video and motion quality, while the paper notes limitations in teeth rendering and expression variety.

4.1. Implementation Details and Metrics

  • Training uses VoxCeleb, with 1,890 aligned videos from 46 subjects for ExpNet and PoseVAE; evaluation uses the first eight seconds of 346 HDTF videos and their first frames as references.The evaluation covers image realism and sharpness, identity preservation, lip synchronization, head-motion diversity, and audio-motion alignment.Source: §4.1 · p. 5

4.2. Compare with other state-of-the-art methods

  • SadTalker shows better overall video quality and head-pose diversity, comparable lip synchronization to fully talking-head methods, and a score similar to real videos.The authors caution that lip-synchronization metrics may reward unnatural lip motion; visual comparisons describe identity and motion advantages over the compared methods.Source: §4.2 · pp. 5–7, Table 1 · p. 6, Figure 6 · p. 6

4.3. User Studies

  • In a user study of 20 testing videos and 20 participants, participants mostly preferred the authors’ method for video and motion quality; 38% judged its lip synchronization better than other methods.Source: §4.3 · p. 7

4.4. Ablation Studies

  • ExpNet’s initial expression β0, lip-reading loss Lread, and lip-only coefficients are important: removing β0 changes identity, while replacing lip-only coefficients with real coefficients reduces lip synchronization.The ablation caption reports that β0 and Lread improve performance, and that lip synchronization drops when using real coefficients.Source: §4.4 · p. 7, Table 3 · p. 7, Figure 8 · p. 7
  • PoseVAE’s conditions improve motion diversity and beat alignment over the baseline, while the mixed-style full model further improves diversity.The fixed one-hot style is used to assess component ablations; Speech2Gesture performs worse in pose evaluation.Source: §4.4 · pp. 7–8, Table 4 · p. 8
  • Compared with PIRenderer [33], SadTalker’s sparse-keypoint face render improves expression reconstruction; removing alignment coefficients avoids unnatural face-aligned video.The comparison uses the same head-pose and expression coefficients; the authors use those coefficients directly as modulation parameters.Source: §4.4 · p. 8, Figure 9 · p. 8

4.5. Limitation

  • The face render can produce teeth artifacts because 3DMMs do not model eye and teeth variation; the system also omits emotion and gaze, leaving generated videos with fixed emotion.The paper suggests blind face restoration networks [43] for teeth artifacts and leaves broader expressions as future work.Source: §4.5 · p. 8, Figure 10 · p. 8

5. Conclusion

SadTalker generates stylized audio-driven talking-head video using learned 3DMM motion coefficients and a 3D-aware face renderer. The conclusion also describes possible uses in other modalities and watermarking for generated-content identification.

Conclusion

  • The system uses 3DMM motion coefficients, ExpNet for realistic expressions, PoseVAE for diverse head poses, and a 3D-aware face renderer inspired by [42].The authors report experiments demonstrating the superiority of the entire framework.Source: §5 · p. 8
  • The predicted 3D facial coefficients can also be used directly for personalized 2D visual dubbing, 2D cartoon animation, 3D face animation [8] [46], and NeRF-based 4D talking-head generation [15].Source: §5 · p. 8

Ethical Considerations

  • The authors propose inserting visible and invisible watermarks to identify generated videos and hope the method can provide research samples for forgery detection.They compare the watermarking approach to Dall-E [32] and Imagen [34].Source: §5 · p. 9

A. Additional Experiments · A.1. PIRenderer v.s Our FaceRender for Face Reenactment

On HDTF cross-identity video-driven face reenactment over 354 videos, SadTalker’s FaceRender is reported to outperform PIRenderer in visual quality by FID and CSIM.

A.1. PIRenderer v.s Our FaceRender for Face Reenactment

  • FaceRender showed better FID and CSIM than PIRenderer on 354 HDTF cross-identity reenactment videos.The numerical comparison is identified as Table A5; the supplied table passage gives its caption but not the metric values. Visual comparisons are shown in Fig. 9 of the main paper.Source: §A.1 · p. 11, Table A5 · p. 11

A.2. Cross-ID Settings and More Test Datasets

In cross-identity evaluation, SadTalker is tested on HDTF and VoxCeleb2 and shows better lip synchronization on both datasets in most metrics, with the same trend for head motion and visual quality. The setting uses audio from a different video than the reference image.

A.2. Cross-ID Settings and More Test Datasets

  • In cross-identity evaluation, driving audio comes from another video; PC-AVS uses the audio’s corresponding video for its reference pose, and this setting is widely used in video-driven face reenactment comparison [36].The same-identity setting instead uses the test video’s first frame as reference and its corresponding audio as the driving signal, generating synchronized expression with diverse head poses.Source: §A.2 · p. 11
  • SadTalker shows better lip synchronization on HDTF and VoxCeleb2 in most metrics in cross-identity evaluation, with the same trend in head motion and final-video visual quality.The evaluation uses the pretrained models and the same metrics as the same-identity HDTF experiment.Source: §A.2 · p. 11
  • Evaluation covers HDTF and VoxCeleb2; the VoxCeleb2 test set has 36k utterances from 118 speakers, with three randomly selected videos per speaker yielding 354 evaluation videos.Frames are cropped as in [36] and resized to 256×256.Source: §A.2 · p. 11

B. More Implementation Details · B.1. Audio Pre-processing Details · B.2. Network Structure Details

Appendix B provides implementation details on audio preprocessing and network structures, with separate descriptions of audio features, ExpNet, PoseVAE, and FaceRender.

B. More Implementation Details

  • Appendix B outlines detailed audio preprocessing, network structures, loss functions, and alignment coefficients across Sections B.1–B.4.This merged block covers the audio-preprocessing and network-structure details.Source: B. More Implementation Details · p. 11

B.1. Audio Pre-processing Details

  • Audio is resampled to 16 kHz and converted to mel-spectrograms using an 800-sample FFT window, 200-sample hop, and 80 Mel filter banks.Each frame uses a 0.2-second mel-spectrogram feature with shape 16×80, following Wav2Lip [30].Source: §B.1 · p. 11

B.2. Network Structure Details

  • ExpNet uses a four-stage ResBlock-C audio encoder initialized from pre-trained Wav2Lip parameters, followed by a single linear layer.The audio encoder is ΦA and the linear layer is ΦM.Source: §B.2 · p. 11
  • ExpNet maps audio embeddings to expression coefficients βg for frames 1 through t, plus first-frame coefficients β0 and a blink-control signal zblink.Source: §B.2 · p. 11
  • PoseVAE uses linear-layer encoder and decoder networks, conditioning on pose differences, a 46-dimensional identity-style vector, and encoded audio features.The decoder adds the reparameterized feature to the style embedding before concatenating audio features.Source: §B.2 · p. 11
  • FaceRender’s mappingNet converts five-frame pose and expression coefficient windows into face-vid2vid [42] motion representations for the center frame.The window spans [t−2:t+2], and the mapped representation comprises yaw, pitch, roll, translation, and δ; other FaceRender networks follow [42].Source: §B.2 · p. 11

B.3. Loss Function Details

The loss design combines coefficient, landmark, and lip-reading supervision for ExpNet, reconstruction, latent-prior, and adversarial losses for PoseVAE, and keypoint regularization for FaceRender.

ExpNet

  • ExpNet distills generated expression coefficients from lip-only coefficients and uses eye-landmark supervision to limit changes outside the eyes.The distillation loss is mean-squared error; eye-blink control signals are generated uniformly and randomly, and non-eye landmarks are constrained relative to landmarks predicted by the lip-only coefficients.Source: §B.3 · pp. 12–13
  • ExpNet’s lip-reading loss compares rendered-mouth sequence logits with logits generated from ground-truth audio using pretrained lip-reading models [24], inspired by [9].The rendered input uses five cropped mouth-region frames; the loss weights for distillation, lip reading, and landmarks are 2, 0.01, and 0.01, respectively.Source: §B.3 · pp. 13–14

PoseVAE

  • PoseVAE combines mean-squared motion reconstruction, KL divergence toward a Gaussian latent distribution, and adversarial training with a PatchGAN discriminator [16].The discriminator applies 1D convolution to head-motion sequences, as in Speech2Gesture [10]; the stated loss weights are 1, 1, and 0.7 for reconstruction, KL, and adversarial terms, respectively.Source: §B.3 · p. 14

FaceRender

  • FaceRender maps explicit 3DMM coefficients into face-vid2vid’s space [42] and adds L1 regularization between mapped and original motion-generator keypoints.The L1 weight is 20; other face-vid2vid [42] losses retain their original weights and are calculated on the final generated image.Source: §B.3 · p. 14

B.4. More Details about the Alignment Coefficients.

SadTalker excludes alignment coefficients from audio-driven motion because they describe face-crop transformations unrelated to audio, and using first-frame values produces unnatural aligned motion.

B.4. More Details about the Alignment Coefficients.

  • Alignment coefficients are translation-and-scale parameters for cropping faces for deep 3D reconstruction [5]; PIRenderer [33] uses 73 motion dimensions: 64 expression, 6 pose, and 3 alignment, known at test time.Source: §B.4 · p. 14
  • Audio cannot teach alignment coefficients because they lack an audio relationship; first-frame alignment yields unnatural aligned motion, so SadTalker retains only 64 expression and 6 pose coefficients.Source: §B.4 · p. 14

Related papers

Loading 2211.12194v2…