PaperlayerView exact revision on arXiv

Source-linked AI summary

SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation

Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, Fei Wang

arXiv:2211.12194v2cs.CV

TL;DR

SadTalker addresses unnatural motion, distorted expressions, and identity changes in audio-driven talking-head animation by generating disentangled 3D motion coefficients and rendering them with a 3D-aware face model. Experiments show state-of-the-art motion synchronization and video quality.

  • Problem

    Audio-driven talking-head methods struggle to produce natural videos because head pose and facial expression are not fully disentangled in 2D or latent representations.

  • Method

    SadTalker separately predicts expression and head-pose 3DMM coefficients with ExpNet and PoseVAE, then uses them to modulate a 3D-aware face renderer.

  • Results

    SadTalker achieves state-of-the-art performance in motion synchronization and video quality.

  • Takeaways & Limitations

    The framework produces realistic single-image talking-head videos with diverse head poses and realistic facial expressions.

  • Takeaways & Limitations

    Because 3DMMs omit eye and teeth variation and the system models only lip motion and blinking, generated videos may have unrealistic teeth and fixed emotion.

Abstract

from arXiv · show

Generating talking head videos through a face image and a piece of speech audio still contains many challenges. ie, unnatural head movement, distorted expression, and identity modification. We argue that these issues are mainly because of learning from the coupled 2D motion fields. On the other hand, explicitly using 3D information also suffers problems of stiff expression and incoherent video. We present SadTalker, which generates 3D motion coefficients (head pose, expression) of the 3DMM from audio and implicitly modulates a novel 3D-aware face render for talking head generation. To learn the realistic motion coefficients, we explicitly model the connections between audio and different types of motion coefficients individually. Precisely, we present ExpNet to learn the accurate facial expression from audio by distilling both coefficients and 3D-rendered faces. As for the head pose, we design PoseVAE via a conditional VAE to synthesize head motion in different styles. Finally, the generated 3D motion coefficients are mapped to the unsupervised 3D keypoints space of the proposed face render, and synthesize the final video. We conducted extensive experiments to demonstrate the superiority of our method in terms of motion and video quality.

1 Xi’an Jiaotong University 2 Tencent AI Lab 3 Ant Group

This section identifies the authors’ institutional affiliations and notes contribution and correspondence designations. It also presents SadTalker as producing diverse, realistic, synchronized talking videos from audio and a single reference image.

  • SadTalker produces diverse, realistic, synchronized talking videos from input audio and a single reference image.
  • The paper marks some authors as having equal contribution and identifies a corresponding author.

1. Introduction

SadTalker addresses the challenge of generating natural-looking talking-head videos by disentangling audio-driven motions through realistic 3D motion coefficients and a 3D-aware face renderer. Its components separately model expression and head pose, producing state-of-the-art motion synchronization and video quality.

  • Motivation: Audio-driven talking-head generation is difficult because lip motion, head pose, and eye blinking have different relationships with speech.Lip movement has the strongest audio connection, while speech can accompany different head poses and eye blinks.
  • Method: SadTalker uses generated realistic 3DMM motion coefficients as an intermediate representation for stylized single-image talking-face animation.The system generates coefficients such as head pose, lip motion, and eye blink from audio, learning each motion individually.
  • Method: ExpNet and PoseVAE separately learn realistic 3DMM motion coefficients for facial expression and head-pose generation.The paper introduces these sub-networks individually to model motion from audio.
  • Method: A semantic-disentangled, 3D-aware face renderer produces the final realistic talking-head video.Expression generation, head-pose generation, and face rendering are trained individually, while inference can proceed end to end.
  • Results: Experiments show state-of-the-art performance in motion synchronization and video quality.Several metrics demonstrate advantages in video and motion quality.

2. Related Work

Related work on talking-face generation includes audio-driven single-image animation, audio-driven video portrait editing, and video-driven single-image face reenactment. These approaches use landmarks, audio disentanglement, or 3DMM representations, but personalized video-portrait methods do not directly apply to arbitrary photos and in-the-wild audio.

  • Audio-driven Single Image Talking Face Generation: Audio-driven single-image methods initially focused on accurate lip motion, while later approaches used facial landmarks or disentangled content and speaker information from audio.ATVGnet uses facial landmarks as an intermediate representation, and MakeItTalk disentangles content and speaker information from the input audio.
  • Audio-driven Video Portrait: Audio-driven video portrait methods edit specific portrait videos using 3DMM information for face reconstruction and animation.AudioDVP, NVP, and AD-NeRF are cited as methods that learn to reenact expression to edit the mouth shape.
  • Audio-driven Video Portrait: Beyond lip movement, related work also addresses head motions and emotional talking faces through 3DMM-based methods.3DMM parameters are practical to fit from a video clip, supporting these personalized-video tasks.
  • Video-Driven Single Image Talking Face Generation: Video-driven single-image talking-face generation, also called face reenactment or face animation, transfers motion from a source image to a target person.Prior methods learn shared intermediate motion representations, including landmark-based, unsupervised landmark-based, and 3DMM-based approaches.

3. Method

SadTalker uses disentangled 3DMM motion coefficients—expression and head pose—as an intermediate representation for audio-driven talking-head generation. ExpNet predicts expressions, PoseVAE synthesizes stylized head motion, and a 3D-aware face render maps these coefficients to final videos.

  • 3.1 Overview: The system extracts identity and motion coefficients from the source image, generates expression and head-pose coefficients separately, and renders the final talking-head video.It models motion as expression β and head pose ρ = [r, t], excluding identity coefficients from generation.
  • 3.2 Motion Coefficients Generation: Expression and head pose are learned independently because pose is global and weakly related to audio, whereas lip motion is local and strongly connected to audio.The separation reduces uncertainty caused by jointly learning motion types with different audio relationships.
  • 3.2 Motion Coefficients Generation: ExpNet predicts framewise expression coefficients from 0.2-second mel-spectrogram features while conditioning on the reference expression β0 to reduce identity uncertainty.A ResNet-based audio encoder embeds each audio feature, and a linear mapping network decodes expression coefficients.
  • 3.2 Motion Coefficients Generation: ExpNet distills lip-only expression coefficients from a Wav2Lip-generated video and adds differentiable 3D-rendered facial losses to improve realistic expression prediction.Using the first frame for Wav2Lip reduces pose variation and non-lip facial-expression effects in the training target.
  • 3.2 Motion Coefficients Generation: PoseVAE learns identity-aware, stylized head movement by modeling residual pose relative to the first-frame pose under audio and style conditions.Its conditional VAE uses sequential head poses during training and samples residual head-pose distributions for stylized generation.
  • 3.3 3D-Aware Face Render: The FaceRender mappingNet converts expression and head-pose coefficients into an unsupervised 3D keypoint space and drives a pretrained face-vid2vid-style animator.Temporal coefficients from a time window are used for smoothing, while only expression and head-pose coefficients drive the mapping.

4. Experiments

Experiments evaluate SadTalker on high-resolution in-the-wild talking-head videos using image quality, sharpness, identity, and lip-synchronization metrics. Comparisons, user studies, and ablations show improved video quality, motion diversity, and coefficient modeling, while revealing limitations in teeth, eyes, and facial-expression coverage.

  • Experimental setup: SadTalker trains FaceRender on over 100k VoxCeleb videos and trains PoseVAE and ExpNet on 1,890 aligned videos from 46 subjects, then evaluates on 346 HDTF videos.HDTF inputs are resized to 256×256, with each video’s first frame used as the reference image.
  • Evaluation: The evaluation measures generated-frame realism with FID, sharpness with CPBD, identity preservation with CSIM, and audio-visual lip synchronization with synchronization metrics.The metrics are intended to assess image quality, identity preservation, and lip synchronization, respectively.
  • Comparison with prior methods: Compared with MakeItTalk, Audio2Head, Wang et al., Wav2Lip, and PC-AVS, SadTalker delivers better overall video quality and head-pose diversity while remaining comparable on lip-synchronization metrics.The paper notes that lip-synchronization metrics can reward unnatural lip movement, whereas SadTalker achieves a score similar to real videos.
  • Comparison with prior methods: Visual comparisons show that SadTalker preserves identity and avoids the blur, front-facing restriction, and 2D-warping distortions observed in competing methods.Wav2Lip produces a blurred half-face; PC-AVS and Audio2Head struggle with identity preservation; Audio2Head generates only front-facing talking faces; MakeItTalk and Audio2Head produce distorted videos.
  • Ablation studies: Ablations show that separately modeling expression and pose is effective: ExpNet benefits from the initial expression β0 and lip-reading loss Lread, while PoseVAE conditions improve motion diversity and audio-beat alignment.Jointly learning all motion coefficients produces less truth-worthy videos; mixed identity styles further improve diversity, and removing crop coefficients avoids unnatural face-aligned motion.
  • Limitations: The system remains limited by 3DMMs’ incomplete modeling of eye and teeth variation and by its focus on lip motion and eye blinking rather than other facial expressions.The mapping network can struggle to synthesize realistic teeth, though blind face restoration can improve this issue.

5. Conclusion

The paper presents a stylized audio-driven talking head system that uses 3DMM motion coefficients as an intermediate representation, with ExpNet, PoseVAE, and a 3D-aware face renderer. It also addresses misuse by proposing visible and invisible watermarks and positioning the method as a source of research samples for forgery detection.

  • Conclusion: The system uses 3DMM motion coefficients as an intermediate representation for stylized audio-driven talking head video generation.It learns relationships involving these motion coefficients to connect audio-driven motion with video generation.
  • Conclusion: ExpNet and PoseVAE generate realistic expressions and diverse head poses from audio.These components are proposed specifically to produce realistic 3D coefficients.
  • Conclusion: The method uses a novel 3D-aware face render to model relationships between 3DMM motion coefficients and real video.The renderer is inspired by an image animation method cited as [42].
  • Ethical Considerations: Because the method can generate very realistic video from a single face image, the authors propose inserting visible and invisible watermarks into generated content.The watermarks are intended for generated-content identification, similar to Dall-E [32] and Imagen [34].
  • Ethical Considerations: The authors hope the method can provide new research samples for forgery detection.This is presented as an additional potential use alongside generated-content identification.

A. Additional Experiments · A.1. PIRenderer v.s Our FaceRender for Face Reenactment

The section compares FaceRender with PIRenderer for cross-identity, video-driven face reenactment on the HDTF dataset. Across 354 videos, FaceRender is reported to provide better visual quality in FID and CSIM.

  • A.1. PIRenderer v.s Our FaceRender for Face Reenactment: The experiment evaluates FaceRender against PIRenderer for video-driven face reenactment.Visual comparisons were previously shown in Fig. 9 of the main paper.
  • A.1. PIRenderer v.s Our FaceRender for Face Reenactment: The comparison uses cross-identity settings.Both methods are evaluated under the same reenactment setting.
  • A.1. PIRenderer v.s Our FaceRender for Face Reenactment: The evaluation is conducted on the HDTF dataset.The passage specifies HDTF as the dataset for the numerical comparison.
  • A.1. PIRenderer v.s Our FaceRender for Face Reenactment: 354 videos are included in the evaluation.The numerical results are reported over 354 videos.
  • A.1. PIRenderer v.s Our FaceRender for Face Reenactment: FaceRender shows much better visual quality than PIRenderer in terms of FID.The passage attributes this finding to the numerical comparison in Table A5.
  • A.1. PIRenderer v.s Our FaceRender for Face Reenactment: FaceRender also shows much better visual quality than PIRenderer in terms of CSIM.The reported FID and CSIM results are presented as evidence for the proposed method’s advantage.

A.2. Cross-ID Settings and More Test Datasets

The appendix evaluates SadTalker beyond same-identity testing by introducing a cross-identity setting in which audio from another video drives the reference image. It reports evaluations on HDTF and VoxCeleb2, including a 354-video VoxCeleb2 test subset.

  • Cross-Identity Setting: Cross-identity testing drives a reference image with audio from another video, unlike same-identity testing, which uses corresponding image and audio.The same-identity setting produces synchronized expressions with diverse head poses; cross-identity evaluation is also common in video-driven face reenactment comparisons.
  • Test Datasets: The method is evaluated on HDTF and VoxCeleb2 under the cross-identity setting, complementing the same-identity HDTF evaluation in the main paper.The appendix reports these cross-identity results in Tables A6 and A7.
  • Test Datasets: VoxCeleb2 contains over 1 million utterances from 6112 speakers, with a test set of 36k utterances from 118 speakers.For evaluation, three videos are randomly selected per speaker, yielding 354 videos.

B. More Implementation Details · B.1. Audio Pre-processing Details · B.2. Network Structure Details

The implementation follows Wav2Lip-style audio preprocessing and specifies separate structures for ExpNet, PoseVAE, and FaceRender. These details define how audio features, motion coefficients, style conditions, and temporal context are processed.

  • B. More Implementation Details: The implementation details cover audio preprocessing, network structures, loss functions, and alignment coefficients across Sections B.1–B.4.
  • B.1. Audio Pre-processing Details: Audio is resampled to 16k Hz and converted into mel-spectrograms using an 800-sample FFT window, 200-sample hop, and 80 Mel filter banks.Each frame yields a 0.2s feature with shape 16×80.
  • B.2. Network Structure Details: ExpNet combines a pretrained Wav2Lip-initialized audio encoder ΦA with a single linear layer ΦM to predict expression coefficients.
  • B.2. Network Structure Details: ExpNet also generates the first-frame expression coefficients β0 and a blink-control signal zblink alongside the sequence βg{1,...,t}.
  • B.2. Network Structure Details: The ExpNet audio encoder uses four ResBlock-C stages, while its motion-prediction head consists of one linear layer.
  • B.2. Network Structure Details: PoseVAE uses several linear layers in both encoder and decoder, conditioning on pose differences, audio features, and a 46-dimensional identity style vector.The training dataset contains 46 identities.
  • B.2. Network Structure Details: PoseVAE adds the re-parameterized feature to the style embedding and concatenates audio features in the decoder, paralleling the encoder’s conditioning design.
  • B.2. Network Structure Details: FaceRender introduces mappingNet to transform 3DMM motion coefficients into unsupervised 3D keypoints using five-frame pose and expression windows.The mapping network produces yaw, pitch, roll, tr, and δ for frame t.

B.3. Loss Function Details · B.4. More Details about the Alignment Coefficients.

ExpNet combines coefficient distillation, landmark-based constraints, and lip-reading supervision, while PoseVAE uses reconstruction, distribution-matching, and adversarial losses. FaceRender maps explicit 3DMM coefficients into face-vid2vid’s keypoint space, and the alignment coefficients transform and crop videos for aligned 3D face reconstruction.

  • B.3. Loss Function Details: ExpNet distills lip-only expression coefficients from wav2lip and 3D face capture using a Mean-Squared loss against generated expression coefficients.The loss is computed over T frames.
  • B.3. Loss Function Details: ExpNet constrains rendered-face landmarks with an eye loss driven by randomly generated blinking signals and minimal changes to non-eye landmarks.The non-eye landmark set M excludes the eye areas.
  • B.3. Loss Function Details: ExpNet adds lip-reading supervision by rendering generated expressions, cropping mouth regions, and minimizing the difference between predicted and ground-truth character-sequence logits.The pretrained video-based lip-reading model takes sequences of 5 cropped frames in this setup.
  • B.3. Loss Function Details: ExpNet sets λdistill, λread, and λlks to 2, 0.01, and 0.01, respectively.These weights define the relative contributions of the distillation, lip-reading, and landmark-related losses.
  • B.3. Loss Function Details: PoseVAE combines Mean-Squared reconstruction, KL-divergence regularization toward a Gaussian latent distribution, and PatchGAN-based adversarial loss on head-motion sequences.The discriminator applies 1D convolution to the head-motion sequence.
  • B.3. Loss Function Details: FaceRender maps explicit 3DMM coefficients into face-vid2vid’s space and adds L1 regularization on unsupervised keypoints, weighted by 20.Other loss weights remain those of face-vid2vid and are calculated on the final generated image.
  • B.4. More Details about the Alignment Coefficients.: Alignment coefficients are translation and scaling parameters that transform and crop arbitrary videos into aligned face videos for deep 3D face reconstruction; PIRenderer’s implicit modulation has 73 motion coefficients.These include expression (64), head pose (6), and alignment coefficients.
Loading 2211.12194v2…