PaperlayerView exact revision on arXiv

Source-linked AI summary

Audio-driven Talking Face Video Generation with Learning-based Personalized Head Pose

Ran Yi, Zipeng Ye, Juyong Zhang, Hujun Bao, Yong-Jin Liu

arXiv:2002.10137v2cs.CVcs.GR

TL;DR

Existing talking-face generators generally use fixed head pose, limiting natural personalized motion. This paper learns personalized pose from a short target video, reconstructs and re-renders 3D facial animation, and refines frames with a memory-augmented GAN. Experiments and user studies report high-quality videos with personalized head pose, expressions, and lip synchronization.

  • Problem

    Most existing talking-face video generation methods consider facial animation with fixed head pose, although natural head movement is important for realistic communication.

  • Method

    The method adapts a general audio-to-expression-and-pose mapping using a short target video, reconstructs 3D facial animation, and refines rendered frames with a memory-augmented GAN.

  • Results

    Experiments and user studies show high-quality talking-face videos with personalized head pose, expressions, and good lip synchronization.

  • Takeaways & Limitations

    The system transfers arbitrary source-person audio to arbitrary target faces while learning personalized talking behavior from about 300 frames.

  • Takeaways & Limitations

    The illumination model assumes a Lambertian surface and approximates illumination with spherical harmonics.

Abstract

from arXiv · show

Real-world talking faces often accompany with natural head movement. However, most existing talking face video generation methods only consider facial animation with fixed head pose. In this paper, we address this problem by proposing a deep neural network model that takes an audio signal A of a source person and a very short video V of a target person as input, and outputs a synthesized high-quality talking face video with personalized head pose (making use of the visual information in V), expression and lip synchronization (by considering both A and V). The most challenging issue in our work is that natural poses often cause in-plane and out-of-plane head rotations, which makes synthesized talking face video far from realistic. To address this challenge, we reconstruct 3D face animation and re-render it into synthesized frames. To fine tune these frames into realistic ones with smooth background transition, we propose a novel memory-augmented GAN module. By first training a general mapping based on a publicly available dataset and fine-tuning the mapping using the input short video of target person, we develop an effective strategy that only requires a small number of frames (about 300 frames) to learn personalized talking behavior including head pose. Extensive experiments and two user studies show that our method can generate high-quality (i.e., personalized head movements, expressions and good lip synchronization) talking face videos, which are naturally looking with more distinguishing head movement effects than the state-of-the-art methods.

1 INTRODUCTION

Existing audio-driven talking-face methods commonly keep head pose fixed, despite natural head movement being important for realistic communication. This paper proposes personalized pose learning from a short target video, 3D face animation, and memory-augmented frame refinement.

  • Most existing audio-driven talking-face methods generate facial animation with fixed head pose.
  • Natural head movement matters for high-quality communication, while fixed poses can feel uncomfortable in interaction.
  • The system uses source-person audio and a short target-person video to learn personalized talking behavior, including head pose.
  • 3D face reconstruction bridges audio-visual head-pose learning and realistic video generation despite in-plane and out-of-plane rotations.
  • The proposed model transfers arbitrary source audio to arbitrary target faces with personalized head pose and lip synchronization.
  • A memory-augmented GAN refines rendered frames into realistic frames for different target identities.

2 RELATED WORK

Related work spans video-driven reenactment, audio-driven talking-face generation, 3D face reconstruction, GAN-based synthesis, and memory networks. The paper combines these directions to refine realistic frames for arbitrary target identities using few-shot adaptation.

  • Talking-face generation is driven either by video frames or audio, with audio-driven methods targeting specific identities or arbitrary faces.
  • Audio-driven methods learn mappings from audio and facial inputs to expressions, landmarks, or talking-face video.
  • 3D face reconstruction methods estimate shape and appearance from images, commonly using 3D Morphable Models and learned CNN mappings.
  • Earlier GAN-based rendering could produce good frames but was limited to a specific target person and required thousands of training examples.
  • The method’s pipeline combines audio-to-expression-and-pose mapping, rendering with background matching, and memory-augmented GAN refinement.
  • Memory networks support few-shot learning by storing critical information, which this paper uses to refine rendered frames for arbitrary people.

3 OUR METHOD

The method combines audio, a short target-person video, and 3D facial animation to generate talking-face videos with personalized pose and realistic identity-preserving frames. A general mapping is fine-tuned with limited target-person data, while a memory-augmented GAN refines rendering and supports multiple identities.

  • 3 OUR METHOD: The system transfers source-person audio to a target-person talking-face video while modeling personalized head pose, facial expression, and lip motion.It uses both audio and visual information from the target person's short video.
  • 3 OUR METHOD: A general audio-to-expression-and-pose mapping is trained on LRW and then fine-tuned using the target person's short video.The mapping uses an LSTM with an audio encoder and continuity losses for expression and pose sequences.
  • 3 OUR METHOD: 3D face reconstruction preserves target identity, texture, and illumination while combining them with audio-driven expression and pose coefficients for rendering.The reconstructed 3DMM represents identity, expression, texture, illumination, and pose; the mixed coefficients are rendered into face-image sequences.
  • 3 OUR METHOD: Backgrounds are matched at keyframes corresponding to critical head movements and linearly interpolated between neighboring keyframes to create smooth transitions.The approach addresses the sparsity of suitable backgrounds in the short target video and modifies poses to fit matched backgrounds.
  • 3 OUR METHOD: Rendered frames are refined by a memory-augmented GAN that conditions generation on identity features and can produce different refinement effects for various target identities.The generator uses attention, while the memory network stores and retrieves representative identity features.
  • 3 OUR METHOD: The personalized fine-tuning strategy requires only about 300 frames from the target video, rather than thousands of frames for a single specified identity.The general mapping is learned first, after which a small target-specific sample adapts the network.

4 EXPERIMENTS

Experiments evaluate the model through ablations, comparisons with state-of-the-art methods, quantitative metrics, and two user studies. Results support the importance of pose estimation, identity conditioning, memory retrieval, and personalized fine-tuning for realistic talking-face generation.

  • Experiment setup: 300 frames are sufficient to fine-tune the general mapping for personalized talking behavior.The personalized mapping Ours-P is fine-tuned from a general mapping using short target-person videos.
  • Ablation study: Without pose estimation, generated videos retain good lip synchronization but appear rigid because the head position remains fixed.This ablation isolates the contribution of the first-stage pose prediction.
  • Ablation study: Removing identity features and the memory network produces poor mouth details, uneven cheeks, and black facial spots.The ablated model cannot maintain personalized refinement across identities and expressions.
  • Ablation study: Replacing the memory network with mean identity features yields less realistic results with weaker fine details than the full model.The full model produces better details such as wrinkles.
  • Comparison with state of the arts: The personalized model produces realistic videos with more distinguishing head movements than state-of-the-art methods, while the general model remains comparable in lip synchronization.Compared methods use one facial image and fixed head positions, whereas Ours-P uses additional frames for personalized pose learning.
  • User studies: 20 participants rated the method best across image quality, lip synchronization, and naturalness in the subjective evaluation.The study compared 30 groups, each containing one original video and four generated videos.
  • Quantitative comparison: The method achieves the best PSNR values and comparable SSIM and LMD values with ATVG on the LRW dataset.PSNR and SSIM measure image quality, while LMD evaluates lip-movement accuracy.
  • Head pose behavior: The average head-pose similarity score is 0.859, with a 0.65 correlation between the metric and subjective rankings.The maximum and minimum HS scores are 0.956 and 0.702, respectively.

5 CONCLUSION

The proposed model generates talking-face videos with personalized head pose by reconstructing and re-rendering 3D facial animation, then refining frames with a memory-augmented GAN. Experiments and user studies report high-quality videos with personalized head pose, a feature not considered by prior state-of-the-art methods.

  • 3D facial animation bridges audio-visual head-pose learning and realistic video generation before memory-augmented GAN refinement.The GAN fine-tunes rendered frames into realistic video frames.
  • Experiments and user studies show high-quality talking-head videos with personalized head pose, a feature absent from state-of-the-art audio-driven methods.

APPENDIX A CORRELATION BETWEEN AUDIO AND HEAD POSE

The method learns a mapping from audio features to facial expression and head pose through general training followed by target-person fine-tuning. Correlation analysis represents audio with MFCC features and pose with pitch, yaw, and roll angles.

  • The network maps audio features to facial expression and head pose using general training followed by target-person fine-tuning.The general mapping is trained on LRW, then adapted using a short target-person video.
  • MFCC features represent audio, while pitch, yaw, and roll represent head pose in the correlation analysis.
  • The analysis compares distances between neighboring MFCC pairs and corresponding pose pairs within the same short video.

APPENDIX B USER STUDY ON THE LENGTH OF INPUT SHORT VIDEO

The study evaluates how personalized-mapping video length affects generated quality using inputs from 100 to 800 frames. Results indicate that at least about 300 frames are needed for good quality, while 300 frames balance quality and training efficiency.

  • The study compares personalized mappings fine-tuned with 100, 200, 300, 500, and 800 frames.These correspond to input videos of 4, 8, 12, 20, and 32 seconds.
  • The expert selected results trained with 300, 500, and 800 frames as highest quality.Fewer-frame results showed lower image quality around the mouth and teeth and sometimes looked strange.
  • 0%, 0%, 36.4%, 36.4%, and 63.6% of users selected the 100-, 200-, 300-, 500-, and 800-frame results as best, respectively.The study included 11 participants, and users could select multiple results.
Loading 2002.10137v2…