PaperlayerView exact revision on arXiv

Source-linked AI summary

Depth-Aware Generative Adversarial Network for Talking Head Video Generation

Fa-Ting Hong, Longhao Zhang, Li Shen, Dan Xu

arXiv:2203.06605v2cs.CV

TL;DR

Talking-head generation needs dense 3D geometry, but video datasets typically lack affordable depth annotations. DaGAN learns facial depth self-supervised, uses it for keypoints and cross-modal attention, and produces more realistic videos with improved facial-detail, expression, and pose preservation.

  • Problem

    Talking-head generation commonly relies on 2D representations, while dense 3D facial geometry is important for recovering face structure, separating cluttered backgrounds, and capturing expression-related micro-movements; annotations are costly and typically unavailable.

  • Method

    DaGAN self-supervisedly recovers pixel-wise face depth from training videos, then combines depth with appearance for facial keypoints and cross-modal attention that guides motion-field generation.

  • Results

    DaGAN produces accurate depth maps and higher-quality, more realistic face videos than state-of-the-art methods, better preserving facial details, expression, and pose.

  • Takeaways & Limitations

    Self-supervised depth learning can benefit motion transfer between faces while enabling more natural-looking talking-head results without expensive 3D geometry data.

Abstract

from arXiv · show

Talking head video generation aims to produce a synthetic human face video that contains the identity and pose information respectively from a given source image and a driving video.Existing works for this task heavily rely on 2D representations (e.g. appearance and motion) learned from the input images. However, dense 3D facial geometry (e.g. pixel-wise depth) is extremely important for this task as it is particularly beneficial for us to essentially generate accurate 3D face structures and distinguish noisy information from the possibly cluttered background. Nevertheless, dense 3D geometry annotations are prohibitively costly for videos and are typically not available for this video generation task. In this paper, we first introduce a self-supervised geometry learning method to automatically recover the dense 3D geometry (i.e.depth) from the face videos without the requirement of any expensive 3D annotation data. Based on the learned dense depth maps, we further propose to leverage them to estimate sparse facial keypoints that capture the critical movement of the human head. In a more dense way, the depth is also utilized to learn 3D-aware cross-modal (i.e. appearance and depth) attention to guide the generation of motion fields for warping source image representations. All these contributions compose a novel depth-aware generative adversarial network (DaGAN) for talking head generation. Extensive experiments conducted demonstrate that our proposed method can generate highly realistic faces, and achieve significant results on the unseen human faces.

1. Introduction

Talking head generation seeks to preserve identity from a source image and pose from a driving video, but existing 2D representations lack dense 3D facial geometry. DaGAN addresses this gap with self-supervised depth recovery and depth-aware generation mechanisms, achieving stronger qualitative generation results than prior methods.

  • Motivation: Talking head generation combines a person’s identity from a source image with pose information from a driving video.The driving video may depict another person.
  • Motivation: Existing methods commonly decouple identity and pose using appearance, motion, and sparse facial keypoints.Estimated keypoints can model relative poses and generate dense motion fields for warping source-image features.
  • Motivation: Dense 3D geometry can improve face-structure recovery, background-noise separation, and recognition of expression-related micro-movements, but annotations are costly and usually unavailable.These benefits motivate learning depth without expensive 3D supervision.
  • Contributions: DaGAN learns pixel-wise depth from face videos self-supervisedly, then uses depth-guided keypoint detection and depth-image cross-modal attention for generation.The keypoint mechanism combines depth-based geometry with image-based appearance, while attention guides motion-field learning.
  • Results: DaGAN recovers accurate depth maps and generates higher-quality face images than state-of-the-art methods on VoxCeleb1 and CelebV.The generated faces better preserve facial details and produce more accurate expressions and poses.

2. Related Works

Prior work includes GAN-based talking-head generation and unsupervised depth estimation, but DaGAN specifically learns facial depth from video without ground-truth depths. It uses that dense geometry to improve keypoint detection, attention, and fine-grained face generation.

  • Generative Adversarial Networks: GANs have been applied to image synthesis and talking-head video generation, including methods that separate identity and pose information.Image-driving approaches model motion through facial keypoints or related representations.
  • Depth Estimation: Unlike prior unsupervised depth work focused on outdoor scenes, DaGAN adapts self-supervised depth learning to facial videos with large head motion.The paper identifies this setting as challenging for unsupervised facial depth estimation.
  • Depth Estimation: DaGAN learns explicit pixel-wise facial depth self-supervisedly to provide dense 3D geometry for perceiving face structure and generating fine-grained spatial details.The method requires only video images rather than ground-truth 3D geometry.

3. The proposed DaGAN Approach

DaGAN learns dense facial depth self-supervised from videos, then uses it with appearance to estimate motion and refine warped features for talking-head generation. Its depth-guided components target identity preservation, facial structure, and expression-related motion.

  • 3. The proposed DaGAN Approach: DaGAN combines a self-supervised depth-learning sub-network, depth-guided keypoint detection, feature warping, and depth-aware attention.The generator uses these components to transform source-image appearance and driving-face motion into a synthesized image.
  • 3.2. Self-supervised Face Depth Learning: The face depth network learns dense geometry from consecutive video frames using geometric warping and photometric consistency, without 3D annotations.It estimates depth, relative camera pose, translation, and camera intrinsics, then trains against reconstructed-image photometric error.
  • 3.3. Motion Modeling by Sparse Keypoints: Depth-guided keypoint detection concatenates each RGB image with its depth map to estimate sparse facial keypoints for source and driving images.The keypoints encode facial motion that is subsequently used to construct dense 2D motion fields.
  • 3.3. Motion Modeling by Sparse Keypoints: The feature-warping module uses motion and occlusion masks to warp source features while preserving source identity and head-motion information.A motion flow mask weights the estimated motion field, while an occlusion map masks regions requiring inpainting.
  • 3.4. Cross-Modal Attention Module: Cross-modal attention fuses depth-derived queries with appearance-derived keys and values to refine warped features for facial structure and micro-movements.A depth encoder produces geometry features, and the decoder converts the refined warped features into the synthesized image.

4. Experiments

Experiments evaluate DaGAN on VoxCeleb1 and CelebV using quantitative, qualitative, and ablation studies. The results show improved head-motion accuracy, facial realism, expression detail, and benefits from depth-based components.

  • Experimental setup: DaGAN is evaluated extensively on the VoxCeleb1 and CelebV talking-head datasets using multiple image, identity, pose, and expression metrics.The evaluation includes SSIM, PSNR, L1, AKD, AED, CSIM, PRMSE, and AUCON.
  • Self-reenactment: 1.64 point improvement on PRMSE is achieved by DaGAN over FOMM, with scores of 1.22 versus 3.20, respectively.The comparison is conducted for self-reenactment and indicates more accurate head movements for DaGAN.
  • Reenactment results: DaGAN produces more realistic images than competitive methods while preserving facial details and generating more accurate expressions and poses.The reported comparisons cover self-reenactment and cross-identity reenactment settings.
  • Ablation study: The baseline, depth-map, and depth-attention ablations show that adding depth components improves generation, with the full method producing the most realistic images.The baseline excludes the face depth network and cross-modal attention module.
  • Ablation study: 0.181 AUCON improvement is obtained by Baseline w/ CAM over Baseline, indicating better expression-related micro-movement generation.The attention module focuses on important facial regions such as the eyes and mouth.
  • Ablation study: Dense depth-aware attention activates mainly around expression-related regions, including the eyes, nose, and mouth.The visualized attention maps support the module’s role in capturing facial micro-movements.

5. Conclusions

DaGAN learns dense facial geometry without ground-truth depth and uses it to improve keypoint estimation and expression-sensitive attention. Ablations and comparisons show more realistic and natural-looking motion-transfer results.

  • DaGAN learns pixel-wise face depth maps self-supervisedly to recover dense 3D facial geometry for talking-head generation.
  • Depth and RGB information are combined to predict more accurate facial keypoints for motion transfer.
  • A cross-modal depth-and-RGB attention mechanism captures expression-related micro-movements and produces finer facial-structure details.
  • Ablation studies show that depth maps benefit motion transfer, while DaGAN produces more realistic and natural-looking results than state-of-the-art methods.

A. Additional Network and Training Details

DaGAN combines multiscale perceptual, adversarial, equivariance, and keypoint-distance losses to train image synthesis and stabilize facial keypoint estimation.

  • Perceptual loss: The perceptual loss computes L1 distances between generated and ground-truth images at four resolutions.The resolutions are 256 × 256, 128 × 128, 64 × 64, and 32 × 32.
  • GAN loss: The GAN loss uses least-square adversarial learning and feature matching on 256 × 256 generated images and ground truths.
  • Equivariance loss: The equivariance loss enforces consistency of estimated keypoints under known spatial transformations.
  • Keypoint-distance loss: The keypoint-distance loss penalizes pairs of facial keypoints that fall below a predefined separation threshold.The threshold α is set to 0.2 in the reported implementation.

A.2. Network architecture details of DaGAN

DaGAN uses dedicated depth, keypoint, occlusion, feature-encoding, upsampling, and discriminator components. These sub-networks process RGB-D inputs and produce keypoints, occlusion information, motion weighting, features, and adversarial predictions.

  • Face depth network: The face depth network uses a ResNet18 encoder without its final fully connected and pooling layers, paired with a decoder.
  • Keypoint estimator: RGB images and corresponding depth maps are concatenated into 4 × 256 × 256 RGB-D inputs for facial keypoint estimation.The estimator outputs K two-dimensional keypoints.
  • Occlusion estimator: The occlusion estimator predicts an occlusion map and a motion-flow mask through two output heads.The occlusion map filters regions requiring inpainting, while the mask weights the motion field.
  • Network blocks: DownBlock2d applies convolution, batch normalization, ReLU, and average pooling, while UpBlock2d uses interpolation for upsampling.The “/2” symbol in other sub-networks denotes average-pooling downsampling.
  • Discriminator: The discriminator downsamples the input four times and outputs a 512 × 26 × 26 prediction map while collecting intermediate features for GAN loss.

B.1. Dataset Details

The evaluation uses VoxCeleb1 for self-reenactment and CelebV for reenacting unseen targets under in-the-wild-like variation.

  • VoxCeleb1 contains 1,251 identities at 256 × 256 resolution and provides 2,083 sampled image sets for self-reenactment evaluation.
  • CelebV contains videos of five celebrities with widely varying characteristics and provides 2,000 sampled image sets for unseen-target reenactment experiments.

B.2. Compare methods

The comparison includes methods that warp images directly, use adaptive normalization or attention, detect facial keypoints, model dense face meshes, or generate keypoints with a novel method.

  • X2Face uses a simple framework to warp the image directly, with VoxCeleb1 results obtained from prior work.
  • NeuralHead adopts AdaIN layers from style transfer, and its reported results come from replication because no reference implementation is available.
  • MarioNETte combines an image attention block, target feature alignment, and landmark transformer to address identity preservation.
  • FOMM detects facial keypoints and models motion between two images using those detected keypoints.
  • MeshG generates a dense face mesh and models a dense motion map with a graph convolutional network, using results from its original paper.
  • OSFV provides a novel keypoint-generation method reimplemented and trained on VoxCeleb1 for comparison.

B.3. More results

The results show that depth-aware attention focuses on facial regions rather than cluttered backgrounds, while DaGAN produces natural-looking faces and foreground-separating depth maps.

  • Depth-aware cross-modal attention attends to key foreground facial regions, whereas the depth-disabled self-attention also attends to cluttered backgrounds.
  • DaGAN produces more natural-looking faces than the other comparison methods in qualitative samples.
  • The estimated depth maps distinguish face foreground areas from image backgrounds in source and driving images.
Loading 2203.06605v2…