PaperlayerView exact revision on arXiv

Source-linked AI summary

DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Video

Zhimeng Zhang, Zhipeng Hu, Wenjin Deng, Changjie Fan, Tangjie Lv, Yu Ding

arXiv:2303.03988v1cs.CV

TL;DR

Few-shot face dubbing on high-resolution videos remains challenging because prior methods struggle to generate high-fidelity results. DINet deforms reference-image features and inpaints the source mouth, and the authors report that it outperforms state-of-the-art works on high-resolution dubbing.

  • Problem

    Few-shot face visually dubbing methods struggle to produce realistic high-resolution videos, with direct pixel generation making high-frequency mouth texture difficult to generate from audio.

    Source: Introduction · p. 1

  • Method

    DINet spatially deforms reference-image features to align mouth shape with driving audio and head pose with the source, then merges them with source features to inpaint the mouth.

    Source: Introduction · p. 2

  • Results

    DINet outperforms state-of-the-art works on high-resolution face visually dubbing in qualitative and quantitative experiments.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The authors conclude that DINet preserves more textural details than existing methods and that experiments validate its performance on high-resolution face visually dubbing.

    Source: Conclusion · p. 7

    DINet is limited to frontal-view head poses in its HDTF and MEAD-Neutral training videos and may be sensitive to reference-image selection.

    Source: Limitations · p. 7

Abstract

from arXiv · show

For few-shot learning, it is still a critical challenge to realize photo-realistic face visually dubbing on high-resolution videos. Previous works fail to generate high-fidelity dubbing results. To address the above problem, this paper proposes a Deformation Inpainting Network (DINet) for high-resolution face visually dubbing. Different from previous works relying on multiple up-sample layers to directly generate pixels from latent embeddings, DINet performs spatial deformation on feature maps of reference images to better preserve high-frequency textural details. Specifically, DINet consists of one deformation part and one inpainting part. In the first part, five reference facial images adaptively perform spatial deformation to create deformed feature maps encoding mouth shapes at each frame, in order to align with the input driving audio and also the head poses of the input source images. In the second part, to produce face visually dubbing, a feature decoder is responsible for adaptively incorporating mouth movements from the deformed feature maps and other attributes (i.e., head pose and upper facial expression) from the source feature maps together. Finally, DINet achieves face visually dubbing with rich textural details. We conduct qualitative and quantitative comparisons to validate our DINet on high-resolution videos. The experimental results show that our method outperforms state-of-the-art works.

Introduction

Few-shot face dubbing aims to synchronize a source video's mouth to driving audio while preserving identity and head pose, but direct pixel-generation methods struggle to produce high-resolution results. DINet deforms reference-image features and combines them with source features, and the paper reports that it outperforms state-of-the-art works on high-resolution dubbing.

Introduction

  • Few-shot dubbing synchronizes a source video's mouth shape to driving audio while keeping identity and head pose consistent, but direct-generation methods struggle on high-resolution video.The paper notes that this difficulty persists despite using high-resolution training data.Source: Introduction · p. 1
  • Under few-shot conditions, mouth texture has few correlations with driving audio, making high-frequency details difficult to generate directly.The authors say networks can ignore correlated textural details when learning audio-lip synchronization across identities and speaking styles.Source: Introduction · p. 1
  • DINet spatially deforms reference facial features to synchronize mouth shape with audio and align head pose with the source face.Deformation moves pixels into appropriate locations rather than generating them from scratch, which the authors say nearly preserves textural details.Source: Introduction · p. 2
  • DINet's inpainting part merges source-face and deformed-reference features to repair pixels in the source mouth region.The deformation part encodes head-pose and speech-content features before deforming the reference faces.Source: Introduction · p. 2
  • DINet outperforms state-of-the-art works on high-resolution face visually dubbing in qualitative and quantitative experiments.The authors report accurate mouth movements while preserving textual details.Source: Introduction · p. 2

Related Work

Prior talking-face work spans one-shot, few-shot dubbing, and person-specific approaches, while spatial deformation methods use affine transforms or dense flow. For few-shot dubbing, DINet replaces direct pixel generation with deformation to produce more realistic high-resolution results.

Talking Face Generation

  • One-shot methods drive a reference face using latent embeddings or intermediate representations such as keypoints, landmarks, and 3DMM.Some split audio-to-animation and animation-to-video into separately trained stages.Source: Talking Face Generation · p. 2
  • Few-shot dubbing methods repair a source face's mouth using driving audio, but approaches based on direct pixel generation fail to realize dubbing on high-resolution videos.Prior work also uses landmarks, SSIM loss, sync loss, or audio-lip memory to improve visual quality or lip synchronization.Source: Talking Face Generation · p. 2
  • Person-specific methods require the target identity in training, with reported data needs ranging from 16 hours to several minutes of footage.Other work addresses changing light or realistic facial expressions.Source: Talking Face Generation · p. 2

Spatial Deformation

  • Spatial deformation approaches use affine transformations or dense flow, and DINet uses AdaAT because it synthesizes the best results.AdaAT computes channel-specific coefficients, increasing the number of affine transformations to hundreds.Source: Spatial Deformation · p. 2
  • DINet's deformation part aligns reference-feature mouth shape to driving audio and head pose to the source, then its inpainting part repairs the source mouth.The framework illustration describes these as the deformation and inpainting parts, respectively.Source: Figure 2 · p. 3

Method

DINet combines spatial deformation of reference-image features with inpainting of the source mouth region. Its training objective combines perceptual, adversarial, and lip-sync losses.

Method

  • DINet has deformation and inpainting parts: the first warps reference features, and the second combines them with source features to repair the mouth region.Source: Method · p. 3, Inpainting Part · p. 3

Deformation Part

  • The deformation part uses audio and pose alignment features to drive AdaAT channel-specific affine warps, producing reference features with audio-synchronous mouth shapes and source-aligned head pose.Source: Deformation Part · p. 3

Inpainting Part

  • A convolutional feature decoder combines source and deformed features to inpaint the masked mouth and produce the dubbed image.Source: Inpainting Part · p. 3

Loss Function

  • Training combines two-scale VGG perceptual loss, LS-GAN loss on single frames and five-frame clips, and lip-sync loss using DeepSpeech features.The experiment sets λp = 10 and λsync = 0.1.Source: Loss Function · p. 4

Experiment

DINet is evaluated on high-resolution talking-face datasets through qualitative and quantitative comparisons, ablations, and a user study. Its comparisons favor DINet on visual quality, while synchronization metrics favor Wav2Lip and PC-AVS.

Datasets

  • HDTF testing uses 20 randomly selected videos, while MEAD-Neutral testing uses 240 videos from 6 identities.MEAD-Neutral contains neutral-emotion, frontal-view videos; HDTF includes in-the-wild 720P or 1080P videos.Source: Datasets · p. 4

Implementation Details

  • Faces are resized to 416 × 320, with a 256 × 256 mouth region, and audio features are aligned to video at 25 fps.Videos are resampled to 25 fps; the authors use pretrained DeepSpeech features to improve generalization given limited subjects.Source: Implementation Details · p. 5

Synthetic Results

  • DINet’s intermediate results show spatially misaligned reference features deformed by AdaAT through scaling, rotation, and translation.The resulting feature maps are combined with source features to produce dubbed images.Source: Synthetic Results · p. 5, Figure 4 · p. 5

Comparisons with State-of-the-Art Works

  • Qualitatively, DINet produces more realistic 416 × 320 results than ATVG, while Wav2Lip variants remain blurry and MakeitTalk produces inaccurate mouth shapes.PC-AVS outputs 224 × 224 videos; the authors attribute its resolution limit to direct generation and contrast it with DINet’s deformation and inpainting.Source: Comparisons with State-of-the-Art Works · pp. 5–6, Figure 5 · p. 6
  • DINet achieves the best results on SSIM, PSNR, and LPIPS, but performs worse than Wav2Lip and PC-AVS on audio-visual synchronization.The comparison covers talking-face generation methods; Table 1 presents the quantitative comparisons.Source: Table 1 · p. 5, Comparisons with State-of-the-Art Works · p. 6

Ablation Study

  • Without deformation, mouths blur; without inpainting, texture remains vivid but upper-face details such as eye blinks and gaze can mismatch.Attention and dense-flow variants lose texture; the authors suggest their pixel-level deformations are less regularized than AdaAT’s region-level deformation. Table 2 reports DINet’s best visual quality by SSIM, PSNR, and LPIPS.Source: Ablation Study · p. 7, Table 2 · p. 7

User Study

  • In a 20-volunteer realism study, DINet receives 3.4, ahead of PC-AVS at 3.3, MakeitTalk at 3.1, ATVG at 2.9, and Wav2Lip at 2.4.Volunteers rated synthetic videos from 1 (pretty bad) to 5 (pretty good).Source: User Study · p. 7

Limitations

DINet remains constrained by challenging visual conditions, limited head poses, and reference-image selection.

Limitations

  • Frontal-view-only HDTF and MEAD-Neutral training restricts DINet to a limited range of head poses.Synthetic images can also be sensitive to which reference images are selected.Source: Limitations · p. 7
  • Changing lighting, dynamic backgrounds, earrings, flowing hair, and camera movement are challenging conditions for reference-image deformation.If the mouth region covers background, the method may generate artifacts outside the face.Source: Limitations · p. 7

Conclusion

DINet combines spatial deformation of reference-image feature maps with inpainting of the source mouth region. The paper reports that this approach preserves more texture than existing methods and is validated on high-resolution face dubbing.

Conclusion

  • DINet uses reference-feature deformation to align mouth shape with driving audio and head pose with the source image, then merges deformed and source features to inpaint the mouth.The network has separate deformation and inpainting parts.Source: Conclusion · p. 7
  • The combination of deformation and inpainting preserves more textural details than existing methods.Source: Conclusion · p. 7
  • Qualitative and quantitative experiments validate DINet on high-resolution face visually dubbing; future work will address the stated limitations.Source: Conclusion · p. 7

Related papers

Loading 2303.03988v1…