PaperlayerView exact revision on arXiv

Source-linked AI summary

AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis

Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, Juyong Zhang

arXiv:2103.11078v3cs.CV

TL;DR

Audio-to-face synthesis must produce natural, photo-realistic speaking video, while many methods rely on intermediate representations that can lose information. AD-NeRF directly maps audio features to dynamic neural radiance fields for head and torso synthesis, demonstrating high-fidelity results and support for arbitrary audio and head-pose manipulation.

  • Problem

    Audio-to-face synthesis must produce natural, photo-realistic speaking video, while many methods rely on intermediate face representations that can lose information.

    Source: §1 · p. 1

  • Method

    AD-NeRF directly maps audio features to dynamic neural radiance fields, using separate fields to model the head and torso.

    Source: §1 · pp. 1–2

  • Results

    The authors demonstrate high-fidelity audio-driven talking-head synthesis and support for arbitrary audio input and free head-pose manipulation.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The framework supports talking-head video editing through pose manipulation and background replacement.

    Source: §1 · p. 2

    For cross-identity audio, mouth parts can look unnatural when the training and driven languages differ.

    Source: §5 · p. 8

Abstract

from arXiv · show

Generating high-fidelity talking head video by fitting with the input audio sequence is a challenging problem that receives considerable attentions recently. In this paper, we address this problem with the aid of neural scene representation networks. Our method is completely different from existing methods that rely on intermediate representations like 2D landmarks or 3D face models to bridge the gap between audio input and video output. Specifically, the feature of input audio signal is directly fed into a conditional implicit function to generate a dynamic neural radiance field, from which a high-fidelity talking-head video corresponding to the audio signal is synthesized using volume rendering. Another advantage of our framework is that not only the head (with hair) region is synthesized as previous methods did, but also the upper body is generated via two individual neural radiance fields. Experimental results demonstrate that our novel framework can (1) produce high-fidelity and natural results, and (2) support free adjustment of audio signals, viewing directions, and background images. Code is available at https://github.com/YudongGuo/AD-NeRF.

1. Introduction

AD-NeRF maps audio features directly to dynamic neural radiance fields, bypassing intermediate face representations to synthesize talking-head video. It models the face and torso separately and supports pose and background editing.

Introduction

  • Prior audio-driven methods often use intermediate face representations that can cause semantic mismatches, and may render only mouths or fix head pose.Earlier approaches also relied on professional artist modeling or complicated motion-capture systems.Source: §1 · p. 1
  • The volumetric representation supports pose manipulation and background replacement, while representing fine-scale facial details such as teeth and hair.The authors report better image quality than existing GAN-based methods.Source: §1 · pp. 1–2
  • The method splits the neural radiance fields into separate face and torso branches because their movements differ, helping produce more natural talking-head sequences.The approach considers head pose and upper-body movement.Source: §1 · pp. 1–2
  • AD-NeRF directly maps audio features to dynamic neural radiance fields, avoiding intermediate representations that may lose information.The method uses audio features and face-parsing maps from a short target video to construct an audio-conditional implicit function.Source: §1 · pp. 1–2
  • Ablation studies show that direct mapping improves accurate lip motion with short-video training data, while the two-branch decomposition helps generate more natural results.Source: §1 · p. 2

2. Related Work

Related work spans audio- and video-driven facial animation and neural scene representations, including neural rendering methods for human heads and bodies. The paper contrasts its direct audio conditioning with prior audio-driven approaches that use intermediate 3D face or landmark representations.

Audio-driven Facial Animation

  • Audio-driven facial animation includes model-based audio-to-lip mapping and data-driven methods, which can generate photo-realistic talking heads.Model-based approaches require expertise to establish audio–lip relationships and are described as inconvenient for general applications beyond advanced digital creations.Source: §2 · p. 2
  • Data-driven methods progressed from still-image lip synthesis [46] to full-frame synthesis and 3D-shape [22, 43, 55] or landmark estimation.The 3D-shape and landmark approaches require more training data because they use latent modalities such as parametric models or low-dimensional landmarks.Source: §2 · p. 2

Video-driven Facial Animation

  • Video-driven facial animation transfers pose and expression from a source actor to a target, using performance capture and GAN-based texture synthesis [24] or expression-style analysis [23].Thies et al. track dynamic 3D faces with RGB-D cameras and later use RGB cameras alone.Source: §2 · p. 2

Implicit Neural Scene Networks

  • Neural scene representations encode scene geometry and appearance in networks sampled in space, while NeRF [30] converts implicit 3D representations into volumetric ray-sampling results.Follow-ups explore in-the-wild training and appearance interpolation [29], deformable fields for non-rigid motion [31] [36], and optimization without precomputed camera parameters [52].Source: §2 · p. 2

Neural Rendering for Human

  • Human neural rendering includes compositional dynamic upper-body fields, unseen-identity generalization, single-portrait estimation [17], dynamic face modeling [16], and sparse-view human-body synthesis [33].These works build on implicit neural scene representations [38] [20], with pixel-aligned features [38] used by Raj et al..Source: §2 · p. 2

3. Method

AD-NeRF maps semantic audio features to conditional neural radiance fields and uses volume rendering to synthesize talking heads. Separate head and torso fields handle their different motion, while audio, pose, and background can be edited at inference.

3.1. Overview

  • The framework trains two neural radiance fields from a target person’s short video and audio, using semantic audio features to condition dynamic head and upper-body representations.The framework renders the fields with volume rendering.Source: §3.1 · pp. 2–3, Figure 1 · p. 3

3.2. Neural Radiance Fields for Talking Heads

  • The conditional MLP takes semantic audio features, viewing direction, and 3D location as inputs and predicts color and density for radiance-field rendering.Audio features are derived from sixteen neighboring DeepSpeech frames; using them avoids an intermediate translation network for expression coefficients or landmarks.Source: §3.2 · p. 3

3.3. Volume Rendering with Radiance Fields

  • Volume rendering accumulates predicted color and density along camera rays, while face pose transforms samples to canonical space and coarse-to-fine sampling refines density estimates.Training uses head pose rather than 3D face shapes.Source: §3.3 · pp. 3–4

3.4. Individual NeRFs Representation

  • Separate head and torso radiance fields address their mismatched movements by transforming head samples to canonical space and conditioning the torso field on head pose.At inference, the head field is rendered first and the torso field fills the remaining body region.Source: §3.4 · p. 4

3.5. Editing of Talking Head Video

  • Replacing audio or adjusting pose coefficients enables audio-driven or pose-manipulated generation, and replacing the background image enables background editing.Using background pixels as the last ray samples helps separate foreground and background regions.Source: §3.5 · pp. 4–5

3.6. Training Details

  • Training uses static-camera, static-background videos at 25 fps and applies parsing, optical-flow-based face pose estimation, and background reconstruction before fitting.For each target person, the training video averages 3–5 minutes. A temporal filter smooths the audio condition using continuous audio codes and 1D convolutions with softmax activation.Source: §3.6 · p. 5

4. Experiments

Experiments assess the audio-conditioning and separate head-and-torso NeRF design, compare AD-NeRF with other talking-head methods, and demonstrate audio-, pose-, and background-editing capabilities. The reported results include improved lip synchronization and reconstruction quality, comparable naturalness to model-based methods, and support for varied audio and viewing directions.

4.1. Implementation Details

  • Each model is trained for 400k iterations with batches of 2048 rays; training two NeRFs for a five-minute 450 × 450 video takes about 36 hours, and rendering a frame takes 12 seconds.The implementation uses PyTorch [32], Adam [25], an initial learning rate of 0.0005, and RTX 3090 GPUs.Source: §4.1 · p. 5

4.2. Ablation Study

  • Direct audio conditioning improves lip synchronization over conditioning through audio-estimated expression codes.The comparison used monocular face tracking to optimize expression parameters and the network structure of Thies et al. [43] to estimate expression codes from audio.Source: §4.2 · p. 5, Figure 3 · p. 5
  • AD-NeRF achieves SSIM scores of 0.92 for our method, 0.88 for intermediate expression, and 0.87 for single NeRF on 500 test frames.Higher SSIM is better; separate head-and-torso fields avoid inaccurate boundary pixels caused when head-pose matrices transform the torso, neck, and shoulders.Source: Figure 4 · p. 5, §4.2 · p. 6

4.3. Evaluations

  • Compared with image-based methods [46], AD-NeRF generates full-size images with backgrounds and natural talking styles, while those methods are limited by input image size.The paper also characterizes image-based methods as producing still face-crop images.Source: §4.3 · p. 6, Figure 6 · p. 6
  • Despite using only 3–5 minutes of training video versus over 3 hours for Thies et al. [43] and 14 hours for Suwajanakorn et al., AD-NeRF produces comparable natural results and participant ratings.The user study rated synchronization, image fidelity, and realism; the paper reports comparable scores for AD-NeRF and the two model-based methods.Source: §4.3 · pp. 6–8, Figure 7 · p. 7
  • AD-NeRF and the video-driven method of Kim et al. [23] both produce high-fidelity talking-head results, using audio and portrait-video frames as their respective inputs.Source: §4.3 · p. 7, Figure 8 · p. 7
  • The evaluation combines SyncNet audio-visual synchronization confidence, lower-face and mouth Action Unit error, and participant ratings of synchronization, fidelity, and realism.Thirty attendees rated 100 clips on a 1–10 scale; the study used template-video head poses outside AD-NeRF’s training set.Source: §4.3 · pp. 7–8

4.4. Applications on Talking Head Editing

  • AD-NeRF generates reasonable results from arbitrary audio across identities, genders, and languages, and supports viewing-direction adjustment and background replacement.The paper presents these editing capabilities as enabled by the trained neural radiance fields.Source: Figure 9 · p. 7, §4.4 · p. 8, Figure 10 · p. 8

5. Limitation

AD-NeRF’s cross-identity results can have unnatural-looking mouth parts, while synthesized torsos can appear blurry because the available inputs do not fully determine torso movement.

5. Limitation

  • In cross-identity audio-driven results, mouth parts sometimes look unnatural because training and driven language are inconsistent.This limitation is reported for the cross-identity setting.Source: §5 · p. 8
  • Synthesized torso parts sometimes look blurry because head pose and audio features cannot fully determine the actual torso movement.The paper notes this in Fig. 5 and the supplemental video.Source: §5 · p. 8

6. Conclusion

AD-NeRF uses two neural radiance fields and volume rendering to synthesize a human head and upper body directly from audio, without intermediate representations. Its trained model supports audio from different identities, genders, and languages, as well as free head-pose manipulation.

6. Conclusion

  • Two elaborately designed NeRFs with volume rendering directly synthesize the human head and upper body from audio without intermediate representations.Source: §6 · p. 8
  • The trained model supports arbitrary audio from different identities, genders, and languages, plus free head-pose manipulation.Source: §6 · p. 8
  • The method is designed for high-fidelity talking-head synthesis.Source: §6 · p. 8

Related papers

Loading 2103.11078v3…