Source-linked AI summary
AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis
Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, Juyong Zhang
TL;DR
High-fidelity audio-driven talking-head synthesis must connect audio to realistic facial and upper-body motion without losing information through intermediate representations. AD-NeRF directly conditions dynamic neural radiance fields on audio and renders separate head and torso fields, producing high-fidelity results while supporting audio, pose, and background editing. The method nevertheless has cross-identity mouth artifacts and occasional torso blurring in settings where available conditioning does not fully determine motion.
Problem
Audio-driven talking-head synthesis must relate audio signals to natural, synchronized, photorealistic facial deformations, while intermediate representations can cause information loss and semantic mismatches.
Method
AD-NeRF directly feeds extracted audio features into an audio-conditional implicit function that represents dynamic neural radiance fields, using separate fields for the head and torso.
Results
AD-NeRF produces high-fidelity and natural talking-head results while supporting free adjustment of audio signals, viewing directions, and background images.
Takeaways & Limitations
The neural-radiance-field representation enables direct audio-to-video synthesis together with head-pose manipulation and background replacement.
Takeaways & Limitations
Cross-identity audio can produce unnatural mouth regions, and torso parts can appear blurry when head pose and audio features do not fully determine torso movement.
Abstract
from arXiv · showhide
Generating high-fidelity talking head video by fitting with the input audio sequence is a challenging problem that receives considerable attentions recently. In this paper, we address this problem with the aid of neural scene representation networks. Our method is completely different from existing methods that rely on intermediate representations like 2D landmarks or 3D face models to bridge the gap between audio input and video output. Specifically, the feature of input audio signal is directly fed into a conditional implicit function to generate a dynamic neural radiance field, from which a high-fidelity talking-head video corresponding to the audio signal is synthesized using volume rendering. Another advantage of our framework is that not only the head (with hair) region is synthesized as previous methods did, but also the upper body is generated via two individual neural radiance fields. Experimental results demonstrate that our novel framework can (1) produce high-fidelity and natural results, and (2) support free adjustment of audio signals, viewing directions, and background images. Code is available at https://github.com/YudongGuo/AD-NeRF.
1. Introduction
AD-NeRF addresses audio-driven talking-head synthesis by directly mapping audio features to dynamic neural radiance fields, avoiding intermediate face representations. It models head and torso separately to produce natural, high-fidelity results and support video editing.
- Motivation: Audio-driven facial video synthesis is challenging because it must connect speech with natural facial motion while preserving synchronized, photorealistic streaming.The task is motivated by applications including digital humans, chatting robots, and virtual video conferences.
- Problem: Existing methods often use 3D face shapes, expression coefficients, or 2D landmarks, which can lose information and cause mismatches between audio and facial deformation.The paper identifies faithful correspondence between audio, expressions, and lip motions as a central difficulty.
- Method: AD-NeRF directly maps audio features to dynamic neural radiance fields without intermediate modalities, enabling neural rendering of fine-scale components such as teeth and hair.The method uses an audio-conditional implicit function to represent the target dynamic subject.
- Method: The framework decomposes portrait scenes into separate head and torso neural radiance fields because head and upper-body movements are not consistent.This design supports more natural talking-head sequences and separates deformation modeling for the two regions.
- Applications: AD-NeRF supports pose manipulation and background replacement in talking-head videos, extending the framework beyond synthesis to editing applications.The contribution statement identifies these editing capabilities as valuable for potential virtual-reality applications.
2. Related Work
Prior audio-driven facial animation methods include model-based systems requiring expert-designed audio-to-lip-motion relationships and data-driven systems using latent intermediate modalities. AD-NeRF is situated among neural scene representation approaches that model scene shape and appearance with neural networks.
- Audio-driven Facial Animation: Model-based facial animation methods require expertise to establish relationships between audio semantics and lip motions, limiting their convenience for general applications.The paper gives phoneme-viseme mapping as an example of such expert-designed relationships.
- Audio-driven Facial Animation: Data-driven audio-driven methods commonly rely on latent intermediate modalities such as parametric facial models or low-dimensional landmarks.The paper contrasts these approaches with AD-NeRF and notes their greater training-data requirements.
- Video-driven Facial Animation: Video-driven facial animation transfers facial pose and expression from a source actor to a target, often using model-based facial performance capture.Prior work includes RGB-D and RGB camera systems, GAN-based skin-texture synthesis, and analyses of expression style.
- Implicit Neural Scene Networks: Neural scene representation uses neural networks to represent the shape and appearance of scenes, providing the broader representation paradigm for AD-NeRF.The related-work discussion introduces neural scene representation networks before discussing implicit neural scene methods.
3. Method
AD-NeRF learns audio-conditioned neural radiance fields from short portrait videos, separately modeling the head and torso before rendering them volumetrically. The framework uses pose and audio conditions to synthesize and edit talking-head videos, including changes to audio, viewing direction, and background.
- Overview: The framework trains two neural radiance fields from a short portrait video and audio track, rendering the head and upper body with volume rendering.Head-NeRF and Torso-NeRF separately reconstruct the two regions.
- Audio-conditional radiance fields: AD-NeRF maps semantic audio features directly to a conditional implicit radiance field, avoiding intermediate face representations such as landmarks or expression coefficients.The implicit function receives audio features alongside spatial location and viewing direction.
- Volume rendering: Volume rendering accumulates densities and RGB values sampled along camera rays to compute each rendered pixel’s color.The implicit model predicts color and density, while accumulated transmittance weights their contribution along each ray.
- Two-branch representation: The head model uses estimated rigid face pose to transform sampled points into canonical space, while the torso model conditions on pose without explicitly transforming torso points.Pose is represented by rotation and translation parameters and is used differently in the two branches.
- Two-stage training: Face parsing divides each training image into static background, head, and torso regions, enabling sequential training of Head-NeRF and Torso-NeRF.The rendered head becomes the new background before the torso branch is trained and later fills the missing body region during inference.
4. Experiments
The experiments evaluate AD-NeRF through ablations, comparisons with image-, model-, and video-driven methods, quantitative metrics, user ratings, and editing demonstrations. Results support direct audio conditioning, separate head/torso fields, comparable naturalness, and flexible manipulation of audio, pose, and backgrounds.
- Implementation Details: 36 hours are required to train two NeRFs on a 5-minute 450 × 450 video, while rendering takes 12 seconds per frame.Both networks use Adam with an initial learning rate of 0.0005 and are trained for 400k iterations.
- Ablation Study: Direct audio conditioning produces more accurate lip motion than conditioning through an intermediate facial-expression representation.The ablation compares rendering from audio codes with audio-estimated expression codes.
- Ablation Study: Separate neural radiance fields for the head and torso improve reconstruction quality by avoiding boundary mismatches caused by transforming the torso with head-pose matrices.A single NeRF produces inaccurate pixels around the upper-body boundary.
- Ablation Study: SSIM scores on 500 test frames are 0.92 for AD-NeRF, 0.88 with intermediate expression, and 0.87 with a single NeRF.Higher SSIM is better.
- Evaluations: Compared with image-based methods, AD-NeRF generates full-size images with backgrounds, while image size limits their generation quality.The study also compares AD-NeRF with model-based and video-driven methods using synchronization, facial-action, and user-study evaluations.
- Applications on Talking Head Editing: AD-NeRF achieves comparable natural results to two model-based methods despite requiring only a short training video, and supports audio, pose, and background editing.The method accepts audio from different identities, genders, and languages, and permits viewing-direction and background replacement adjustments within the training range.
5. Limitation
AD-NeRF has two reported limitations: cross-identity driving can produce unnatural mouth regions, and torso motion can appear blurry.
- Cross-identity audio-driven results sometimes produce unnatural mouth parts when training and driven languages are inconsistent.
- Torso regions sometimes appear blurry because head pose and audio features cannot fully determine actual torso movement.
6. Conclusion
The paper presents a neural-radiance-field method for high-fidelity talking-head synthesis that directly generates the head and upper body from audio. It uses two NeRFs with volume rendering and supports arbitrary audio inputs and head-pose manipulation.
- AD-NeRF performs high-fidelity talking-head synthesis using neural radiance fields.
- Two individually designed NeRFs and volume rendering directly synthesize the human head and upper body from audio without intermediate representations.
- The trained model accepts arbitrary audio from different identities, genders, and languages, while supporting free head-pose manipulation.