Source-linked AI summary
Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decomposition
Jiaxiang Tang, Kaisiyuan Wang, Hang Zhou, Xiaokang Chen, Dongliang He, Tianshu Hu, Jingtuo Liu, Gang Zeng, Jingdong Wang
TL;DR
Audio-driven NeRF portraits can be realistic, but their slow training and inference hinder practical use. RAD-NeRF separates portrait features into low-dimensional grids for head and torso modeling, and reports inference 500 times faster than previous works with better rendering quality.
Problem
Audio-driven NeRF portrait methods can render realistic results, but slow video production and training that takes days per person limit their practical use.
Source: §1 · p. 1Method
RAD-NeRF uses separate low-dimensional grids for audio and spatial head features and a lightweight Pseudo-3D Deformable Module for torso modeling.
Source: §1 · p. 2Results
500 times faster than previous works with better rendering quality.
Source: §6 · p. 9Takeaways & Limitations
Experiments and user studies verify enhanced talking-portrait synthesis quality.
Source: §6 · p. 9NeRF-based talking-portrait methods require accurate face semantic parsing to separate the head and torso.
Source: §5 · p. 8
Abstract
from arXiv · showhide
While dynamic Neural Radiance Fields (NeRF) have shown success in high-fidelity 3D modeling of talking portraits, the slow training and inference speed severely obstruct their potential usage. In this paper, we propose an efficient NeRF-based framework that enables real-time synthesizing of talking portraits and faster convergence by leveraging the recent success of grid-based NeRF. Our key insight is to decompose the inherently high-dimensional talking portrait representation into three low-dimensional feature grids. Specifically, a Decomposed Audio-spatial Encoding Module models the dynamic head with a 3D spatial grid and a 2D audio grid. The torso is handled with another 2D grid in a lightweight Pseudo-3D Deformable Module. Both modules focus on efficiency under the premise of good rendering quality. Extensive experiments demonstrate that our method can generate realistic and audio-lips synchronized talking portrait videos, while also being highly efficient compared to previous methods.
1. Introduction
Existing audio-driven NeRF portraits can render realistically, but slow training and inference constrain their practical use. RAD-NeRF decomposes head and torso representations into efficient grid-based modules and reports faster rendering with better quality.
1. Introduction
- Audio-conditioned NeRF portraits offer realistic rendering, but AD-NeRF takes 12 seconds to render a 450 × 450 frame on an RTX 3090 and training takes days per person.The paper identifies this speed as far from real-time use at 25 FPS.Source: §1 · p. 1
- Grid-based NeRF acceleration methods for static scenes do not directly apply to dynamic portraits, where adding audio dimensions to interpolation grows computational complexity exponentially.The paper also identifies efficient torso modeling as a separate challenge because prior approaches use a full 3D field or entangled deformation field.Source: §1 · pp. 1–2
- RAD-NeRF decomposes the high-dimensional portrait representation into three low-dimensional trainable grids, separating audio and spatial representations for dynamic head modeling.The audio and spatial coordinates use separate lower-dimensional grids to reduce interpolation cost.Source: §1 · p. 2
- A lightweight Pseudo-3D Deformable Module models the torso with a 2D feature grid and targets natural motion synchronized with the head.The design is motivated by less involved topological changes in torso motion.Source: §1 · p. 2
- 500× faster than previous works with better rendering quality; the framework also supports explicit head-pose, eye-blink, and background-image controls.The reported result is presented as a comparison with previous works; no specific baseline or benchmark values are supplied here.Source: §1 · p. 2
2. Related Work
Prior talking-portrait approaches include image-based synthesis, structural face representations, and NeRFs, each with limitations in control, error accumulation, or speed. Grid-based NeRF methods accelerate static scenes, while dynamic and audio-driven portraits require adaptations for their motion and conditioning.
2.1. Audio-driven Talking Portrait Synthesis
- Image-based methods synthesize audio-corresponding images but have fixed resolution and cannot control head pose; structural representations can introduce estimation errors.Earlier methods also include phoneme-mouth rules and stitching-based techniques.Source: §2.1 · p. 2
- NeRF-based portrait methods can render photorealistically at arbitrary resolution with less training data, but current audio-driven methods remain slow to train and infer.The related work names NeRFace and audio-conditioned NeRF approaches.Source: §2.1 · p. 2
2.2. Neural Radiance Field
- Grid accelerations store scene features explicitly and reduce MLP computation, but the described grid-based methods target static scenes.Examples include dense feature grids, multi-resolution hash tables, and compact low-rank tensor components.Source: §2.2 · p. 2
- Existing dynamic-NeRF acceleration methods are deformation-based or support only time-dependent dynamics, whereas this work is designed for audio-driven portrait synthesis.The architecture caption shows separate audio and spatial grid encoders for the head, and a pose-dependent torso deformation path.Source: §2.2 · p. 3, Figure 1 · p. 3
3. Method
RAD-NeRF decomposes audio-driven portrait modeling into grid-based head and torso modules, using modulation for the head and deformation for the torso. It combines these representations with portrait-specific controls and training procedures.
3.1. Preliminaries
- The head module selects modulation for topology-changing facial motion, while grid encoders retrieve spatial features through interpolation rather than costly MLP forwarding.A small MLP combines spatial, audio, eye, and appearance features to produce density and color.Source: §3.1 · p. 4, §3.2 · p. 4
3.2. Decomposed Audio-spatial Encoding Module
- Separate 3D spatial and low-dimensional audio grids avoid a higher-dimensional concatenated grid, after audio is compressed to a spatial-dependent coordinate.The audio coordinate has dimension D ∈ [1, 2, 3].Source: §3.2 · p. 4
- The head MLP uses spatial, audio, eye, and appearance features; the landmark-derived eye ratio lets users adjust eye blinks at test time.The eye ratio is typically 0% to 0.5%.Source: §3.2 · p. 4
3.3. Pseudo-3D Deformable Module
- For the nearly static torso, a head-pose-conditioned deformation moves one image-space coordinate before a 2D grid lookup; head and torso images can be composited over any background.The paper reports that this lightweight representation models torso dynamics and synthesizes torso images matching the head.Source: §3.3 · p. 5
3.4. Training Details
- Training maintains one maximum occupancy grid across audio conditions, regularizes audio coordinates outside the face, and fine-tunes landmark-selected lip patches with LPIPS and MSE.The maximum grid is built by retaining maximum occupancy values from randomly sampled training audio conditions.Source: §3.4 · p. 5
4. Experiment
The experiments report faster training and inference alongside strong portrait quality, then examine evaluation settings, explicit controls, and design choices through ablations.
4.1. Implementation Details
- Head training takes around 5 hours and torso training around 2 hours on one NVIDIA V100 GPU.Training uses 20,000 steps for each part, with torso training following head convergence.Source: §4.1 · pp. 5–6
4.2. Quantitative Evaluation
- Self-driven tests use same-identity ground truth with PSNR, LPIPS, and lip-landmark distance; cross-driven tests use SyncNet confidence and Action Units Error without same-identity ground truth.The cross-driven measures assess audio-lip synchronization and lip-related muscle-action consistency.Source: §4.2 · p. 6
- 500× faster than AD-NeRF, with about 5× faster convergence; self-driven quality improves on most metrics and cross-driven performance is comparable to recent methods.The comparison covers self-driven reconstruction and cross-driven synchronization and lip-movement consistency.Source: §4.2 · p. 6, Table 1 · p. 6, Table 2 · p. 7
4.3. Qualitative Evaluation
- Self-driven visuals show sharper, more accurate lips than AD-NeRF; cross-driven lips remain robust during large mouth movements and quality matches SynObama despite less training video.The reported comparison uses five minutes of training video versus 14 hours for SynObama.Source: Figure 3 · p. 6, §4.3 · p. 7, Figure 4 · p. 7
- In a 20-attendee study of 36 cross-driven videos, ratings across synchronization, image quality, and realness further support portrait quality.Each aspect was rated on a 1–5 scale, with higher ratings better.Source: §4.3 · p. 7, Table 3 · p. 8
- Beyond lips, the method supports explicit control of eyes, head poses, and background images.The qualitative evaluation also illustrates eye closure and left/right views.Source: Figure 5 · p. 7, §4.3 · p. 7
4.4. Ablation Study
- A 2D audio coordinate best suits the setting: smaller dimensions speed inference but can harm quality, while larger ones slow inference and hinder convergence.The ablation compares inference speed and image quality across audio coordinate dimensions.Source: §4.4 · p. 7, Table 4 · p. 8
- The decomposed audio-spatial grid gives the best performance-speed balance; implicit backbones blur depth, while deformation struggles with lip-topology changes.The ablation compares implicit, composed-grid, and decomposed-grid backbones.Source: §4.4 · pp. 7–8, Figure 6 · p. 8
5. Broader Impact
The paper notes potential misuse of realistic portrait synthesis and identifies constraints involving hair, head pose, and language.
5. Broader Impact
- Realistic talking portraits made from a short video of a specific person could be misused; the authors commit to sharing generated videos to support deepfake detection.They frame appropriate use as supporting digital human technology.Source: §5 · p. 8
- Accurate face semantic parsing is needed to separate head and torso, and current methods fail on shoulder-length hair except in a static-head-pose setting.Source: §5 · p. 8
- English ASR features may make lips less accurate for other languages; the authors suggest audio-to-phone models as a possible remedy.Source: §5 · p. 8
6. Conclusion
The conclusion presents a real-time audio-driven portrait method with separate modules for head and torso modeling. It reports inference 500 times faster than previous works with better rendering quality.
6. Conclusion
- The method combines a decomposed audio-spatial encoding module for head modeling with a pseudo-3D deformable module for torso modeling.Together, the modules enable efficient inference for audio-driven neural talking portraits.Source: §6 · pp. 8–9
- Inference is 500 times faster than previous works, with better rendering quality.The conclusion also notes experiments and user studies to verify synthesis quality.Source: §6 · p. 9
A.1. Data Processing
The data-processing description covers audio-feature extraction and preparation of torso training images. Audio features are derived from ASR classification logits and temporally smoothed.
A.1. Data Processing
- The Audio Feature Extractor uses Deep-Speech to produce classification logits for each 20 ms audio clip, with Wav2Vec also reported to yield similar performance.The logits are filtered by an audio-attention module for temporal smoothing to produce final audio condition codes.Source: §A.1 · p. 11
- Audio slices are processed with a sliding window, then 1D convolutions and self-attention temporally smooth their predicted labels into final audio features.The figure caption describes the prediction as classification logits from an off-the-shelf ASR model.Source: Figure 7 · p. 11
- For torso training, the neck is in-painted into the torso image by replicating segmented neck pixels and exponentially darkening them to simulate shadow.The authors say this helps avoid head-torso disconnections, especially in cross-driven settings.Source: §A.1 · p. 11
A.2. Additional Implementation Details
The implementation details specify grid and MLP settings, audio-feature smoothing at test time, and a qualitative comparison figure with marked image-quality and lip-accuracy regions.
A.2. Additional Implementation Details
- The spatial grid has 16 levels from 16^3 to 2048^3 with two channels per level, while the audio and torso grids use matching hyperparameters with 2D input.Audio condition codes have dimension 64; the audio and density MLPs have three 64-hidden-unit layers, and the color MLP has two.Source: §A.2 · p. 11
- Figure 8 compares SynObama, SSP-NeRF, and the proposed method; yellow boxes mark low image quality and red boxes mark inaccurate lips.The supplied figure labels identify the compared methods.Source: Figure 8 · p. 12, §A.2 · p. 12
- At test time, momentum is applied to audio features to reduce lip-motion jitters and produce more natural videos.The smoothing hyperparameter β controls the update between the previous smoothed feature and the current audio feature.Source: §A.2 · p. 12
B. Additional Experimental Results
The additional qualitative comparison reports sharper, more accurate lips than SSP-NeRF, using its best-quality demonstration video because its code was unavailable for reproduction.
B. Additional Experimental Results
- Against SSP-NeRF, the method produces sharper, more accurate lips in a qualitative comparison driven by audio corresponding to SSP-NeRF’s cropped best-quality demonstration video.The authors note that SSP-NeRF code was unavailable for reproducing its results.Source: B. Additional Experimental Results · p. 12