Source-linked AI summary
Efficient Region-Aware Neural Radiance Fields for High-Fidelity Talking Portrait Synthesis
Jiahe Li, Jiawei Zhang, Xiao Bai, Jun Zhou, Lin Gu
TL;DR
ER-NeRF targets efficient, high-fidelity talking portrait synthesis, where hash collisions challenge dynamic-head reconstruction and spatial regions relate differently to speech. It combines tri-plane hash representation with region-aware audio attention and reports improved realistic synthesis with higher efficiency.
Problem
Hash collisions complicate dynamic talking-head reconstruction, while spatial regions have different relationships to speech audio and local motion.
Source: §1 · p. 2, §3.2 · p. 4Method
ER-NeRF combines a tri-plane hash representation that prunes spatial regions with cross-modal attention that adapts audio features to spatial regions.
Source: §1 · p. 2Results
The authors report significant improvement in realistic talking portrait synthesis with higher efficiency.
Source: §6 · p. 8Takeaways & Limitations
The proposed tri-plane representation is reported to support high-quality rendering, real-time inference, fast convergence, and compact model size.
Source: §1 · p. 2With a small single-video training set, lip-audio synchronization remains weak for out-of-domain audio such as cross-lingual speech or singing.
Source: §A.10 · p. 14
Abstract
from arXiv · showhide
This paper presents ER-NeRF, a novel conditional Neural Radiance Fields (NeRF) based architecture for talking portrait synthesis that can concurrently achieve fast convergence, real-time rendering, and state-of-the-art performance with small model size. Our idea is to explicitly exploit the unequal contribution of spatial regions to guide talking portrait modeling. Specifically, to improve the accuracy of dynamic head reconstruction, a compact and expressive NeRF-based Tri-Plane Hash Representation is introduced by pruning empty spatial regions with three planar hash encoders. For speech audio, we propose a Region Attention Module to generate region-aware condition feature via an attention mechanism. Different from existing methods that utilize an MLP-based encoder to learn the cross-modal relation implicitly, the attention mechanism builds an explicit connection between audio features and spatial regions to capture the priors of local motions. Moreover, a direct and fast Adaptive Pose Encoding is introduced to optimize the head-torso separation problem by mapping the complex transformation of the head pose into spatial coordinates. Extensive experiments demonstrate that our method renders better high-fidelity and audio-lips synchronized talking portrait videos, with realistic details and high efficiency compared to previous methods.
1. Introduction
ER-NeRF targets talking-portrait synthesis that is both high-fidelity and efficient, addressing slow NeRF inference and limitations in regional audio-motion modeling. It combines region-aware spatial representation, audio attention, and pose encoding; the paper reports realistic synthesis with high efficiency and improved objective and human-study results.
1. Introduction
- NeRF-based portrait methods offer high synthesis quality, but vanilla methods are too slow for real-time use, while RAD-NeRF's complex audio-motion grid encoder limits convergence and reconstruction quality.RAD-NeRF uses Instant-NGP for fast inference but learns regional audio-motion mapping implicitly.Source: §1 · p. 1
- Tri-Plane Hash Representation factorizes 3D space into three orthogonal planes and prunes feature grids, reducing hash collisions to improve dynamic head reconstruction.The paper links fewer hash-collision noises to more accurate head structures and finer dynamic motions.Source: §1 · p. 2
- Region Attention Module uses cross-modal attention to adapt audio features to spatial regions, supporting accurate facial motion while leaving static portions unaffected by changing signals.This replaces concatenating raw features and learning audiovisual correlation through a large MLP-based encoder.Source: §1 · p. 2
- Adaptive Pose Encoding maps complex pose transformations onto spatial coordinates to give torso-NeRF a clearer positional relation and address head-torso separation.The encoding lets torso-NeRF learn its pose implicitly.Source: §1 · p. 2
- ER-NeRF renders realistic talking portraits with high efficiency and outperforms state-of-the-art methods on objective evaluation and human studies.The framework aims for high-quality rendering, fast convergence, real-time inference, and a compact model size.Source: §1 · p. 2
2. Related Work
Prior talking-portrait work spans 2D synthesis and NeRF-based approaches, with trade-offs in pose structure, inference speed, and model complexity. ER-NeRF combines efficient spatial representations with region-aware conditioning for talking-portrait synthesis.
2D-Based Talking Portrait Synthesis
- 2D portrait methods range from phoneme-mouth rules and image-conditioned lip synthesis to landmark, 3D-model, and diffusion approaches, with trade-offs in intermediate-representation errors, inference speed, and head-pose control.Without an explicit 3D structure representation, 2D methods have drawbacks in naturalness and consistency of head-pose control.Source: §2 · p. 2
NeRF-based Talking Portrait Synthesis
- Earlier NeRF portrait methods use slow, memory-costly vanilla renderers; SSP-NeRF models local motion, while RAD-NeRF improves visual quality and efficiency using Instant-NGP but retains a complex audio-processing module.End-to-end methods use large MLP encoders to learn audio-region connections, increasing complexity and training difficulty.Source: §2 · p. 2
ER-NeRF framework
- ER-NeRF's overview separates head rendering through tri-plane geometry and region-weighted audio and eye-blink features from torso rendering through Adaptive Pose Encoding.The head decoder combines region-aware features with geometry and view direction to predict color and density.Source: Figure 2 · p. 3
Efficient Neural Representation
- Efficient neural representations balance speed and memory with sparse feature grids, while plane-based methods factorize space and hash grids store sparse multiresolution details.Most such methods target static scenes, and existing dynamic methods described here are unsuitable for audio-driven talking portraits.Source: §2 · p. 3
3. Method
ER-NeRF combines a person-specific NeRF setup with tri-plane hash encoding, region-aware conditioning, and adaptive pose encoding. Its training also uses coarse-to-fine optimization to refine image details.
3.1. Preliminaries and Problem Setting
- Training uses a few minutes of front-view video from one person, with head and torso trained and rendered separately.Head poses provide camera parameters; audio features come from pretrained DeepSpeech, and semantic parsing separates the head, torso, and background.Source: §3.1 · pp. 3–4
3.2. Tri-Plane Hash Representation
- Tri-Plane Hash Representation factorizes 3D space into three orthogonal 2D hash grids, reducing collisions that burden the decoder during dynamic talking-head reconstruction.For near-frontal query rays, the paper estimates a 5× collision reduction at N = 16 and R ≈ 256, with the same model size.Source: §3.2 · p. 4
3.3. Region Attention Module
- Region Attention Module uses spatial geometry features to reweight audio channels, emphasizing audio in dynamic regions while suppressing it in static regions.The module combines external attention across multiresolution features with channel attention; eye-blink conditioning uses a separate scalar attention vector.Source: §3.3 · pp. 4–5
3.4. Training Details
- Adaptive Pose Encoding transforms trainable canonical-space key points by the head pose, then projects them to image coordinates that condition torso-NeRF.The method uses three key points to provide clearer positional information for implicit torso-pose learning.Source: §3.4 · p. 5
- Training first uses MSE for coarse image fitting, then fine-tunes sampled patches with weighted LPIPS to improve sharp details.The paper describes this as a two-stage coarse-to-fine process.Source: §3.4 · p. 5
4. Experiments
Across head reconstruction and unseen-audio lip-synchronization settings, ER-NeRF performs strongly in visual quality, synchronization, and efficiency. Qualitative comparisons and ablations examine portrait details, pose robustness, and its representation and attention components.
4.1. Experimental Settings
- Experiments use four high-definition clips averaging about 6,500 frames at 25 FPS; evaluation covers head reconstruction and unseen-audio lip synchronization.The reported metrics include PSNR, LPIPS, FID, landmark distance, SyncNet confidence, and action-units error.Source: §4.1 · p. 6, §4.2 · p. 6
4.3. Qualitative Evaluation
- Figure 5 compares key frames and portrait details; ER-NeRF renders more detail and personalized lip synchronization, with more robust head–torso alignment than the compared NeRF methods.The comparison includes torso renders for NeRF-based methods.Source: Figure 5 · p. 7, §4.3 · pp. 7–8
4.2. Quantitative Evaluation
- ER-NeRF achieves the best visual reconstruction and lip synchronization in head reconstruction, performs best on three reported efficiency metrics, and exceeds some baselines in lip-sync generalization.It has the highest Sync score among NeRF-based methods in the lip-synchronization setting.Source: Table 2 · p. 6, §4.2 · p. 7
4.3. Qualitative Evaluation
- On out-of-range poses, ER-NeRF retains image quality and structural accuracy; in the user study, it leads in lip-sync accuracy and realness, while SynObama scores higher on image quality.The user study sampled 28 clips and involved 18 attendees.Source: Table 3 · p. 7, Figure 6 · p. 8, §4.3 · p. 8
4.4. Ablation Study
- Ablations report that tri-hash improves image quality and lip synchronization, while region attention improves motion modeling; channel-wise attention outperforms feature-wise attention.Region attention with existing backbones also scores better on image quality and lip synchronization than current state-of-the-art methods, with half the training time and fewer parameters.Source: §4.4 · p. 8, Table 4 · p. 8, Table 5 · p. 8
5. Ethical Consideration
The authors note potential malicious use of ER-NeRF and plan to share generated results to support development of stronger deepfake detectors.
5. Ethical Consideration
- The authors identify potential malicious use and plan to share generated results to help develop stronger deepfake detectors.They frame responsible use as supporting machine-learning research and the digital industry.Source: §5 · p. 8
6. Conclusion
The conclusion presents ER-NeRF as an efficient framework for high-quality talking portrait synthesis, centered on tri-plane hash representation and region attention.
6. Conclusion
- ER-NeRF combines Tri-Plane Hash Representation and a Region Attention Module for efficient, high-quality talking portrait synthesis.The authors report significant improvement in realistic talking portrait synthesis with higher efficiency.Source: §6 · p. 8
A.1. Overview
The supplemental document contains implementation details, additional experiments, ethical considerations, and discussion.
A.1. Overview
- The supplement covers torso-NeRF with Adaptive Head Encoding, architecture and user-study details, additional experiments and analysis, ethical considerations, and discussion.Source: §A.1 · p. 12
A.2. Torso-NeRF Details
Adaptive Pose Encoding maps head-pose transformations into torso-conditioning coordinates, which drive a 2D deformable field for torso rendering. The user study reports the best torso stability and image-quality ratings for ER-NeRF.
A.2. Torso-NeRF Details
- Trainable canonical-space keypoints are transformed by the head pose, converted to ordinary coordinates, and projected onto Z = 1.The network learns the coefficient γ used in the coordinate conversion.Source: §A.2 · p. 12
- The resulting keypoint coordinates condition a 2D deformable field that predicts pixel-wise torso color and transparency.An MLP predicts a coordinate offset, a multiresolution hash encoder encodes the deformed pixel coordinate, and another MLP predicts color and alpha.Source: §A.2 · p. 12
- Training optimizes the keypoints to represent the implicit relationship between head and torso poses, using only linear transformations during forwarding.The passage states that torso quality improves without a significant increase in computation.Source: §A.2 · p. 12
- ER-NeRF achieves the best torso Stability and Image Quality ratings in a user study scored from 1 to 5, with higher scores better.The encoding is added without a deep neural network, according to the study description.Source: §A.2 · p. 12, Table 6 · p. 12
A.3. Architecture Details
The architecture uses pretrained DeepSpeech features and compact MLPs for audio and eye-blinking attention. The tri-plane hash encoders and their decoders are configured with specified resolutions and layer widths.
A.3. Architecture Details
- Raw audio features come from pretrained DeepSpeech and pass through the existing NeRF audio-attention processing, with output dimension set to 32.The feature processing follows prior NeRF-based works, except for changing the output dimension from 64 to 32.Source: §A.3 · p. 12
- The speech-audio attention vector uses a two-layer MLP with 64 hidden dimensions, while the eye-blinking branch uses two layers with 16 hidden dimensions.Source: §A.3 · p. 12
- Each 2D hash encoder has 14 resolution levels spanning 64 to 512; the density and color MLP decoders use three and two 64-wide layers, respectively.One entry is assigned to each hash-encoder level.Source: §A.3 · p. 12
A.4. User Study Details
The user study evaluates generated portraits with ratings from 18 participants aged 20–30. Participants compare anonymized results together in a high-resolution video and assess both portrait and torso qualities.
A.4. User Study Details
- The study involved 18 participants aged 20–30, who viewed generated videos and ground truth together in one high-resolution video.Source: §A.4 · p. 12
- Results were assigned numbers rather than method names to support fair comparisons.Source: §A.4 · p. 12
- Participants rated lip-sync accuracy, video realness, and image quality; torso evaluations additionally covered stability and image quality.Source: §A.4 · p. 12
A.5. Tri-Plane Hash Representation
The additional material reports qualitative portrait comparisons and experiments on hash collisions, LPIPS finetuning, and eye-blinking motion. It also discusses comparison caveats, ethical risks, and limitations involving per-scene training, out-of-domain audio, and torso blur.
A.5. Tri-Plane Hash Representation
- For Tri-Hash Representation, frontal-plane collisions are O(R^2), and side-plane collisions are O(λR^2 + RN), giving overall complexity O(R^2 + 2RN).At some side areas, only the nearest N points can be sampled due to occlusion; λ is an adjustment.Source: §A.5 · p. 13
A.6. Additional Experiments
- LPIPS finetuning has a stronger impact on ER-NeRF’s high-frequency details than on RAD-NeRF, whose more complex architecture is harder to optimize.The comparison attributes the training-difficulty difference to RAD-NeRF’s additional hash grid and higher-dimensional audio features; Figure 8 shows validation LPIPS losses on the Obama and May datasets.Source: §A.6 · p. 13, Figure 8 · p. 13
- Directly concatenating AU45 without region attention produces unnatural blinking-related movements, while the attention mechanism improves robustness.The reported artifacts include jittering and unreasonable lip movements accompanying eye blinking.Source: §A.6 · p. 13, Figure 9 · p. 14
A.7. Comparison with GeneFace and DFRF
- The GeneFace and DFRF comparisons are presented for reference, not as entirely fair comparisons, because the methods target different settings.GeneFace was omitted from the main-paper baselines because its code was released too close to the submission deadline.Source: §A.7 · pp. 13–14, Table 8 · p. 14, Table 9 · p. 14
A.9. Ethics Considerations
- The paper identifies high-fidelity talking portraits as a misuse risk and proposes video watermarking, detection research, and limits on deepfake use.It notes that distinguishing high-fidelity portraits from recent NeRF-based methods remains challenging.Source: §A.9 · p. 14
A.10. Limitation and Future Work
- ER-NeRF requires per-scene training for new target portraits, and its single-video training can yield weak synchronization on cross-lingual speech or singing, alongside blurry torso regions.The paper identifies these as limitations and names large audiovisual priors and torso-representation improvements as future directions.Source: §A.10 · p. 14
A.8. Additional Qualitative Comparison
- ER-NeRF outperforms most baselines in image quality while retaining high lip-sync accuracy in the Testset A lip-synchronization setting.The qualitative comparison shows synthesized head results; this setting synthesizes only the head part.Source: §A.8 · p. 14, Figure 10 · p. 15