Source-linked AI summary
Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation
Xian Liu, Yinghao Xu, Qianyi Wu, Hang Zhou, Wayne Wu, Bolei Zhou
TL;DR
Audio-driven portrait generation must capture fine local facial semantics and divergent head-torso motion without relying on accurate explicit structure. SSP-NeRF addresses this with semantic-aware ray sampling and torso deformation in one unified NeRF, and the paper reports more realistic portraits than previous methods.
Problem
Audio-driven portrait generation must handle distinct semantic facial dynamics and non-rigid head-torso motion, while prior approaches often rely on accurate structural information.
Method
SSP-NeRF combines Semantic-Aware Dynamic Ray Sampling with a Torso Deformation module in one unified NeRF.
Results
SSP-NeRF renders more realistic high-fidelity audio-driven video portraits than previous methods on objective evaluations and human studies.
Takeaways & Limitations
The approach supports efficient unified-NeRF portrait rendering without accurate structural information while addressing local facial detail and global torso motion.
Takeaways & Limitations
The method assumes semantic parsing maps are available for training images.
Abstract
from arXiv · showhide
Animating high-fidelity video portrait with speech audio is crucial for virtual reality and digital entertainment. While most previous studies rely on accurate explicit structural information, recent works explore the implicit scene representation of Neural Radiance Fields (NeRF) for realistic generation. In order to capture the inconsistent motions as well as the semantic difference between human head and torso, some work models them via two individual sets of NeRF, leading to unnatural results. In this work, we propose Semantic-aware Speaking Portrait NeRF (SSP-NeRF), which creates delicate audio-driven portraits using one unified set of NeRF. The proposed model can handle the detailed local facial semantics and the global head-torso relationship through two semantic-aware modules. Specifically, we first propose a Semantic-Aware Dynamic Ray Sampling module with an additional parsing branch that facilitates audio-driven volume rendering. Moreover, to enable portrait rendering in one unified neural radiance field, a Torso Deformation module is designed to stabilize the large-scale non-rigid torso motions. Extensive evaluations demonstrate that our proposed approach renders more realistic video portraits compared to previous methods. Project page: https://alvinliu0.github.io/projects/SSP-NeRF
1. Introduction
SSP-NeRF addresses semantic differences across portrait regions and the divergent non-rigid motions of the head and torso within one unified NeRF. It combines semantic-aware ray allocation with torso deformation to produce stable, high-fidelity audio-driven portraits without accurate structural information.
- Challenges: Audio-driven portrait generation must model facial regions with different appearance and motion patterns while preserving the global head-torso relationship.The paper identifies semantic imbalance and divergent head-torso motion as central challenges for unified NeRF modeling.
- Semantic-aware modeling: SSP-NeRF uses Semantic-Aware Dynamic Ray Sampling to focus rendering and learning on semantically important facial regions.Its parsing branch supervises facial semantics, dynamically adjusts ray counts by region, and emphasizes small areas such as lips and teeth.
- Results: The method targets high-fidelity, stable portraits with improved lip synchronization and image quality compared with prior approaches.The stated contributions claim efficient unified-NeRF rendering and improvements on objective evaluations and human studies.
- Unified portrait representation: A Torso Deformation module implicitly models large-scale non-rigid torso motion while retaining a unified neural radiance field.The module predicts torso displacement from head-canonical information to address the mismatch between head and torso motion.
2. Related Work
Prior audio-driven talking-head methods use stitching, end-to-end generation, or explicit structural representations, with limitations in resolution, flexibility, or deformation accuracy.
- Stitching methods: Early methods modify mouth shapes using predefined phoneme-mouth correspondence rules and stitching techniques.These approaches construct talking heads from manually specified correspondence between speech units and mouth shapes.
- End-to-end methods: Deep neural audio-driven methods improve feature extraction or lip synchronization but may generate only fixed-sized, low-resolution images.The passage cites identity-word disentanglement and pretrained lip-sync supervision as representative strategies.
- Model-based methods: Model-based methods bridge audio and facial images through explicit 2D landmarks or 3D representations.These structural intermediates provide an explicit route from speech to facial appearance.
3. Our Approach
SSP-NeRF renders audio-driven portraits with one unified NeRF by combining semantic-aware ray sampling with torso deformation. These modules focus modeling on difficult facial regions while stabilizing non-rigid torso motion.
- Overview: SSP-NeRF uses one unified NeRF to model the portrait while addressing distinct facial semantics and head-torso motion.The framework introduces semantic-aware sampling and deformation rather than separating head and torso into individual NeRFs.
- Semantic-Aware Dynamic Ray Sampling: The implicit portrait parsing branch predicts semantic information alongside density and color, using semantic labels that are view-invariant.The branch extends NeRF with semantic prediction and does not condition semantic parsing on viewing direction.
- Semantic-Aware Dynamic Ray Sampling: Dynamic ray sampling allocates rays across semantic categories according to their previous-epoch average losses, emphasizing small or difficult regions.Area-agnostic category losses prevent large regions from dominating, while higher losses increase attention to hard-to-learn parts such as eyes, lips, and teeth.
- Torso Deformation Module: The Torso Deformation module learns location displacements from time, head pose, canonical pose, and 3D coordinates to model non-rigid torso motion.Audio is excluded from torso deformation because torso pose changes slightly and is weakly related to speech audio; the canonical pose is the first-frame head pose.
- Volume Rendering and Network Training: Deformed ray paths are used to volume-render both color and semantic logits for the unified portrait representation.The deformation adapts NeRF rendering to warped coordinates while preserving semantic-aware color and semantic prediction.
4. Experiments
Experiments evaluate SSP-NeRF across cropped and full-resolution portrait settings, qualitative comparisons, user studies, and module ablations. The method achieves strong image quality and lip synchronization while using a compact model and unified rendering pipeline.
- Experimental settings: SSP-NeRF is trained on short portrait videos, unlike prior studies requiring large-corpus data or hours-long videos.The collected videos average 6,750 frames at 25 fps, and the method uses short videos of a few minutes.
- Experimental settings: The evaluation measures image quality with PSNR and SSIM, and mouth-shape or lip-sync accuracy with LMD and SyncNet Confidence.For LMD, lower values are better; for the other metrics, higher values are better.
- Experimental settings: The experiments compare SSP-NeRF with representative landmark-, image-reconstruction-, model-based, and NeRF-based baselines across cropped and full-resolution settings.Cropped evaluation uses Testsets A, B, and C; full-resolution evaluation uses Testset A against AD-NeRF.
- Qualitative evaluation: Qualitative comparisons show SSP-NeRF produces more lip-synced, higher-quality portraits, while competing methods exhibit inaccurate mouths, static heads, identity loss, or incomplete portraits.AD-NeRF produces full-resolution portraits but shows head-torso separation and blurry results under separated rendering with uniform ray sampling.
- User study: Users prefer SSP-NeRF over AD-NeRF because of fine-grained local rendering and stable torso motions, with Fleiss-Kappa 0.816 indicating almost perfect rating agreement.The user study evaluates lip-sync accuracy, video realness, and image quality on a 1–5 scale.
- Ablation study: Ablations show that torso deformation mainly affects the torso, while removing audio-aware deformation causes a dramatic lip-sync drop and removing semantic sampling weakens part-wise appearance and geometry modeling.The deformation heatmap concentrates displacements in the torso, and dynamic ray sampling further facilitates fine-grained results.
5. Broader Impact
SSP-NeRF has applications in digital humans and filmmaking but may also be misused for identity theft, deepfakes, and media manipulation. The authors share generated results with the deepfake-detection community and identify synthesis speed and occasional unnatural mouths as limitations.
- Realistic talking-portrait animation supports digital human and filmmaking applications but can also enable identity theft, deepfake generation, and media manipulation.
- The authors share generated results with the deepfake-detection community because limited realistic data constrains detector performance.
- Image synthesis remains slow because rendering high-quality images is computationally heavy.
- A language gap between training and driven audio can occasionally make the synthesized mouth look unnatural.
6. Conclusion
The paper presents SSP-NeRF for audio-driven portrait generation with one unified NeRF. Its semantic-aware ray sampling models local portrait dynamics, while torso deformation learns global torso motion; experiments report more realistic portraits than previous methods.
- SSP-NeRF generates audio-driven portraits with one unified set of NeRF.
- Semantic-Aware Dynamic Ray Sampling captures detailed appearance and local dynamics for each portrait part without accurate structural information.
- Torso Deformation learns global torso motion and prevents separated head-torso results.
- Extensive experiments show that SSP-NeRF synthesizes more realistic video portraits than previous methods.