Source-linked AI summary
GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis
Zhenhui Ye, Ziyue Jiang, Yi Ren, Jinglin Liu, JinZheng He, Zhou Zhao
TL;DR
GeneFace addresses limited out-of-domain generalization and mean-face artifacts in NeRF-based talking-face generation. It combines a large-corpus variational motion generator, domain-adaptive post-net, and NeRF renderer, and reports more generalized and high-fidelity results than previous methods.
Problem
Previous NeRF-based talking-face methods have limited out-of-domain audio generalization from small training sets and suffer mean-face artifacts from one-to-many audio-to-motion mappings.
Method
GeneFace uses a variational motion generator trained on a large lip-reading corpus, an adversarial domain-adaptive post-net, and a NeRF renderer conditioned on predicted facial motion.
Results
GeneFace achieves more generalized and high-fidelity talking-face generation than previous methods and outperforms GAN-based and NeRF-based baselines on objective and subjective metrics.
Takeaways & Limitations
The system improves lip synchronization and expressiveness of synthesized talking-head video.
Takeaways & Limitations
The method can produce tiny landmark fluctuations that cause artifacts, and its vanilla-NeRF renderer has long training and inference times.
Abstract
from arXiv · showhide
Generating photo-realistic video portrait with arbitrary speech audio is a crucial problem in film-making and virtual reality. Recently, several works explore the usage of neural radiance field in this task to improve 3D realness and image fidelity. However, the generalizability of previous NeRF-based methods to out-of-domain audio is limited by the small scale of training data. In this work, we propose GeneFace, a generalized and high-fidelity NeRF-based talking face generation method, which can generate natural results corresponding to various out-of-domain audio. Specifically, we learn a variaitional motion generator on a large lip-reading corpus, and introduce a domain adaptative post-net to calibrate the result. Moreover, we learn a NeRF-based renderer conditioned on the predicted facial motion. A head-aware torso-NeRF is proposed to eliminate the head-torso separation problem. Extensive experiments show that our method achieves more generalized and high-fidelity talking face generation compared to previous methods.
1 INTRODUCTION
Previous NeRF-based talking-face systems struggle with out-of-domain audio and one-to-many audio-to-motion mappings. GeneFace addresses these challenges with a large-corpus variational motion generator, domain adaptation, and a NeRF renderer.
- Challenges: Previous NeRF-based methods generalize weakly to out-of-domain audio because they train on only a few minutes of video.The limited training data makes models less robust to cross-lingual audio and singing voice.
- Challenges: One-to-many audio-to-motion mappings cause regression models to produce over-smoothed, blurry mouth movements and poor lip synchronization.The resulting mean-face behavior can yield half-opened mouths when multiple facial motions are valid.
- Approach: GeneFace learns a variational audio-to-motion model from hundreds of hours of lip-reading data to generate accurate and expressive 3D facial landmarks.A flow-based prior is used to improve temporal modeling and generative diversity.
- Approach: A semi-supervised adversarial post-net adapts predicted landmarks from the multi-speaker domain to the target person domain.This bridges the domain gap between large-corpus motion training and the short target-person video.
- Approach: A NeRF-based renderer synthesizes high-fidelity frames conditioned on predicted 3D facial landmarks.The framework is organized into audio-to-motion, motion domain adaptation, and motion-to-image stages.
- Results: Experiments show that GeneFace outperforms state-of-the-art GAN-based and NeRF-based baselines on objective and subjective metrics.The reported comparison covers both quantitative and perceptual evaluation perspectives.
2 RELATED WORK
GeneFace combines 3D facial motion representations with NeRF rendering to retain controllable motion, 3D naturalness, and detailed appearance. It differs from prior NeRF systems by learning audio-to-motion mappings on a large lip-reading corpus.
- Audio-driven Talking Head Generation: 3DMM-based methods enable head-pose and facial-movement control but represent the human head coarsely, omitting details such as hair, wrinkles, and teeth.These omissions make realistic rendering challenging for GAN-based methods.
- Neural Radiance Field for Rendering Face: NeRF models continuous 3D scenes and can render high-fidelity portraits with detailed appearance, including hair and wrinkles.This motivates their use as neural renderers for human portraits and talking faces.
- Neural Radiance Field for Rendering Face: Prior NeRF-based talking-face systems model audio-driven face generation, but their limited training data constrains generalization to out-of-domain audio.GeneFace targets this gap by introducing a generative audio-to-motion model trained on a large lip-reading corpus.
- GeneFace: GeneFace bridges 3DMM-based motion control and NeRF-based image quality by combining predicted motion representations with neural rendering.Its design aims to retain 3D naturalness while improving image fidelity and audio generalizability.
3 GENEFACE
GeneFace uses a three-stage pipeline: a variational motion generator, a domain-adaptive post-net, and a landmark-conditioned NeRF renderer with a head-aware torso component.
- Overview: The system transforms HuBERT audio features into 3D facial landmarks, refines them for the target-person domain, and renders high-fidelity frames with NeRF.The three stages are the variational motion generator, post-net, and NeRF-based renderer.
- 3.1 Variational Motion Generator: The variational motion generator uses HuBERT audio features and 68 reconstructed-mesh keypoints to model detailed facial movement.Its encoder and decoder use dilated convolutions to capture long-term temporal relationships.
- 3.1 Variational Motion Generator: A flow-based prior addresses frame independence and mean-seeking behavior that limit vanilla VAE landmark-sequence generation.The training process uses a Monte-Carlo ELBO and a sync-expert to guide audio-landmark synchronization.
- 3.2 Domain Adaptive Post-net: The post-net adapts landmarks predicted from the multi-speaker corpus to the target-person domain using semi-supervised adversarial training.It preserves temporal consistency and lip synchronization while mapping frames into the target domain.
- 3.3 NeRF-based Renderer: The conditional NeRF uses viewing direction, 3D location, and normalized landmarks from three neighboring frames to control facial geometry and color.The renderer aggregates radiance along camera rays using standard volume rendering.
- 3.3 NeRF-based Renderer: The head-aware torso-NeRF conditions torso rendering on the head-NeRF output color to address head-torso separation artifacts.The method trains separate head and torso NeRFs and supplies the rendered head appearance pixel-wise to the torso model.
4 EXPERIMENTS
GeneFace is evaluated using quantitative metrics, qualitative comparisons, user studies, and ablations. It maintains lip synchronization on out-of-domain audio, achieves strong visual quality, and benefits from its proposed components.
- Evaluation Metrics: The evaluation measures image quality with FID, lip synchronization with LMD and SyncNet confidence, and OOD generalizability using cross-lingual, crossgender, and singing audios.The OOD audio set specifically tests cross-lingual, crossgender, and singing voice conditions.
- Quantitative Evaluation: GeneFace maintains good synchronization on out-of-domain audio, while person-specific baselines show significant sync-score drops.GeneFace performs best in LMD and achieves a better sync score than other baselines.
- Qualitative Evaluation: GeneFace produces higher-fidelity results than AD-NeRF by avoiding head-torso separation, blurry mouths, and unsynchronized lips in qualitative comparisons.The comparison marks these artifacts with brown, blue, and red arrows, respectively.
- User Study: User studies with 20 attendees rating 10 English, Chinese, and German clips assess lip-sync accuracy, video realness, and image quality on a 1–5 MOS scale.The study uses average scores and reports 95% confidence intervals in Table 2.
- User Study: GeneFace achieves the highest video-realness and lip-sync-accuracy scores among the compared methods, although LSP has slightly better image quality.Person-specific methods outperform one-shot methods on video realness and image quality.
- Ablation Study: Removing the variational prior flow, sync-expert, post-net, or head-aware torso conditioning degrades synchronization, FID, or both.The ablations associate prior flow and sync-expert with synchronization, post-net with FID and Sync, and head awareness with avoiding head-torso separation.
5 CONCLUSION
GeneFace targets weak generalizability and mean-face outputs in NeRF-based talking-face generation. Its variational motion generator, domain-adaptive post-net, and head-aware torso-NeRF support more generalized and high-fidelity synthesis.
- 5 CONCLUSION: GeneFace addresses weak generalizability and mean-face problems with a generic audio-to-motion mapping learned from a large corpus.The method also adapts predicted motion to the target person domain and uses head-aware torso-NeRF rendering.
ETHICS STATEMENT
The paper discusses ethical risks of talking-face synthesis and identifies deepfake misuse as a concern. The included technical passages describe supporting model components but do not add a separate ethics conclusion.
- ETHICS STATEMENT: Talking-face synthesis may be misused to spread fake information or obtain illegal profits, motivating deepfake detection and licensing restrictions.The paper recommends considering digital face forensics and restricting the open-source license.
B DETAILED EXPERIMENTAL SETTINGS
The detailed experimental-settings section points to an appendix table for GeneFace hyper-parameters.
- B DETAILED EXPERIMENTAL SETTINGS: GeneFace hyper-parameters are listed in Table 4.
C.1 QUALITATIVE RESULTS WITH ALL BASELINES
Qualitative comparisons reveal distinct weaknesses among the baselines, including blurry mouths, identity loss, and unnatural lip transitions.
- Wav2Lip achieves competitive lip-sync but produces blurry mouth results.
- MakeItTalk and PC-AVS fail to preserve speaker identity, producing unrealistic results.
- LSP generates unnatural lip movements during transitions between different syllables.
C.2 EVALUATION ON 3D LANDMARK L2 ERROR
The landmark evaluation uses L2 error to assess the variational generator, while visualization examines domain alignment between source and target motions.
- 3D landmark L2 error evaluates the predicted landmark quality across VAE+Flow, vanilla VAE, and regression models.The comparison is reported in Table 5.
- Removing the prior flow or replacing the variational model with regression leads to a performance drop.
- T-SNE visualization shows a domain gap between LRS3 and the target video, which the post-net successfully bridges.
D LIMITATIONS AND FUTURE WORK
GeneFace has two reported limitations: temporal fluctuations can create visual artifacts, and the vanilla-NeRF backend requires long training and inference times.
- Tiny fluctuations in generated landmark sequences can produce artifacts such as shaking hair.A heuristic Gaussian filter currently alleviates this issue.
- The vanilla-NeRF renderer leads to long training and inference times.Future work targets accelerated and lightweight NeRF methods.
- Figure 9 presents keyframe comparisons using phonetic symbols and synthesized talking heads from all baselines.