Source-linked AI summary
Talking Head Synthesis with Facial Landmark Guidance via 3D Gaussian Splatting
Ziheng Yang, Yinfeng Yu, Yongming Li
TL;DR
Speech features do not explicitly specify facial spatial layouts, limiting accurate and stable audio-driven 3D deformation. The paper addresses this with local landmark-guided spatial enhancement and global landmark compensation for 3DGS attributes. Experiments report improved visual quality, facial structure accuracy, and lip synchronization in self-driven and cross-driven settings.
Problem
Speech representations lack explicit facial spatial structure, making accurate mouth and expression deformation difficult in audio-driven 3D synthesis.
Method
The framework uses predicted 3D facial landmarks to enrich expression-sensitive spatial points and encodes all landmarks into a global condition for 3DGS attribute correction.
Results
Experiments in self-driven and cross-driven settings improve visual quality, facial structure accuracy, and lip synchronization.
Takeaways & Limitations
Landmark guidance provides local and whole-face structural information for more consistent speech-driven facial modeling.
Abstract
from arXiv · showhide
Audio-driven digital human generation plays an important role in virtual communication, immersive interaction, and media production. With the development of Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), recent talking-head systems have obtained more faithful 3D facial geometry and appearance modeling. A remaining difficulty is that speech features mainly describe temporal acoustic patterns rather than explicit facial layouts. As a result, directly driving 3D facial deformation with audio may produce inaccurate mouth motion, weak expression details, and local artifacts. To address this issue, we propose a facial-keypoint-guided spatial enhancement module. The predicted landmarks provide structural cues for selecting and enriching spatial points around expression-sensitive facial regions. We further introduce a global landmark compensation mechanism, where the full set of keypoints is encoded into a conditioning vector to refine 3DGS attributes. This compensation supplies whole-face structural information to the underlying shape representation. Experiments under self-driven and cross-driven settings show that the proposed method improves visual quality, facial realism, and lip synchronization.
I. INTRODUCTION
Speech-driven talking-head synthesis must convert temporal acoustic information into spatially coherent facial motion. The proposed framework uses predicted facial landmarks for local spatial enhancement and global structural compensation, improving generation quality, facial realism, and lip synchronization.
- The method targets speech-driven facial synthesis for applications including virtual avatars, film dubbing, remote communication, and real-time human–computer interaction.
- Audio features lack explicit facial spatial structure, making rapid mouth motion and expression changes prone to local artifacts and unstable deformation.
- A pretrained audio-to-landmark model predicts 3D facial keypoints that guide spatial representations toward expression- and lip-sensitive regions.
- Global landmark compensation encodes all frame keypoints into a conditioning vector that refines 3DGS attributes alongside local enhancement.
II. METHOD
The method injects predicted landmark structure into spatial 3D point features so facially relevant regions receive stronger modeling emphasis. This spatial enhancement preserves the original geometric distribution while enriching expression-sensitive areas.
- Audio-to-keypoint prediction produces 68 3D landmarks per frame, providing spatial guidance absent from ordinary speech representations.
- Tri-Plane Hash Encoder features for spatial points and landmarks are compared with cosine similarity to identify structurally related points.
- Points whose maximum landmark similarity exceeds threshold θ are selected, with θ = 0.6 in the implementation.
- Matched landmark features are added to selected spatial-point features, while unselected points retain their original features.
- The resulting landmark-aware representation emphasizes the mouth, eyes, and other semantically important regions for facial motion reconstruction and audio–lip synchronization.
C. Landmark Compensation
Landmark compensation supplies whole-face structural guidance beyond point-wise matching. A global landmark condition is combined with each local point feature to predict adaptive corrections to 3DGS attributes.
- C. Landmark Compensation: Local point-wise enhancement may not capture coordination across facial regions during large expressions or when facial shape differs from training data.
- C. Landmark Compensation: All 68 encoded landmarks are concatenated and passed through an MLP to produce a global conditioning vector.
- C. Landmark Compensation: The global vector is broadcast to every spatial point and concatenated with its local feature before compensation prediction.
- C. Landmark Compensation: Two MLPs predict fine-grained translation, rotation, and scale offsets plus a point-specific factor τ_i that adaptively controls compensation strength.
- C. Landmark Compensation: The compensation branch combines local and global landmark information to provide structural correction and more stable, coherent facial motion.
D. Geometry Regularizer
The geometry regularizer addresses unstable unseen-view geometry in 3DGS by constraining generated depth and surface normals against estimates from real images.
- D. Geometry Regularizer: Sparse viewpoints and limited training frames can cause unstable 3DGS geometry in unseen views, motivating depth and surface-normal constraints.
E. Training Detail
Training follows TalkingGaussian’s face–mouth decomposition, with separate static and dynamic optimization stages. A geometry-structured loss is added during dynamic optimization to improve facial geometry.
- E. Training Detail: TalkingGaussian’s face–mouth decomposition separates training into static-field and dynamic-field optimization.The static stage learns coarse head structure, while the dynamic stage incorporates the proposed geometry-structured loss.
- E. Training Detail: The dynamic stage adds a geometry-structured loss to improve facial geometry.
A. Experimental Settings
Experiments use public talking-head video data, compare against representative 2D and 3D baselines, and run on a Tesla T4 with the baseline optimization schedule. Qualitative evaluation examines facial and oral clarity and visible artifacts.
- A. Experimental Settings: Experiments use videos of two male speakers, “Lieu” and “Obama”, and one female speaker, “May”.Obama videos are resized to 450 × 450; the other videos use 512 × 512.
- A. Experimental Settings: The comparison includes Wav2Lip and IP_LAP as 2D baselines, plus seven representative 3D talking-head methods.The 3D baselines are AD-NeRF, RAD-NeRF, ER-NeRF, Geneface, Talking Gaussian, Gaussian Talker, and DEGSTalk.
- A. Experimental Settings: Qualitative comparisons report clearer facial and oral regions with fewer visible artifacts for the proposed method.
- A. Experimental Settings: All experiments run on a Tesla T4, with oral and facial branches optimized for 50,000 iterations.Training takes about one hour, using Adam and AdamW optimizers.
B. Quantitative Evaluation
Evaluation measures reconstruction quality, facial structure accuracy, and lip synchronization under self-driven and cross-driven protocols. Tables I and II organize quantitative comparisons for these settings.
- B. Quantitative Evaluation: Evaluation covers image reconstruction quality, facial structure accuracy, and lip synchronization.PSNR and LPIPS measure reconstruction quality, LMD measures landmark-level accuracy, and LSE-C/LSE-D measure synchronization.
- B. Quantitative Evaluation: The self-driven protocol trains and tests on different segments of the same speaker’s video.
- B. Quantitative Evaluation: The cross-driven protocol tests audio from other videos to assess synchronization when driving utterances differ from training data.Two audio clips from the synthesized Obama dataset are used for cross-driven evaluation.
- B. Quantitative Evaluation: Table I reports quantitative comparisons in the self-reconstruction setting.
- B. Quantitative Evaluation: Table II reports quantitative comparisons in the cross-driven setting, with the best result highlighted in bold.
3) Evaluation Results:
The proposed method improves quantitative and qualitative outcomes across self-driven and cross-driven evaluation, while ablations attribute gains to audio, spatial-enhancement, and geometry components. A threshold of θ = 0.6 balances selectivity with facial-region coverage.
- 3) Evaluation Results:: In self-reconstruction, the proposed method achieves the best PSNR, LPIPS, and LMD scores, while cross-driven testing gives it the lowest LSE-D and highest LSE-C for both audio clips.The authors interpret these results as improved visual fidelity, facial structure accuracy, and synchronization under differing driving audio.
- 3) Evaluation Results:: Qualitatively, the method produces clearer mouth shapes, more stable facial regions, and better lip-motion agreement with speech.Fig. 2 highlights blur, artifacts, and unnatural facial details in red boxes.
- 3) Evaluation Results:: Replacing Audio Net with DeepSpeech mainly degrades LSE-D and LSE-C, while removing the Audio Encoder also weakens synchronization metrics.The ablation suggests the proposed audio representation and temporal compression support speech-to-lip alignment.
- 3) Evaluation Results:: Removing Spatial Enhancement worsens LMD and LPIPS, and the Geometry Regularizer reduces local facial artifacts while making faces more natural.
- 3) Evaluation Results:: θ = 0.6 selects about 20% of points and covers the mouth region and neighborhood, balancing selectivity with spatial coverage.At θ = 0.4 selection is less selective; at θ = 0.8 coverage becomes insufficient.
IV. CONCLUSION
The paper presents a facial-landmark-guided 3DGS framework that improves speech-driven talking-head synthesis across self-driven and cross-driven settings.
- The framework improves visual quality, facial structure accuracy, and lip synchronization in both self-driven and cross-driven settings.Ablation results verify the contribution of each component.
- Predicted landmarks enhance spatial point features around expression-sensitive regions, while global landmark compensation refines 3DGS attributes with whole-face structure.The Geometry Regularizer stabilizes facial geometry, and the lightweight Audio Net provides compact speech representations for motion prediction.
- The method combines spatial enhancement, global structural correction, geometry regularization, and compact audio modeling within one facial-landmark-guided 3DGS framework.