Source-linked AI summary
SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis
Ziqiao Peng, Wentao Hu, Yue Shi, Xiangyu Zhu, Xiaomei Zhang, Hao Zhao, Jun He, Hongyan Liu, Zhaoxin Fan
TL;DR
Existing talking-head methods struggle to synchronize identity, lip movements, expressions, and head poses, limiting realistic speech-driven synthesis. SyncTalk combines NeRF-based identity modeling with facial and pose synchronization modules and portrait refinement. It reports better performance than prior methods, including higher user-study scores across all evaluations.
Problem
GAN methods can vary identity across frames, while NeRF methods can preserve identity yet struggle with lip synchronization, expression control, and stable head poses.
Source: §1 · p. 2Method
SyncTalk combines NeRF-based tri-plane hash modeling with Face-Sync and Head-Sync modules and a Portrait-Sync Generator.
Source: §1 · p. 2Results
User-study participants rated SyncTalk above previous methods in all evaluations, with video authenticity 20% above second-place IP-LAP.
Source: §4.3 · p. 8Takeaways & Limitations
SyncTalk is presented as a framework for generating realistic, synchronized talking-head videos while maintaining subject identity.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Achieving high synchronization in the synthesis of realistic, speech-driven talking head videos presents a significant challenge. Traditional Generative Adversarial Networks (GAN) struggle to maintain consistent facial identity, while Neural Radiance Fields (NeRF) methods, although they can address this issue, often produce mismatched lip movements, inadequate facial expressions, and unstable head poses. A lifelike talking head requires synchronized coordination of subject identity, lip movements, facial expressions, and head poses. The absence of these synchronizations is a fundamental flaw, leading to unrealistic and artificial outcomes. To address the critical issue of synchronization, identified as the "devil" in creating realistic talking heads, we introduce SyncTalk. This NeRF-based method effectively maintains subject identity, enhancing synchronization and realism in talking head synthesis. SyncTalk employs a Face-Sync Controller to align lip movements with speech and innovatively uses a 3D facial blendshape model to capture accurate facial expressions. Our Head-Sync Stabilizer optimizes head poses, achieving more natural head movements. The Portrait-Sync Generator restores hair details and blends the generated head with the torso for a seamless visual experience. Extensive experiments and user studies demonstrate that SyncTalk outperforms state-of-the-art methods in synchronization and realism. We recommend watching the supplementary video: https://ziqiaopeng.github.io/synctalk
1. Introduction
Realistic speech-driven talking heads remain difficult because existing approaches struggle to synchronize identity, lips, expressions, and head pose. SyncTalk combines NeRF-based identity modeling with modules for facial and pose synchronization and portrait refinement.
1. Introduction
- Existing GAN methods can model lip movements but often vary facial identity across frames, while NeRF methods preserve identity yet struggle with lip sync, expression control, and stable head poses.These mismatches diminish overall video realism.Source: §1 · p. 2
- SyncTalk uses NeRF-based tri-plane hash representations to maintain subject identity while targeting synchronized, realistic speech-driven talking heads.The framework adds Face-Sync and Head-Sync modules and a Portrait-Sync Generator for visual refinement.Source: §1 · p. 2
- The Face-Sync Controller combines an audio-visual encoder and facial animation capturer to support accurate lip synchronization and dynamic facial expressions.The method uses a semantically enriched 3D blendshape model with 52 parameters for controlling expression regions.Source: §1 · p. 2
- The Head-Sync Stabilizer tracks facial keypoints and applies bundle adjustment to produce smooth, synchronous head motion.The Portrait-Sync Generator repairs NeRF artifacts, including hair and background flaws, and outputs high-resolution video.Source: §1 · p. 2
2. Related Work
Prior talking-head methods face different synchronization and identity challenges: GAN approaches often struggle with consistent identity, while NeRF approaches retain detail but can lack control of speech-driven motion. SyncTalk combines synchronization modules with NeRF-based modeling and portrait refinement.
2.1. GAN-based Talking Head Synthesis
- GAN-based methods include lip-region and full-face synthesis, but can struggle to preserve identity or synchronize facial expressions and head poses.Wav2Lip reconstructs lips using five reference frames and struggles to maintain identity.Source: §2.1 · p. 2
2.2. NeRF-based Talking Head Synthesis
- NeRF-based talking-head methods preserve identity and facial detail, but prior approaches can produce inaccurate lip movements, limited expression control, and unstable head poses.Some methods are limited to blinking and can produce head–torso separation.Source: §2.1 · p. 2, §2.2 · p. 3
- SyncTalk uses a Face-Sync Controller to model audio–lip relationships and a Head-Sync Stabilizer to stabilize poses, addressing synchronization gaps in prior NeRF methods.Its three-part pipeline also renders with a Dynamic Portrait Renderer and restores details such as hair and background.Source: §2.2 · p. 3, Figure 2 · p. 3
3. Method
SyncTalk combines audio-driven facial control, head-pose stabilization, and portrait rendering in a three-part NeRF pipeline. Its components target synchronization of lip movements, expressions, and head pose while refining rendered detail.
3.1. Overview
- SyncTalk has three parts: the Face-Sync Controller controls lip movements and expressions, the Head-Sync Stabilizer provides stable head pose, and the Dynamic Portrait Renderer renders facial frames.The overview assigns lip and expression control, pose stabilization, and frame rendering to these respective components.Source: §3.1 · p. 3
3.2. Face-Sync Controller
- The Face-Sync Controller trains an audio-visual encoder on synchronized and unsynchronized samples, using a discriminator and reconstruction objective to derive audio features related to lip movements.The encoder is trained on LRS2; the discriminator distinguishes fully overlapping face and audio windows from non-overlapping ones.Source: §3.2 · pp. 3–4
- A 52-parameter semantic blendshape model captures expressions, with seven core coefficients controlling eyebrow, forehead, and eye regions independently of lip movements.Masked attention separates lip and expression regions to reduce feature entanglement and artifacts from their coupling.Source: §3.2 · p. 4, Figure 3 · p. 4
3.3. Head-Sync Stabilizer
- The Head-Sync Stabilizer estimates focal length, rotation, and translation by aligning projected 3DMM landmarks with video landmarks, then refines pose across frames.The initial focal-length search reinitializes rotation and translation for each candidate.Source: §3.3 · pp. 4–5
- Optical-flow keypoint tracking and two-stage bundle adjustment refine facial keypoint alignment and jointly optimize 3D keypoints and head pose; the resulting pose and translation are described as smooth and stable.The first stage aligns initialized 3D keypoints with tracked image points; the second jointly refines keypoints, rotation, and translation.Source: §3.3 · p. 5
3.4. Dynamic Portrait Renderer
- The Dynamic Portrait Renderer uses three oriented 2D hash grids to encode spatial coordinates and conditions rendering on view direction, lip features, and expression features.The renderer’s implicit function maps these inputs to color and density.Source: §3.4 · p. 5
- Training proceeds from MSE color loss to a refinement stage combining MSE and LPIPS on random image patches, while the Portrait-Sync Generator restores hair detail and fills dark head-torso gaps with average neck color.The generator also merges the rendered face area with the original image using the synchronized head pose.Source: §3.4 · pp. 5–6
- The authors report state-of-the-art performance on most head-reconstruction metrics in Table 1, where “Portrait” denotes use of the Portrait-Sync Generator.The supplied caption gives no individual metric values.Source: Table 1 · p. 6
4. Experiments
Experiments compare SyncTalk with GAN- and NeRF-based methods and report improvements in image quality, synchronization, human ratings, and rendering speed. Ablations examine the effects of its synchronization and detail-restoration components.
4.1. Experimental Settings
- The comparison uses edited English and French video sequences averaging about 8,843 frames, with most videos at 512 × 512 resolution.The AD-NeRF [14] video is 450 × 450; training takes approximately two hours on an NVIDIA RTX 3090 GPU.Source: §4.1 · p. 6
4.2. Quantitative Evaluation
- 3× improvement on LPIPS over previous state-of-the-art ER-NeRF [21], with superior image quality across aspects and state-of-the-art lip synchronization on out-of-distribution audio.The lip-synchronization table evaluates two audio samples driving the same subject.Source: §4.2 · pp. 6–7, Table 2 · p. 6
- Portrait mode renders at 50 FPS on an NVIDIA RTX 3090 GPU at 512 × 512 resolution, exceeding the 25 FPS input rate.With data preloaded on the GPU, head output reaches 52 FPS.Source: §4.2 · p. 7
4.3. Qualitative Evaluation
- SyncTalk achieves the best overall visual effect, with comparisons highlighting identity, lip shape, expression, and head–torso continuity.Comparisons include Wav2Lip [34], IP-LAP, GeneFace [44], and ER-NeRF [21].Source: §4.3 · p. 7, Figure 4 · p. 7
- Across user-study evaluations, SyncTalk ranks highest and scores 20% above IP-LAP in video authenticity.The study uses 24 clips and 35 participants rating five perspectives.Source: §4.3 · pp. 7–8, Table 3 · p. 8
4.4. Ablation Study
- The Audio-Visual Encoder ablation worsens all three metrics, increasing LMD error by 21.15%; replacing facial animation capture with ER-NeRF [21]’s blink module affects eyebrow movement and image quality.The ablation evaluates PSNR, LPIPS, and LMD.Source: §4.4 · p. 8, Table 4 · p. 8
- Removing the Head-Sync Stabilizer degrades metrics and causes head-pose jitter and head–torso separation, while removing Portrait-Sync leaves hair-detail restoration and segmentation boundaries worse.The ablation is reported for subject “May.”Source: §4.4 · p. 8, Table 4 · p. 8, Figure 5 · p. 8
5. Conclusion
SyncTalk is a NeRF-based approach to realistic speech-driven talking head synthesis, designed to maintain identity while synchronizing facial and head motion. The conclusion reports superior performance in realism and synchronization compared with existing methods.
5. Conclusion
- SyncTalk combines a Facial Sync Controller, Head Sync Stabilizer, and Portrait Sync Generator to maintain identity and synthesize synchronized lips, expressions, and stable head poses.The conclusion reports superior performance in realistic and synchronized talking-head video generation compared with existing methods.Source: §5 · p. 8