Source-linked AI summary
Talking-head Generation with Rhythmic Head Motion
Lele Chen, Guofeng Cui, Celong Liu, Zhong Li, Ziyi Kou, Yi Xu, Chenliang Xu
TL;DR
The paper addresses the difficulty of generating lip-synced talking-head videos with natural, controllable head motion. It explicitly models head motion and facial expressions, then combines 3D-aware generation, hybrid reference embedding, and nonlinear composition. Experiments on benchmark datasets show improved performance over prior methods, while extreme poses and unmodeled camera, lighting, and audio conditions remain limitations.
Problem
Existing methods either omit head-motion modeling or rely on landmarks or video frames as guidance, leaving natural and controllable talking-head generation unresolved.
Method
The method explicitly separates head motion and facial expressions and uses 3D-aware generation, hybrid reference embedding, and nonlinear composition to synthesize future frames.
Results
The framework achieves significantly better quantitative and qualitative results than state-of-the-art methods across several benchmark evaluations.
Takeaways & Limitations
The approach generates controllable, photo-realistic talking-head videos with natural head movements and synchronized facial movements to speech audio.
Takeaways & Limitations
The method struggles with extreme poses and omits camera motion, lighting conditions, and audio noise modeling.
Abstract
from arXiv · showhide
When people deliver a speech, they naturally move heads, and this rhythmic head motion conveys prosodic information. However, generating a lip-synced video while moving head naturally is challenging. While remarkably successful, existing works either generate still talkingface videos or rely on landmark/video frames as sparse/dense mapping guidance to generate head movements, which leads to unrealistic or uncontrollable video synthesis. To overcome the limitations, we propose a 3D-aware generative network along with a hybrid embedding module and a non-linear composition module. Through modeling the head motion and facial expressions1 explicitly, manipulating 3D animation carefully, and embedding reference images dynamically, our approach achieves controllable, photo-realistic, and temporally coherent talking-head videos with natural head movements. Thoughtful experiments on several standard benchmarks demonstrate that our method achieves significantly better results than the state-of-the-art methods in both quantitative and qualitative comparisons. The code is available on https://github.com/ lelechen63/Talking-head-Generation-with-Rhythmic-Head-Motion.
1 Introduction
The paper targets lip-synced talking-head generation with natural, controllable head movements by explicitly separating head motion, facial expressions, and appearance. It addresses limitations of still talking-face methods and guidance-dependent moving-head methods through 3D-aware generation and reference-video modeling.
- Head movements convey nonverbal information during speech and matter for applications including hearing assistance, virtual characters, security, and supervised learning.
- Existing audio-driven methods generate lip-synced faces but omit head-motion modeling, while landmark or image-driven methods produce moving heads without controllable expressions.
- The proposed method decomposes talking-head deformation into intrinsic subject traits, head movements, and facial expressions to make motion generation controllable.
- A short reference video supplies head-motion characteristics, while K sampled reference images provide appearance information for generating future frames from arbitrary audio.
- The framework combines a hybrid embedding network, 3D modeling, and nonlinear composition to aggregate references and reduce discontinuities or artifacts during apparent motion.
- Experiments on VoxCeleb2 and LRS3-TED under audio-driven and landmark-driven settings report that the framework addresses limitations of existing methods.
2 Related Work
Prior talking-head systems either depend on person-specific video retrieval, generate only limited facial regions, or struggle to preserve identity and handle deformation. The paper therefore introduces a hybrid embedding network that dynamically aggregates appearance from reference images with head movements.
- Some prior systems generate only a small facial region and cannot disentangle head motion from facial movement.
- Graphics-based methods can produce accurate lip synchronization but require extensive footage of a specific person and do not generalize to unseen subjects.
- Conventional reference encoders can harm identity preservation through deep convolutional layers, motivating dynamic or bilinear alternatives.
- The proposed hybrid embedding network aggregates appearance information from reference images with apparent head movements.
3 Method
The method maps reference frames and driving audio to future talking-head frames by separately learning facial expressions and head motion, then combining them with appearance information. Its generator uses 3D-aware pose manipulation, hybrid reference embedding, and nonlinear composition.
- The system takes sampled target-subject video frames and driving audio, then synthesizes future frames conveying the audio with realistic head movements.
- The full model decouples facial expression learning, head-motion learning, and frame synthesis into separate sub-models.
- The framework overview organizes generation around facial-expression and head-motion learners plus 3D-aware, hybrid-embedding, and nonlinear-composition modules.
- The facial expression learner uses audio and a subject-specific reference image to predict PCA components representing facial expressions.
- The head-motion learner disentangles reference motion in 3D geometry space and predicts future head movements from driving audio and the disentangled motion.
- The 3D-aware generator fuses head motion and facial expressions with reference appearance to synthesize target frames.
4 3D-Aware Generation
The 3D-aware generation pipeline reconstructs and projects a subject-specific 3D face, dynamically embeds reference images, and composes aligned inputs to generate target frames. Its nonlinear composition module is designed to reduce artifacts from spatial misalignment, especially during apparent head motion.
- 3D-Aware Module: A frontal reference frame is unprojected into a textured 3D mesh, whose topology remains fixed across the video.The mesh is rendered with target poses using a differentiable rasterizer.
- 3D-Aware Module: The motion matcher selects a reference frame with the nearest background by comparing head-movement geometry costs.The selected frame is passed to the nonlinear composition module, and training perturbs selection costs for robustness.
- Hybrid Embedding Module: The hybrid embedding network dynamically aggregates K reference features using activation, image, landmark, and fusion encoders.Activation maps approximate similarity between query and reference frames, while fusion combines image and landmark features.
- Non-Linear Composition Module: The composition module combines a warped projected image, a warped matched image, and generator features through parallel SPADE blocks.The decoder repeats this nonlinear combination at every layer before producing the output frame.
- Non-Linear Composition Module: A nonlinear combination replaces direct image matting because misalignment can create visible artifacts during videos with apparent head motion.The ablation results report greater robustness to spatial misalignment, particularly under apparent head motion.
- Objective Function: Training uses multi-scale discriminators together with feature matching, perceptual, and flow losses to optimize the generator.The loss weights control the relative importance of the loss terms.
5 Experiments Setup
The experiments evaluate the approach across LRW, VoxCeleb2, and LRS3-TED using audio-to-video and landmark-to-video settings, with quantitative and qualitative comparisons.
- Datasets: Evaluation uses LRW, VoxCeleb2, and LRS3-TED, following the datasets’ established data splits.LRW contains 500 spoken words, VoxCeleb2 contains over 1 million utterances from 6,112 celebrities, and LRS3-TED includes more diverse head movements.
- Evaluation Protocol: Table 1 compares audio-to-video methods on LRW and VoxCeleb2, with the proposed model trained from scratch.Leading scores are bolded in the table.
- Evaluation Protocol: Table 2 compares landmark-to-video methods on LRS3-TED and VoxCeleb2, with method suffixes indicating the number of target-subject reference frames K.The models in the table are trained from scratch.
- Implementation Details: Training uses ADAM with learning rate 2 × 10^-4, β1 = 0.5, β2 = 0.999, K values of 1, 8, and 32, and τ = 64.Training on VoxCeleb2 and LRS3-TED takes five days on eight 32GB V100 GPUs; GRID and CREMA training takes less than one day.
- Qualitative Comparisons: Figure 7 compares the model with Vougioukas et al. on CREMA-D after VoxCeleb2 pretraining and CREMA-D finetuning.The orange dashed box marks reference images.
6 Results and Analysis
The method generates lip-synced videos while preserving identity and moving the head, outperforming comparison methods across visual, quantitative, ablation, and user-study evaluations.
- Metrics: The evaluation measures fidelity with FID, low-level similarity with SSIM, and identity preservation with CSIM.CSIM uses cosine similarity between face-recognition embeddings to measure identity mismatch.
- Visual Comparisons: The model produces lip-synced frames while preserving identity information on LRS3-TED and VoxCeleb2 testing sets.The comparison is illustrated in Figure 9.
- Visual Comparisons: Compared with visual-driven methods, the approach moves the head accurately, maintains accurate lip motion, and performs especially well under target-reference deformation.The authors attribute this advantage to the 3D-aware module, which provides rough geometry information; the method achieves the best performance on most evaluation metrics and higher CSIM.
- Qualitative Comparisons: Figure 7 exposes that Vougioukas et al. generates only facial regions at a fixed scale because of intrinsic limitations.The figure shows the raw network output.
- Ablation Studies: Removing the 3D-aware module sharply reduces identity preservation, while removing nonlinear composition or warping introduces artifacts near face edges or eyes.The authors attribute the identity effect to aligned facial texture and the artifacts to alignment issues caused by head movements.
- User Studies: Human evaluations find that the proposed methods outperform comparison methods in synchronization and authenticity across audio- and landmark-conditioned videos.The study covers LRW, VoxCeleb2, and LRS3-TED testing videos.
- User Studies: Videos with learned head movements receive better user-study performance than videos without head motion, indicating a preference for natural head movements over still faces.The comparison is reported for the proposed videos with head movements and those without head motion.
7 Conclusion and Discussion
The paper concludes that explicit modeling and 3D-aware generation produce photo-realistic talking-head videos with natural head movements, while acknowledging failures under extreme poses and omitted factors.
- Conclusion: The approach explicitly models head motion and facial expressions to synthesize talking-head videos with natural head movements.Its 3D-aware, hybrid embedding, and nonlinear composition modules support photo-realistic video synthesis.
- Limitations: The model struggles with extreme poses when the reference frames provide no visual clues, and it omits camera motion, lighting conditions, and audio noise.These factors may affect synthesis performance.
Supplemental Materials
The supplemental materials detail the head motion learner, facial expression learner, and unprojection network, while noting that reproduced comparison results may differ from the original methods.
- Supplemental Materials: The supplementary comparison results were replicated by the authors and may be better for other methods once their code becomes publicly available.The authors state that they plan to update those results when comparison-method code is available.
- Head Motion Learner: The head motion learner contains an encoder, extrapolator, and discriminator for predicting future head motion from audio and reference motion.The discriminator compares real and generated motion sequences using their mean and standard deviation.
- Facial Expression Learner: The facial expression learner uses audio chunks and reference landmark PCA components to predict current facial expressions.Twenty PCA coefficients represent facial expression, and the coefficients are reconstructed into 3D landmark points during inference.
- Unprojection Network: The unprojection network predicts position-map images from RGB reference images using an encoder-decoder trained on 300W-LP.The training data provides images across viewing angles with annotated 3DMM coefficients for generating 3D point clouds.
B More Results
Additional results include testing on Barack Obama’s weekly address footage and demonstrate controllable generation of head motion and facial expression.
- Controllable Videos: The model generates controllable videos with desired head motion and facial expressions on the VoxCeleb2 dataset.The result is presented as a testing example in Fig. 16.
- Unprojector: The unprojector uses the method proposed in [12].Fig. 14 shows the detailed unprojection network.
- Obama Footage: Fig. 15 presents two example testing results on President Barack Obama’s weekly address footage.The figure reports testing results on that footage dataset.
B.2 Test on President Barack Obama Footage Dataset
The authors fine-tuned the model on five Obama weekly-address videos and used the remaining videos for testing, reporting two example results.
- Test on President Barack Obama Footage Dataset: The model was fine-tuned with K = 8 on five videos from the President Barack Obama’s weekly address footage dataset.The remaining videos were reserved as the testing set.
- Test on President Barack Obama Footage Dataset: Fig. 15 shows two example testing results from the Obama footage dataset.The table passage identifies the associated ablation-study dataset separately as VoxCeleb2.
B.3 Ablation Studies
The ablation studies examine four model components and compare generated-video realism and temporal coherence through human evaluation and controllable-results visualization.
- B.3 Ablation Studies: Human subjects compare the generated videos with three prior methods for realism and temporal coherence.Samples come from LRW, VoxCeleb2, and LRS3 testing data.
- B.3 Ablation Studies: The controllable-results visualization changes target facial expressions while preserving head motion, or changes target head motion while preserving facial expression.The upper and lower portions of Fig. 16 present these two manipulation settings.