Source-linked AI summary
Identity-Preserving Talking Face Generation with Landmark and Appearance Priors
Weizhi Zhong, Chaowei Fang, Yinqi Cai, Pengxu Wei, Gangming Zhao, Liang Lin, Guanbin Li
TL;DR
Person-generic talking-face generation must produce realistic, lip-synchronized videos while preserving identity without target-speaker training videos. The paper proposes a two-stage landmark-generation and video-rendering framework with landmark and appearance priors, and reports better realism, lip synchronization, and identity preservation than existing person-generic methods.
Problem
Person-generic methods struggle to generate realistic, lip-synchronized videos that preserve identity, while person-specific methods require target-speaker videos for training or fine-tuning.
Method
A two-stage framework generates audio-conditioned lip and jaw landmarks using facial priors, then renders video frames using aligned reference appearance, landmarks, and audio features.
Results
The method produces more realistic, lip-synced, and identity-preserving videos than existing person-generic talking-face generation methods.
Takeaways & Limitations
Combining personalized landmark priors with aligned appearance references supports person-generic talking-face generation across realism, synchronization, and identity objectives.
Takeaways & Limitations
Audio-to-landmark prediction remains ambiguous because the same pronunciation can correspond to multiple facial shapes, and reference-image coverage and alignment remain important constraints.
Abstract
from arXiv · showhide
Generating talking face videos from audio attracts lots of research interest. A few person-specific methods can generate vivid videos but require the target speaker's videos for training or fine-tuning. Existing person-generic methods have difficulty in generating realistic and lip-synced videos while preserving identity information. To tackle this problem, we propose a two-stage framework consisting of audio-to-landmark generation and landmark-to-video rendering procedures. First, we devise a novel Transformer-based landmark generator to infer lip and jaw landmarks from the audio. Prior landmark characteristics of the speaker's face are employed to make the generated landmarks coincide with the facial outline of the speaker. Then, a video rendering model is built to translate the generated landmarks into face images. During this stage, prior appearance information is extracted from the lower-half occluded target face and static reference images, which helps generate realistic and identity-preserving visual content. For effectively exploring the prior information of static reference images, we align static reference images with the target face's pose and expression based on motion fields. Moreover, auditory features are reused to guarantee that the generated face images are well synchronized with the audio. Extensive experiments demonstrate that our method can produce more realistic, lip-synced, and identity-preserving videos than existing person-generic talking face generation methods.
1. Introduction
The paper targets person-generic talking-face generation by completing a speaker’s lower-half face coherently with audio while preserving realism and identity. Its two-stage framework combines personalized landmark prediction with appearance-guided video rendering, and experiments report improvements over existing methods.
- 1. Introduction: The method addresses person-generic talking-face generation, where videos must remain audio-coherent, visually realistic, and identity-preserving without target-speaker retraining.Person-specific methods may require target-speaker videos for retraining or fine-tuning, motivating the person-generic setting.
- 1. Introduction: Audio-to-landmark prediction is difficult because identical pronunciations can correspond to multiple facial shapes and prior methods may inadequately model audio–landmark correlations or long-range temporal dependencies.The paper motivates Transformer-based modeling partly by limitations of simple feature fusion and LSTM-based temporal modeling.
- 1. Introduction: A single reference image may miss facial details, while unaligned multiple references can hinder meaningful feature extraction and introduce synthesis artifacts.The paper therefore uses multiple references with spatial alignment during rendering.
- 1. Introduction: The proposed framework first generates audio-conditioned lip and jaw landmarks using prior facial-outline information, then renders the lower-half face from those landmarks.Pose prior landmarks and reference landmarks help align predictions with the speaker’s facial outline.
- 1. Introduction: The rendering model combines landmarks, prior visual appearance, and auditory features to produce realistic, identity-preserving, and lip-synchronized frames.Static references are aligned to the target pose and expression through a motion-field-based alignment module.
- 1. Introduction: Extensive experiments on LRS2 and LRS3 show superiority over existing methods in realism, identity preservation, and lip synchronization.The supplied introduction reports the overall benchmark comparison without quantitative values.
2. Related Work
Related work divides audio-driven talking-face generation into person-specific and person-generic approaches. Person-specific methods can achieve high fidelity but require target-speaker videos, while landmark-based methods provide an intermediate route for generic generation.
- 2. Related Work: Person-specific methods can synthesize high-fidelity talking-face videos but require target-speaker videos for retraining or fine-tuning.This requirement may be inaccessible in real-world scenarios.
- 2. Related Work: Landmark-based methods commonly learn mappings from audio to facial landmarks before synthesizing talking-face imagery.Examples include recurrent models that predict mouth landmarks and then synthesize mouth texture.
- 2. Related Work: The proposed landmark-to-video renderer combines prior appearance information, landmarks, and audio, achieving better realism, lip synchronization, and identity preservation than existing methods.This comparison is stated as the paper’s reported performance advantage over prior methods.
3. Proposed Method
The method uses a two-stage pipeline: a Transformer-based audio-to-landmark generator predicts lip and jaw motion, followed by landmark-to-video rendering that combines aligned reference appearance, masked target-face content, and audio features.
- 3.1. Audio-To-Landmark Generation: The framework first predicts lip and jaw landmarks from audio, reference landmarks, and pose-prior landmarks, then combines them with pose information to construct target sketches.The generator uses audio, pose, and reference embeddings with Transformer modules to model their relationships and predict landmarks over adjacent frames.
- 3.1.2 Loss Function for Landmark Generation: The landmark generator applies L1 landmark reconstruction and continuity regularization to constrain accuracy and temporal smoothness.The losses compare predicted and ground-truth lip and jaw landmarks and their frame-to-frame changes.
- 3.2. Landmark-To-Video Rendering: The rendering stage assembles predicted and pose-prior landmarks into target sketches and uses neighboring sketches to predict each target face image with temporal context.For each frame, 2k + 1 target sketches are supplied to the rendering process.
- 3.2.1 Reference Images Warping: An alignment module estimates motion fields that warp multiple reference images and their features to the target head pose and expression, while learned weights aggregate them.The alignment module encodes reference sketches and predicts per-reference spatial weighting for aggregation.
- 3.2.2 Sketch-To-Face Translation: A translation module converts target sketches and the lower-half masked target face into the final face image using warped reference content and auditory features.Audio features are injected through AdaIN layers, while warped images and features are incorporated through feature modulation.
- 3.2.2 Sketch-To-Face Translation: During inference, the generated full face is composited with the original background through a Gaussian-smoothed face mask to reduce background artifacts.The post-processing step addresses small portions of background that may be included in the generated face.
4. Experiments
Experiments on LRS2 and LRS3 compare the method with person-generic baselines using quantitative metrics, user ratings, visualizations, and ablations. The method achieves stronger visual quality, identity preservation, and lip synchronization, while its Transformer, alignment, audio, and multi-reference components each contribute measurable gains.
- Talking Face Reconstruction: 10.09% and 8.05% larger CSIM than Wav2Lip on LRS2 and LRS3, respectively, while achieving the best visual-quality metrics and smaller LipLMD.These results are reported for talking-face reconstruction in Table 1.
- Video Dubbing: The method achieves a comparable SyncScore to PC-AVS and a much higher SyncScore than other methods in video dubbing.Because dubbed videos lack ground truth, SyncScore is used to assess audio-visual synchronization.
- User Study: 47.6%, 10.1%, and 23.4% higher MOS than the second-best method for image quality, lip synchronization, and identity preservation, respectively.The user study involved 25 participants rating generated videos from 1 to 5.
- Visualization of Generated Images: Qualitative comparisons show fewer artifacts, more ground-truth-like images, and better mouth-shape consistency than the compared methods.The baselines include EAMM, PC-AVS, Wav2Lip, MakeItTalk, and ATVGnet.
- Effectiveness of Transformer Encoder: The Transformer-based landmark generator outperforms its bidirectional-LSTM variant because it better models temporal dependencies and landmark-audio relationships.This ablation replaces the Transformer with an LSTM and averages reference embeddings.
- Ablation Study: Removing reference-image warping or audio features worsens visual-quality and lip-synchronization metrics, while using 5–25 reference images outperforms using one.The ablations support both aligned references and reused audio features, and show gains from multiple references during inference.
5. Conclusion
The proposed two-stage person-generic method combines transformer-based landmark generation with reference-guided rendering to produce realistic, lip-synced, identity-preserving talking face videos.
- The method combines audio-to-landmark generation with landmark-to-video rendering for person-generic talking face generation.