Source-linked AI summary
A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild
K R Prajwal, Rudrabha Mukhopadhyay, Vinay Namboodiri, C V Jawahar
TL;DR
Lip-syncing arbitrary identities in dynamic, unconstrained videos remains difficult. Wav2Lip uses a lip-sync expert and new evaluation framework, achieving accuracy that matches real synced videos.
Problem
Existing methods struggle to synchronize arbitrary identities in dynamic, unconstrained videos with target speech amid pose, scale, and illumination changes.
Method
Wav2Lip trains video generation with a powerful lip-sync discriminator and introduces rigorous benchmarks, SyncNet-based metrics, and the ReSyncED dataset.
Results
Wav2Lip achieves lip-sync accuracy matching real synced videos, while human evaluators preferred its videos over existing methods and unsynced versions more than 90% of the time.
Takeaways & Limitations
The proposed benchmarks, metrics, and real-world dataset provide a framework for judging lip synchronization in unconstrained videos.
Abstract
from arXiv · showhide
In this work, we investigate the problem of lip-syncing a talking face video of an arbitrary identity to match a target speech segment. Current works excel at producing accurate lip movements on a static image or videos of specific people seen during the training phase. However, they fail to accurately morph the lip movements of arbitrary identities in dynamic, unconstrained talking face videos, resulting in significant parts of the video being out-of-sync with the new audio. We identify key reasons pertaining to this and hence resolve them by learning from a powerful lip-sync discriminator. Next, we propose new, rigorous evaluation benchmarks and metrics to accurately measure lip synchronization in unconstrained videos. Extensive quantitative evaluations on our challenging benchmarks show that the lip-sync accuracy of the videos generated by our Wav2Lip model is almost as good as real synced videos. We provide a demo video clearly showing the substantial impact of our Wav2Lip model and evaluation benchmarks on our website: \url{cvit.iiit.ac.in/research/projects/cvit-projects/a-lip-sync-expert-is-all-you-need-for-speech-to-lip-generation-in-the-wild}. The code and models are released at this GitHub repository: \url{github.com/Rudrabha/Wav2Lip}. You can also try out the interactive demo at this link: \url{bhaasha.iiit.ac.in/lipsync}.
CCS CONCEPTS
The paper concerns computer vision, learning from critiques, and phonology/morphology.
- The work is categorized under computer vision, learning from critiques, and phonology/morphology.
KEYWORDS
The paper centers on lip synchronization, video generation, and talking face generation.
- The paper addresses lip sync.
- The paper concerns video generation.
- The paper concerns talking face generation.
1 INTRODUCTION
The paper targets speaker-independent lip synchronization for arbitrary identities and speech in unconstrained talking-face videos, where existing methods struggle with varied lip shapes and real-world visual changes. It introduces Wav2Lip, a stronger lip-sync discriminator, and rigorous benchmarks and metrics for evaluation.
- Limitations of Existing Methods: Existing speech-to-lip methods achieve strong quality for specific trained speakers but do not readily support generic identities and speech inputs.Earlier approaches learned speech-to-landmark mappings from single speakers, while newer methods directly generated images for specific speakers.
- Limitations of Existing Methods: Speaker-independent models struggle to morph diverse lip shapes in unconstrained videos without additional speaker-specific data.This difficulty is compounded by rapid pose, scale, and illumination changes and the need to blend generated faces seamlessly into target videos.
- Proposed Approach: Existing approaches inadequately penalize incorrect lip shapes because they rely on reconstruction losses or weak lip-sync discriminators.The paper adapts a powerful lip-sync discriminator to enforce consistently accurate and realistic lip motion.
- Evaluation Framework: The paper introduces an evaluation framework with new benchmarks and metrics, including ReSyncED, a real-world dataset for testing lip-sync models on completely unseen content.These resources are intended to enable fair evaluation of lip synchronization in unconstrained videos.
- Contributions: The authors propose Wav2Lip for more accurate lip-syncing of arbitrary talking-face videos with arbitrary speech in the wild.The model is presented as significantly more accurate than previous works for this setting.
2 RELATED WORK
Prior talking-face methods are often restricted to specific speakers or vocabularies, while only a few works address lip-syncing arbitrary identities, voices, and languages. Speaker-specific approaches also require substantial per-speaker data and cannot generalize to new identities or voices.
- Speaker- and vocabulary-constrained generation: Recent methods generate realistic talking-face videos for Barack Obama by mapping input audio to corresponding lip landmarks.Because they are trained on a single speaker, these methods do not synthesize new identities or voices.
- Speaker- and vocabulary-constrained generation: Speaker-specific video-editing methods can add or remove speech phrases but still require about an hour of data per speaker.
- Lip-syncing arbitrary identities: Only two prominent works are identified as lip-syncing arbitrary identities, voices, and languages without restricting training to a small identity set or vocabulary.This enables lip-syncing random identities for any speech at test time.
3 ACCURATE SPEECH-DRIVEN LIP-SYNCING FOR VIDEOS IN THE WILD · 3.1 Pixel-level Reconstruction loss is a Weak Judge of Lip-sync · 3.2 A Weak Lip-sync Discriminator
The paper attributes inaccurate lip-sync in unconstrained videos to inadequate loss functions and introduces a design centered on learning from a well-trained lip-sync expert. Whole-image reconstruction underweights lips, while a temporal-context-aware expert discriminator substantially outperforms LipGAN’s discriminator at detecting off-sync pairs.
- 3 ACCURATE SPEECH-DRIVEN LIP-SYNCING FOR VIDEOS IN THE WILD: The core architecture generates accurate lip-sync by learning from a well-trained lip-sync expert.
- 3 ACCURATE SPEECH-DRIVEN LIP-SYNCING FOR VIDEOS IN THE WILD: Existing architectures use inadequate L1 reconstruction and discriminator losses to penalize inaccurate lip-sync generation.
- 3.1 Pixel-level Reconstruction loss is a Weak Judge of Lip-sync: Less than 4% of the total reconstruction loss corresponds to the lip region because the face reconstruction loss covers the whole image.The loss also optimizes pose, identity, and background reconstruction.
- 3.1 Pixel-level Reconstruction loss is a Weak Judge of Lip-sync: Whole-image reconstruction prioritizes surrounding image optimization before fine-grained lip-shape correction.The network begins morphing lips only at around half-w, according to the supplied passage.
- 3.2 A Weak Lip-sync Discriminator: 91% accuracy was achieved by the expert discriminator when detecting off-sync audio-lip pairs on the LRS2 test set.
- 3.2 A Weak Lip-sync Discriminator: 56% accuracy was achieved by LipGAN’s discriminator on the same off-sync audio-lip detection task and test set.LipGAN’s discriminator uses a single frame, whereas a small temporal context is helpful for detecting lip-sync.
3.3 A Lip-sync Expert Is All You Need
The method adapts a pre-trained SyncNet into an expert lip-sync discriminator that guides accurate lip generation without fine-tuning on noisy generated frames. The modified expert achieves 91% accuracy on LRS2, compared with 56% for LipGAN’s discriminator.
- Expert discriminator: The method adapts a pre-trained SyncNet as an expert lip-sync discriminator and avoids further fine-tuning it on generated frames.The discriminator is trained to detect synchronization accurately before guiding lip generation.
- Expert discriminator: SyncNet judges synchronization from consecutive lower-half face frames and an audio segment by distinguishing aligned from mismatched temporal windows.It uses separate face and audio encoders built from stacked 2D convolutions and computes their L2 distance.
- Expert discriminator: The adapted discriminator uses color images, a deeper residual architecture, and cosine-similarity with binary cross-entropy loss.These changes tailor SyncNet to the lip-generation task.
- Expert discriminator: 91% accuracy is achieved by the expert discriminator on the LRS2 test set, versus 56% for LipGAN’s discriminator.Training uses approximately 29 hours of LRS2 data, batches of 64, Tv = 5 frames, Adam, and an initial learning rate of 1e−3.
3.4 Generating Accurate Lip-sync by learning from a Lip-sync Expert
The generator is trained with a frozen, pre-trained lip-sync expert discriminator to penalize inaccurate lip movements. It combines identity, speech, and face-decoding blocks while generating contiguous temporal windows for expert-based discrimination.
- Generator architecture: The generator’s key contribution is training a LipGAN-like architecture with an expert discriminator; it contains identity encoder, speech encoder, and face decoder blocks.The identity encoder processes a random reference frame concatenated with a masked target-face pose prior.
- Temporal training: Because the generator produces frames independently, training samples contiguous windows of Tv = 5 frames and generates all frames in each window.This sampling promotes temporal consistency of pose across the discriminator’s temporal input.
- Temporal training: Generated time-steps are stacked along the channel dimension into the expert discriminator, which receives only the lower half of the generated face.The discriminator input has shape (N,H/2,W , 3·Tv).
- Expert-guided training: The expert discriminator remains frozen, and its lip-sync-based loss forces the generator toward realistic synchronization.The discriminator’s strength comes from lip-sync concepts learned from real videos, minimizing the lip-sync loss Esync.
3.5 Generating Photo-realistic Faces
The model combines a frozen lip-sync discriminator with a visual-quality discriminator to improve synchronization while reducing blur and artifacts. Its objective balances reconstruction, synchronization, and adversarial losses, while inference preserves source pose through a masked face-crop prior.
- Discriminator Design: A visual-quality discriminator is trained alongside the lip-sync discriminator to mitigate blur and artifacts caused by enforcing accurate lip shapes.The two discriminators target sync accuracy and visual quality, respectively.
- Training Objective: The generator minimizes a weighted sum of reconstruction, synchronization, and adversarial losses, with synchronization and adversarial weights set to 0.03 and 0.07.Ltotal = (1 − sw − sд) · Lrecon + sw · Esync + sд · Lдen.
- Training Setup: Training uses only the LRS2 train set with batch size 80 and Adam optimization, while the lip-sync discriminator remains frozen.The initial learning rate is 1e−4, with β1 = 0.5 and β2 = 0.999 for the generator and visual-quality discriminator.
- Inference: During inference, the model generates videos frame-by-frame using the current face crop and a lower-half-masked copy as a pose prior, avoiding pose changes.The corresponding audio segment is also provided to the speech sub-network.
4 QUANTITATIVE EVALUATION
The evaluation section shows that prior speaker-independent lip-sync benchmarks are inconsistent, non-reproducible, and poorly suited to temporal or fine-grained synchronization assessment. It introduces SyncNet-based metrics and consistent benchmarks for unconstrained videos, then evaluates Wav2Lip quantitatively and with human judgments.
- Evaluation limitations: Prior evaluation randomly changes reference frames, forcing pose changes unlike real-world inference and failing to measure the intended use case.This also makes test data inconsistent across works, hindering fair comparison and reproducibility.
- Evaluation limitations: Random reference frames destroy temporal consistency by introducing arbitrary poses and scales, preventing evaluation of temporal-consistency methods.The framework therefore cannot support a metric for this aspect of lip-syncing.
- Evaluation limitations: SSIM, PSNR, and LMD do not reliably capture fine-grained lip-sync errors, motivating metrics designed specifically for synchronization.LMD can be unreliable because lip landmarks on generated faces may be inaccurate.
- Proposed evaluation: The proposed Lip-Sync Error-Distance and Lip-Sync Error-Confidence metrics use pretrained SyncNet to measure synchronization in arbitrary video-audio pairs.Higher confidence indicates better audio-video correlation, while lower confidence indicates severely out-of-sync portions.
- Proposed evaluation: The new protocol forms consistent pseudo-random audio-video test sets with 14K LRS2 pairs, 28K LRW pairs, and 14K LRS3 pairs, covering frontal and profile views.The toolkit is intended for consistent and reliable benchmarking in the wild.
- Experiments: Wav2Lip is evaluated with LSE-D, LSE-C, and FID on three test splits, plus quantitative and human evaluation on unconstrained real-world videos.The human study uses 14 evaluators and assesses sync accuracy, visual quality, overall experience, and preference.
5 APPLICATIONS & FAIR USE
Wav2Lip supports large-scale audiovisual translation and creation, including lip-syncing lecture videos to dubbed local-language speech. Because near-realistic lip-syncing for arbitrary identities and voices may be misused, the authors require synthetic results to be clearly identified and promote detection research.
- Applications: Wav2Lip can lip-sync online English lectures to automatically dubbed speech in other local languages.The paper presents this as an example application for videos in the wild.
- Fair Use: Near-realistic lip-syncing for any identity and voice raises concerns about potential misuse.The authors emphasize fair use as lip-sync capabilities become increasingly applicable.
- Fair Use: Results created with the code and models must unambiguously present themselves as synthetic.This is the authors’ explicit recommendation for responsible use.
- Fair Use: The work’s complete open-source release is intended to encourage efforts to detect manipulated video content and its misuse.The authors frame detection research as a complementary outcome of releasing the code and models.
6 CONCLUSION
The work proposes accurate lip-synced video generation in the wild by addressing failures in unconstrained talking-face lip-syncing and revising quantitative evaluation. It argues that a pre-trained, accurate lip-sync expert can enforce accurate, natural lip motion.
- The work proposes a novel approach for generating accurate lip-synced videos in the wild.
- It identifies two major reasons current approaches are inaccurate on unconstrained talking-face videos.
- A pre-trained, accurate lip-sync expert is proposed to enforce accurate, natural lip-motion generation.
- The work re-examines the current quantitative evaluation framework and highlights several major issues.