Source-linked AI summary
Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis
K R Prajwal, Rudrabha Mukhopadhyay, Vinay Namboodiri, C V Jawahar
TL;DR
Lip-to-speech synthesis must infer speech from ambiguous lip movements, while prior work lacks either long speaker-specific recordings or natural, large-vocabulary speech. The paper releases the Lip2Wav dataset and proposes a sequence-to-sequence model for individual speakers; it reports almost 4× greater intelligibility than previous works in unconstrained settings.
Problem
Prior lip-to-text datasets provide about two minutes per speaker, while lip-to-speech datasets are small, narrow-vocabulary, and artificially constrained.
Source: §1 · p. 2Method
Results
Almost 4× greater intelligibility than previous works is reported in unconstrained environments, with human evaluation also finding more natural speech with rich prosody.
Source: §1 · p. 2Takeaways & Limitations
Lip2Wav provides a public large-scale benchmark for evaluating single-speaker lip-to-speech synthesis in unconstrained settings.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Humans involuntarily tend to infer parts of the conversation from lip movements when the speech is absent or corrupted by external noise. In this work, we explore the task of lip to speech synthesis, i.e., learning to generate natural speech given only the lip movements of a speaker. Acknowledging the importance of contextual and speaker-specific cues for accurate lip-reading, we take a different path from existing works. We focus on learning accurate lip sequences to speech mappings for individual speakers in unconstrained, large vocabulary settings. To this end, we collect and release a large-scale benchmark dataset, the first of its kind, specifically to train and evaluate the single-speaker lip to speech task in natural settings. We propose a novel approach with key design choices to achieve accurate, natural lip to speech synthesis in such unconstrained scenarios for the first time. Extensive evaluation using quantitative, qualitative metrics and human evaluation shows that our method is four times more intelligible than previous works in this space. Please check out our demo video for a quick overview of the paper, method, and qualitative results. https://www.youtube.com/watch?v=HziA-jmlk_4&feature=youtu.be
1. Introduction
Lip-to-speech inference is ambiguous, and prior work lacks either sufficient speaker-specific data or natural, large-vocabulary speech. The paper introduces Lip2Wav, a person-specific dataset and sequence-to-sequence approach for unconstrained speech generation.
1. Introduction
- Lip-to-speech inference is ambiguous because only 25% to 30% of English is discernible through lip-reading alone, prompting reliance on speaker and context cues.Homophenes such as /p/, /b/, and /m/ have auditorily distinct sounds but similar lip shapes.Source: §1 · p. 1
- Prior lip-to-text datasets have thousands of speakers but about two minutes per speaker, while lip-to-speech work uses small datasets and narrow-vocabulary artificial settings.The authors motivate learning an individual speaker’s speech patterns through extended observation.Source: §1 · p. 2
- Lip2Wav releases 120 hours of unconstrained video from five speakers, with 800× more data per speaker than multi-speaker datasets and a vocabulary about 100× larger than single-speaker lip-to-speech datasets.The dataset and sequence-to-sequence model target accurate, natural speech for individual speakers.Source: §1 · p. 2
- The paper studies unconstrained, large-vocabulary silent-lip-to-speech generation and releases a person-specific dataset with no restrictions on head pose or sentence length.The data, code, and trained models are publicly released.Source: §1 · p. 2
- Almost 4× more intelligible speech than previous works is reported for the sequence-to-sequence approach in unconstrained environments.Human evaluations also find the generated speech more natural with rich prosody.Source: §1 · p. 2
2. Related Work
Prior lip-to-speech systems rely on short sequences, constrained datasets, or features that may not suit natural head motion. Lip-to-text and text-to-speech work provide contrasting approaches, while Lip2Wav adapts sequence-to-sequence text-to-speech modeling to face sequences.
2.1. Lip to Speech Generation
- Earlier systems generate LPC features from fewer than 15 frames, while the authors argue that this architecture poorly models head motion, silences, and large vocabularies.They also characterize low-dimensional LPC features as insufficient for information-rich, natural-sounding speech.Source: §2.1 · p. 2
- Prior lip-to-speech results primarily use GRID, with 56 tokens and minimal head motion, unlike the paper’s YouTube clips with natural speech variation and head movements.The authors present their work as the first study of this task in a large-vocabulary setting with thousands of words and sentences.Source: §2.1 · p. 2
2.2. Lip to Text Generation
- Open-vocabulary lip-to-text work uses transformer sequence-to-sequence models, but lip-to-speech must infer linguistic content and generate the target speaker’s rich prosody.The paper therefore focuses on single-speaker unconstrained settings and precise individual speaking styles.Source: §2.2 · p. 3
2.3. Text to Speech Generation
- Lip2Wav modifies Tacotron 2 to condition sequence-to-sequence speech generation on face sequences instead of text.The cited text-to-speech models generate mel spectrograms autoregressively using attention.Source: §2.3 · p. 3
3. Speaker-specific Lip2Wav Dataset
Existing datasets do not test unconstrained, single-speaker lip-to-speech synthesis: some are narrow-vocabulary and constrained, while others provide little data per speaker. Lip2Wav is designed as a natural-speech benchmark for this setting.
3. Speaker-specific Lip2Wav Dataset
- Existing datasets fall into two extremes: unconstrained open-vocabulary multi-speaker data with only 2–5 minutes per speaker, or single-speaker data with narrow vocabularies and artificial settings.Neither extreme tests unconstrained single-speaker lip-to-speech synthesis.Source: §3 · p. 3
- Lip2Wav contains about 120 hours of English talking-face video across five speakers, with about 20 hours and vocabularies exceeding 5,000 words per speaker.Videos come from online lectures and chess-analysis content; vocabulary counts include words occurring at least five times.Source: §3 · p. 3
- The dataset is intended as a benchmark for learning and evaluating accurate person-specific lip-to-speech models in natural settings.The authors characterize it as more unconstrained and realistic than GRID or TIMIT.Source: §3 · p. 3, Table 1 · p. 3
4. Lip to Speech Synthesis in the Wild
Lip2Wav maps face-video sequences to speech through a sequence-to-sequence architecture that predicts mel-spectrograms from encoded lip movements. Its design combines temporal visual features, autoregressive decoding, and training and context choices for ambiguous speech inference.
4. Lip to Speech Synthesis in the Wild
- Lip2Wav targets generating corresponding speech from face-image sequences with lip motion, using design choices for natural speech in unconstrained settings.The section introduces this task as lip-to-speech synthesis in the wild.Source: §4 · p. 3
4.1. Problem Formulation
- Unlike prior 2D-CNN decoders, Lip2Wav models each speech time-step conditioned on earlier speech and the input face sequence.Its jointly trained face encoder and attention-based speech decoder enable an implicit speech-level language model to help disambiguate homophenes.Source: §4.1 · p. 4
4.2. Speech Representation
- Lip2Wav generates melspectrograms, balancing the robotic sound of LPC features against the computational cost of generating 16,000 waveform samples per second.Audio is sampled at 16 kHz, with an 800-sample window, 200-sample hop, and 80 mel dimensions.Source: §4.2 · p. 4
4.3. Spatio-temporal Face Encoder
- A stack of 3D convolutions extracts spatiotemporal lip features while downsampling spatial dimensions and preserving the input sequence’s temporal length.The encoder outputs a D-dimensional vector for each of the T facial frames.Source: §4.3 · p. 4, Figure 3 · p. 4
4.4. Attention-based Speech Decoder
- The attention-based decoder adapts Tacotron 2 to generate melspectrograms from encoded face embeddings, with end-to-end training using L1 reconstruction loss.The loss compares generated and ground-truth melspectrograms.Source: §4.4 · p. 4
4.5. Gradual Teacher Forcing Decay
- Teacher forcing is gradually decayed after about 30K iterations to shift attention toward lip input and reduce implicit language-model overfitting to training vocabulary.The authors hypothesize that early teacher forcing helps the decoder learn an implicit speech-level language model for disambiguating homophenes.Source: §4.5 · p. 5
4.6. Context Window Size
- Lip2Wav uses a visual context window about 6× larger than prior works, a choice the authors report yields significantly more accurate speech.The larger context is intended to help disambiguate homophenes.Source: §4.6 · p. 5
5. Benchmark Datasets and Training Details
The experiments use constrained corpora for comparison and a new speaker-specific dataset, with video-separated train, validation, and unseen test splits. Training uses sampled short clips, while inference generates speech autoregressively and supports long lip sequences through overlapping windows.
5.1. Datasets
- Lip2Wav is trained on GRID, TCD-TIMIT, and all five speakers of its new speaker-specific dataset, using 90-5-5% train, validation, and unseen test splits.For the new dataset, splits use different videos so no part of the same video appears in both training and testing.Source: §5.1 · p. 5
5.2. Training Methodology and Hyper-parameters
- Training examples are randomly sampled contiguous 3-second clips, with face crops resized to 48×48 and batch size 32; small datasets use half the hidden dimension.For unconstrained single-speaker experiments, convergence was achieved in about 200K iterations; Adam uses an initial learning rate of 10^-3.Source: §5.2 · p. 5
5.3. Speech Generation at Test Time
- At inference, Lip2Wav generates speech autoregressively from lip movements and handles any sequence length by concatenating consecutive windows with a small overlap.The generated melspectrogram is converted to waveform with Griffin-Lim; the authors report neural vocoders perform poorly with their less accurate melspectrograms.Source: §5.3 · p. 5
6. Experiments and Results
The experiments compare Lip2Wav with prior methods on constrained and unconstrained single-speaker datasets, then assess human judgments and word-level multi-speaker synthesis. Across these evaluations, the authors report improved intelligibility and speech quality, while identifying homophenes as a continuing source of errors.
6. Experiments and Results
- Evaluation uses objective speech quality and intelligibility metrics, WER where transcripts are available, and human assessments; WER is not computed on Lip2Wav because it lacks text transcripts.The GRID and TCD-TIMIT results use unseen test splits, and the Lip2Wav dataset splits separate videos across training, validation, and test.Source: §6 · p. 5
6.1. Lip to Speech in Constrained Settings
- On GRID and TCD-TIMIT, Lip2Wav outperforms competing methods across all objective metrics, with a particularly visible difference on TCD-TIMIT examples containing novel words.The authors attribute the TCD-TIMIT result to learning correlations across short phoneme sequences and pronouncing new words better than previous methods.Source: §6.1 · p. 6, Table 2 · p. 6, Table 3 · p. 6
6.2. Lip to Speech in Unconstrained Settings
- Almost 4× more intelligible speech is achieved by Lip2Wav than previous methods in unconstrained single-speaker settings.The comparison uses the five-speaker Lip2Wav dataset and reports speech intelligibility and quality metrics.Source: Table 4 · p. 6, §6.2 · p. 6
6.3.1 Objective Human Evaluation
- Human evaluation finds fewer mispronunciations than the current state-of-the-art method, three times fewer word skips, and homophenes as a dominant error source.The objective evaluation manually assessed mispronunciations, word skips, and homophene-based errors in 50 test files.Source: §6.3.1 · pp. 6–7, Table 5 · p. 6
6.3.2 Subjective Human Evaluation
- Participants rated Lip2Wav higher in quality and legibility than the previous state-of-the-art, while even fully accurate lip-to-text output followed by TTS did not achieve the desired prosody and naturalness.The comparison included manually transcribed text followed by multi-speaker TTS, as well as human speech for reference.Source: §6.3.2 · p. 7, Table 6 · p. 7
6.4. Multi-speaker Word-level Lip to Speech
- The experiments report word-level multi-speaker lip-to-speech synthesis on LRW, where Lip2Wav outperforms the lip-to-text baseline without text-level supervision.The speech metrics for LRW word-level synthesis are not directly comparable with the single-speaker results, which include variable-length sequences and pauses.Source: Table 7 · p. 7, §6.4 · p. 7
7. Ablation Studies
The ablations examine context size, attention alignment, teacher forcing, and visual encoders using unseen-test predictions from one Lip2Wav speaker. The reported findings favor larger context, gradually decayed teacher forcing, and a 3D-CNN encoder, while visualizations show attention to the mouth and corresponding lip frames.
7. Ablation Studies
- The ablation results use unseen-test predictions from the “Hardware Security” speaker in the Lip2Wav dataset.Source: §7 · p. 7
7.1. Larger context window helps in disambiguation
- A 3-second visual context window improves speech accuracy by helping disambiguate visemes through phoneme and word co-occurrences; its size is limited by memory constraints.The context is close to 6× the 0.3–0.5 seconds used in previous works.Source: §7.1 · p. 7, Table 8 · p. 8
7.2. Model is Highly Attentive to the Mouth
- The encoder activations emphasize the speaker’s mouth, and decoder alignment indicates conditioning on the corresponding video frame’s lips during speech generation.Figures 4 and 5 visualize the encoder activations and decoder alignment, respectively.Source: §7.2 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8
7.3. Teacher Forcing vs Non-Teacher Forcing
- Gradually decaying teacher forcing from approximately 30K iterations improves test scores and helps prevent overfitting to the training vocabulary.The authors report that the decay encourages the model to use visual input rather than predict only from previously uttered speech.Source: §7.3 · p. 8, Table 9 · p. 8
7.4. Effect of Different Visual Encoders
- Among the tested visual encoders, the 3D-CNN achieves the best performance while the speech decoder is held fixed.The stated role of the 3D-CNN is to capture spatial and temporal information in unconstrained settings.Source: §7.4 · p. 8, Table 10 · p. 8
8. Conclusion
The work frames unconstrained, large-vocabulary single-speaker lip-to-speech synthesis as a sequence-to-sequence task and reports more accurate, natural speech than previous methods. It identifies speaker-specific lip-to-text modeling, homophene handling, and vocabulary generalization as future directions.
8. Conclusion
- The sequence-to-sequence approach produces significantly more accurate and natural speech than previous methods.The work evaluates the model with quantitative metrics and human studies.Source: §8 · p. 8
- The work publicly releases its code and data.Source: §8 · p. 8
- Future work could study lip-to-text generation from a speaker-specific perspective and explicitly address homophenes.The authors also identify generalization beyond a speaker’s typical vocabulary domain as a fruitful direction.Source: §8 · p. 9