Source-linked AI summary
Large-Scale Visual Speech Recognition
Brendan Shillingford, Yannis Assael, Matthew W. Hoffman, Thomas Paine, Cían Hughes, Utsav Prabhu, Hank Liao, Hasim Sak, Kanishka Rao, Lorrayne Bennett, Marie Mulville, Ben Coppin, Ben Laurie, Andrew Senior, Nando de Freitas
TL;DR
Visual speech recognition has been limited by narrow vocabularies and relatively small datasets. This paper introduces a large-scale dataset and scalable lipreading system, achieving a 40.9% WER on a held-out set versus 86.4% or 92.9% for professional lipreaders.
Problem
Prior lipreading approaches were often limited to narrow vocabularies, relatively small datasets, and single-word classification.
Method
The paper constructs a 3,886-hour dataset and combines phoneme-based neural lipreading with production-grade word-level decoding.
Results
40.9% WER was achieved on a held-out set, compared with 86.4% or 92.9% for professional lipreaders and 89.8% or 76.8% for prior approaches.
Takeaways & Limitations
The combined system nearly halves the error rate of previous state-of-the-art methods on this dataset.
Takeaways & Limitations
The approach relies on a dominant-pronunciation assumption, which can create pronunciation–orthography discrepancies in some languages.
Abstract
from arXiv · showhide
This work presents a scalable solution to open-vocabulary visual speech recognition. To achieve this, we constructed the largest existing visual speech recognition dataset, consisting of pairs of text and video clips of faces speaking (3,886 hours of video). In tandem, we designed and trained an integrated lipreading system, consisting of a video processing pipeline that maps raw video to stable videos of lips and sequences of phonemes, a scalable deep neural network that maps the lip videos to sequences of phoneme distributions, and a production-level speech decoder that outputs sequences of words. The proposed system achieves a word error rate (WER) of 40.9% as measured on a held-out set. In comparison, professional lipreaders achieve either 86.4% or 92.9% WER on the same dataset when having access to additional types of contextual information. Our approach significantly improves on other lipreading approaches, including variants of LipNet and of Watch, Attend, and Spell (WAS), which are only capable of 89.8% and 76.8% WER respectively.
1 INTRODUCTION AND MOTIVATION
The work addresses lipreading’s narrow-vocabulary and small-dataset limitations by introducing a scalable visual speech recognition system motivated partly by assisting people with speech impairments. It combines a 3,886-hour dataset, phoneme-recognition modeling, and production-grade decoding to achieve 40.9% WER.
- Motivation and limitations: Lipreading approaches had often been limited to narrow vocabularies and relatively small datasets.Prior work also frequently focused on single-word classification.
- Motivation and limitations: Speech impairments motivate the work because visual speech recognition could benefit hundreds of thousands of patients worldwide.The paper cites 103,925 U.S. tracheostomies performed in 2014 as an example of a condition that can cause difficulty or inability to speak.
- System contributions: The proposed system transforms raw video into word sequences using the 3,886-hour Large-Scale Visual Speech Recognition dataset.The dataset pairs phoneme sequences with video clips of speaking faces and is distilled from YouTube videos.
- System contributions: The Vision to Phoneme network produces phoneme distributions from video frames, while production-grade word decoding enables vocabulary expansion without retraining the neural network.This separates phoneme prediction from word decoding under computational and memory constraints.
- Results: 40.9% WER was achieved on a held-out set, compared with 86.4% or 92.9% for professional lipreaders and 89.8% or 76.8% for prior approaches.The professional-lipreader values depend on the amount of context provided; the prior values correspond to LipNet and WAS variants, respectively.
2 RELATED WORK
Prior lipreading research largely emphasized single-word classification, while sentence-level systems introduced end-to-end sequence modeling on limited or smaller real-world datasets. V2P instead predicts phoneme distributions for decoder-based word recognition and uses a larger, more diverse training dataset designed for large vocabularies.
- Prior work: Earlier lipreading work largely focused on single-word classification and often relied on substantial prior knowledge.Recent traditional pipelines incorporated speech, vision, and machine-learning components.
- Deep learning advances: Generalization across speakers and motion-feature extraction were open problems until deep learning enabled progress, although many deep models remained focused on single-word classification.These approaches included visual-only, multimodal, and audiovisual representations.
- Sentence-level lipreading: LipNet was the first end-to-end sentence-level lipreading model, combining spatiotemporal convolutions, GRUs, and CTC-trained character prediction.On GRID’s 28 hours of 5-word sentences, it achieved 4.8% WER for overlapping speakers and 11.4% WER for unseen speakers.
- V2P: V2P predicts phoneme distributions that a decoder converts into words, enabling large vocabularies without retraining the deep network.The design is also memory- and computationally efficient and does not require pre-trained features.
- Dataset scale: The proposed data-processing pipeline produces a substantially larger and more diverse training dataset than previous lipreading efforts.Earlier large-vocabulary datasets included IBM ViaVoice, LRS, MV-LRS, and LRS3-TED, sourced from broadcasts or conference talks.
3 A DATA PIPELINE FOR LARGE-SCALE VISUAL SPEECH RECOGNITION
The LSVSR dataset is built from public YouTube videos through independent, large-scale parallel-processing modules and filters. The pipeline produces aligned lip-video and phoneme-sequence data by progressively filtering, tracking, canonicalizing, and verifying speech segments.
- Dataset construction: The pipeline uses large-scale parallel processing with independent modules and filters implemented on top of FlumeJava to extract data from public YouTube videos.This processing strategy is used to create the LSVSR dataset.
- Dataset construction: 140,000 hours of audio segments are paired with transcripts before post-processing yields 128 × 128 video frames and aligned phoneme sequences.Only about 2% of pre-selected YouTube clips satisfy the filtering criteria.
- Filtering and alignment: Segments are restricted to 1–12 seconds, non-English utterances are removed, short evaluation utterances are excluded, and phonemes are obtained by forced alignment.The forced alignment uses a lexicon with multiple pronunciations.
- Visual preprocessing: Video quality processing removes shot-boundary, blurry, small-face, and low-frame-rate clips, while FaceNet detects and tracks faces in remaining segments.The quality filter requires eye-to-eye width of at least 80 pixels and frame rates of at least 23fps.
- Visual preprocessing: Landmark smoothing, ±30° pose filtering, canonical-face alignment, lip cropping, speaking detection, and V2P-Sync verify and standardize speaking-face clips.V2P-Sync verifies audio-video alignment using longer segments and spatiotemporal convolutions.
4 AN EFFICIENT SPATIOTEMPORAL MODEL OF VISUAL SPEECH RECOGNITION
The V2P model combines a volumetric visual module, temporal aggregation into phoneme distributions, CTC training, and WFST decoding into word sequences. Modeling phonemes separates sound uncertainty from word uncertainty, allowing the decoder to handle pronunciation-to-word ambiguity.
- Model pipeline: V2P applies a 3D convolutional module to video, aggregates features temporally into phoneme distributions, trains with CTC, and decodes words using finite-state transducers.The decoder uses a combination of three weighted finite-state transducers.
- Neural network architecture: The vision module adapts VGG into a volumetric architecture to explicitly extract motion features from video.The volumetric adaptation proved crucial in preliminary empirical evaluation.
- Neural network architecture: The temporal module uses 3 bidirectional LSTMs with hidden state 768, group normalization, and an MLP producing exactly T conditionally independent phoneme distributions.Alternative recurrent and dilated-convolutional architectures were evaluated, with the presented architecture performing best.
- Connectionist temporal classification (CTC): CTC trains sequence distributions without requiring alignments between input video timesteps and label tokens.CTC computes sequence probability by marginalizing over possible alignments, including blank tokens needed to distinguish repeated labels.
- Rationale for phonemes and CTC: Phoneme targets avoid character-level CTC’s homophone problem by modeling sound uncertainty with the temporal network and word uncertainty with the decoder.Phonemes have a one-to-many mapping to words, whereas character sequences can map many-to-many to words.
5 EVALUATION
Evaluation on a held-out LSVSR test set shows that V2P substantially outperforms professional lipreaders and prior visual speech recognition systems. V2P also generalizes to LRS3-TED, outperforming a state-of-the-art model trained on that dataset.
- LSVSR test-set results: 37 minutes: The held-out LSVSR test set contains approximately 63,000 video frames and 7,100 words.The evaluation uses V2P trained on LSVSR with hyperparameters tuned on a validation set.
- Phoneme prediction analysis: Direct phoneme prediction avoids designing phoneme-to-viseme mappings by modeling inherent uncertainty in the predictive distribution.Edit-distance alignments are used to identify frequently inserted or missed phonemes.
- Cross-dataset generalization: V2P outperforms the TM-seq2seq model on LRS3-TED test-set accuracy despite no training or fine-tuning on that dataset.LRS3-TED includes face angles between ±90° and clips shorter than one second, unlike LSVSR.
6 CONCLUSIONS
The work presents a novel large-scale visual speech recognition system built around a dataset vastly exceeding previous approaches in vocabulary and example sequences. Its scalable model produces phoneme and word sequences from processed video and nearly halves prior state-of-the-art error rates on the dataset.
- 6 CONCLUSIONS: The system introduces a dataset an order of magnitude larger than previous approaches in vocabulary and example-sequence count.Its scale is described in terms of both vocabulary breadth and the sheer number of example sequences.
- 6 CONCLUSIONS: The scalable model produces phoneme and word sequences from processed video clips.It is integrated with the data-processing pipeline used to construct the large-scale dataset.
- 6 CONCLUSIONS: Nearly halving the error rate of previous state-of-the-art methods, the system achieves substantially stronger performance on this dataset.The conclusion characterizes the improvement relative to prior state-of-the-art methods evaluated on the same dataset.
A MEDICAL APPLICATIONS
The work proposes lip-reading technology as an alternative communication strategy for patients with aphonia or dysphonia who cannot speak effectively. Potential applications span acute hospital care and long-term community life, including communication by telephone and for patients with limited limb function.
- Medical need: Aphonia prevents voiced sound, while dysphonia disrupts speech despite retaining some speaking ability; both can follow laryngeal injury, disease, or treatment.Examples include thyroidectomy-related nerve damage, laryngectomy, tracheostomy, and spasmodic dysphonia.
- Acute care applications: 103,925 tracheostomies were performed in the U.S. in 2014, with an average hospital stay of 29 days, creating a substantial acute-care communication need.Unplanned procedures may leave patients unprepared for voice loss, while limb impairment can make writing difficult.
- Community applications: After discharge, lip-reading technology could support independence, social functioning, employment, and telephone communication for patients who remain unable to speak effectively.Many patients do not achieve functional spoken communication even after adapting to alternatives such as voice prostheses, electro-larynx, or esophageal speech.
- Acute care applications: In acute care, sudden inability to communicate can distress patients and contribute to poorer quality of care, especially after unplanned procedures.The section highlights patients who awaken unable to speak after events such as cervical spinal cord injury, sudden airway obstruction, or emergency tracheotomy.
- Community applications: For patients unable to speak or move their arms, the technology could enable faster communication than current eye-tracking or facial-muscle-based approaches.The proposed benefit depends on patients retaining the ability to move their facial muscles.
B PHONEME CONFUSION MATRIX
The phoneme confusion matrix is built from edit-distance alignments between predicted and ground-truth phoneme sequences. Correct phonemes dominate, while visually similar phonemes—especially /d/, /n/, and /t/—are commonly confused, and insertion/deletion counts identify frequent omissions and erroneous insertions.
- Alignment method: Edit-distance dynamic programming and backtracing align predicted phoneme sequences with ground truth, pairing edit operations with sequence positions.The alignments support both the confusion matrix and insertion/deletion analysis.
- Confusion patterns: The confusion matrix’s diagonal is strongly dominant, indicating that correct phoneme predictions substantially outnumber substitutions.The matrix is obtained by counting correct phonemes and substitutions after alignment.
- Confusion patterns: Visually similar phonemes are commonly confused, especially {/d/, /n/, /t/}, with {/b/, /p/} confused to a lesser extent.These groups reflect expected confusions arising from visual similarity.
- Insertion and deletion patterns: Insertion and deletion counts show which phonemes are most commonly omitted or, less frequently, erroneously inserted.These counts produce the insertion/deletion chart shown in the main text.
D V2P-SYNC ARCHITECTURE
V2P-Sync comprises separate video- and audio-embedding neural networks. The networks are optimized with Adam using batch normalization and a batch size of 128.
- D V2P-SYNC ARCHITECTURE: 128 is the batch size used to optimize the V2P-Sync networks.Optimization also uses batch normalization.
- D V2P-SYNC ARCHITECTURE: 10−4 is the Adam learning rate, with default momentum coefficients 0.9 and 0.999 and ϵ = 10−8.The optimizer uses default hyperparameters, including ϵ for numerical stability.
- D V2P-SYNC ARCHITECTURE: V2P-Sync includes distinct video embedding and audio embedding neural network architectures.These are specified in Tables 3 and 4, respectively.
E V2P ARCHITECTURE
The V2P architecture is trained with Adam and a curriculum that progressively increases video duration from 2 to 12 seconds. Image transformations are also used for augmentation.
- Optimization: Adam optimization uses a learning rate of 10^-4, momentum coefficients of 0.9 and 0.999, and ϵ = 10^-8.The momentum coefficients are the first and second coefficients, respectively, and ϵ supports numerical stability.
- Curriculum learning: The curriculum schedule increases the maximum video duration from 2 seconds to 12 seconds over 200,000 training steps.The schedule limits video duration to accelerate learning.
- Data augmentation: Image transformations are applied to augment the image input.
F FACE ROTATION VS. PERFORMANCE HEATMAP
V2P performs similarly across head pan and tilt rotations from −30° to 30°, matching the range used during training.
- Face rotation versus performance heatmap: V2P performs similarly at all pan and tilt angles in [−30°, 30°], the range at which it was trained.Figure 9 presents this result as a performance heatmap over head rotations.