Source-linked AI summary
LipNet: End-to-End Sentence-level Lipreading
Yannis M. Assael, Brendan Shillingford, Shimon Whiteson, Nando de Freitas
TL;DR
Lipreading is difficult because visual speech is ambiguous, and prior end-to-end systems mainly performed word classification rather than sentence-level prediction. LipNet maps variable-length video sequences to text using spatiotemporal convolutions, recurrent networks, and CTC. On GRID, it achieved 95.2% sentence-level word accuracy and outperformed reported human and prior word-level baselines.
Problem
Visual speech is difficult to disambiguate without context, while existing end-to-end lipreading models generally perform word rather than sentence-level sequence prediction.
Method
LipNet performs character-level, end-to-end sentence prediction with spatiotemporal convolutional networks, recurrent networks, and CTC.
Results
95.2% sentence-level word accuracy was achieved on GRID with an overlapped-speaker split, compared with 86.4% for the previous aligned word-classification result.
Takeaways & Limitations
LipNet demonstrates end-to-end sentence-level lipreading without hand-engineered visual features or a separately trained sequence model.
Takeaways & Limitations
The authors identify larger datasets as a future direction for improving LipNet’s performance.
Abstract
from arXiv · showhide
Lipreading is the task of decoding text from the movement of a speaker's mouth. Traditional approaches separated the problem into two stages: designing or learning visual features, and prediction. More recent deep lipreading approaches are end-to-end trainable (Wand et al., 2016; Chung & Zisserman, 2016a). However, existing work on models trained end-to-end perform only word classification, rather than sentence-level sequence prediction. Studies have shown that human lipreading performance increases for longer words (Easton & Basala, 1982), indicating the importance of features capturing temporal context in an ambiguous communication channel. Motivated by this observation, we present LipNet, a model that maps a variable-length sequence of video frames to text, making use of spatiotemporal convolutions, a recurrent network, and the connectionist temporal classification loss, trained entirely end-to-end. To the best of our knowledge, LipNet is the first end-to-end sentence-level lipreading model that simultaneously learns spatiotemporal visual features and a sequence model. On the GRID corpus, LipNet achieves 95.2% accuracy in sentence-level, overlapped speaker split task, outperforming experienced human lipreaders and the previous 86.4% word-level state-of-the-art accuracy (Gergen et al., 2016).
1 INTRODUCTION
Lipreading is difficult because visual speech is ambiguous and requires spatiotemporal context, while prior end-to-end systems largely stopped at word classification. LipNet addresses this gap with end-to-end sentence prediction and achieves strong GRID-corpus results.
- Motivation: Human lipreading is difficult without context because visually similar phonemes are commonly confused.Fisher identified five commonly confused visual-phoneme categories among 23 initial consonant phonemes.
- Motivation: 17±12% and 21±11% accuracy were achieved by hearing-impaired people on limited sets of monosyllabic and compound words, respectively.These results motivate automated lipreading, which has applications including hearing aids, silent dictation, and speech recognition in noisy environments.
- Research gap: Existing deep lipreading systems generally perform word classification rather than sentence-level sequence prediction.Machine lipreading also requires extracting features that capture both position and motion.
- Contribution: LipNet combines spatiotemporal convolutional networks, recurrent networks, and CTC for character-level, end-to-end sentence prediction.The model maps video-frame sequences to text without separating visual-feature learning from sequence prediction.
- Results: 95.2% sentence-level word accuracy was achieved on GRID with an overlapped-speaker split, versus 86.4% for the previous aligned word-classification result.LipNet also achieved 88.6% accuracy across unseen speakers.
- Results: 1.69× higher accuracy than hearing-impaired lipreaders was reported on the same GRID sentences, where the human average was 52.3%.The comparison used hearing-impaired people who can lipread.
- Analysis: LipNet’s few erroneous predictions occurred mostly within viseme groups, where visual context can remain insufficient for disambiguation.Saliency visualisation also showed attention to phonologically important video regions.
2 RELATED WORK
Prior lipreading research used handcrafted pipelines, HMMs, or deep models restricted to word and phrase classification. LipNet extends end-to-end learning to sentence-level sequence prediction using CTC, evaluated on the sentence-level GRID corpus.
- Automated lipreading: Earlier automated lipreading systems commonly relied on handcrafted image or video features and temporal preprocessing.Examples include optical flow, movement detection, and other engineered vision pipelines.
- Automated lipreading: Goldschen et al. first demonstrated visual-only sentence-level lipreading with HMMs on a limited dataset using hand-segmented phones.Neti et al. later combined hand-engineered visual features with audio in sentence-level audiovisual speech recognition.
- Automated lipreading: Gergen et al. reported 86.4% speaker-dependent accuracy on GRID using LDA-transformed mouth-region features in an HMM/GMM system.Speaker generalisation and motion-feature extraction remained open problems that LipNet addresses.
- Classification with deep learning: Deep-learning lipreading approaches before LipNet generally classified words or phonemes rather than complete sentence sequences.They included multimodal representation learning and deep features used within traditional HMM or GMM-HMM pipelines.
- Classification with deep learning: Chung and Zisserman’s spatiotemporal models underperformed their spatial models by around 14% on average and could not handle variable sequence lengths.Their work evaluated word classification and did not attempt sentence-level sequence prediction.
- Classification with deep learning: Other deep models addressed phrase or word classification, speaker-dependent settings, or separately trained visual and recurrent components.Reported examples include 10-phrase classification, no speaker independence, and frozen VGG features before RNN training.
- Datasets: The GRID corpus provides 34 speakers, 1,000 sentences per speaker, 28 hours of recordings, and 34,000 sentences for sentence-level evaluation.Its simple six-category grammar yields 64,000 possible sentences, and it was selected because it is sentence-level and has the most data.
3 LIPNET
LipNet maps variable-length video-frame sequences to text through spatiotemporal convolutions, bidirectional recurrent processing, and CTC-trained sequence prediction. Its architecture combines visual feature extraction, temporal aggregation, and character-sequence output end-to-end.
- LipNet maps variable-length sequences of video frames to text sequences and is trained end-to-end.
- Spatiotemporal convolutions process video by convolving across time and spatial dimensions, extending conventional spatial convolution.
- Two bidirectional GRUs aggregate the convolutional features so each hidden state depends on inputs from all time-steps.
- CTC removes the need for input-target alignment by marginalising over token sequences that collapse to the same label sequence.The CTC probability is computed efficiently with dynamic programming.
- The architecture stacks three spatiotemporal convolution, dropout, and spatial max-pooling blocks before two Bi-GRUs, linear outputs, softmax, and CTC loss.The softmax vocabulary includes a special CTC blank token.
4 LIPREADING EVALUATION
LipNet is evaluated on GRID using unseen- and overlapped-speaker splits, human and architectural baselines, and error-based metrics. Results emphasize the value of spatiotemporal convolution, recurrent aggregation, and language-model decoding, while analyses examine learned attention and viseme confusions.
- Evaluation setup: LipNet is evaluated on GRID using unseen- and overlapped-speaker splits, with augmentation applied only from the GRID corpus.The unseen-speaker split holds out four speakers, while the overlapped-speaker split evaluates a subset of each speaker’s sentences.
- Baselines: The evaluation compares LipNet with three hearing-impaired lipreaders and three ablation models: Baseline-LSTM, Baseline-2D, and Baseline-NoLM.The ablations vary the architecture’s recurrent or convolutional components and disable language-model decoding in one condition.
- Metrics: WER and CER measure the minimum normalized insertions, substitutions, and deletions needed to transform predictions into ground-truth text.Predictions are produced using CTC beam search; WER is usually equivalent to classification error here because most errors are substitutions.
- Performance: 11.4% WER is achieved by LipNet on unseen speakers, 2.3× lower than Baseline-2D’s 26.7% WER.LipNet also achieves 2.4× lower error than Baseline-2D on overlapped speakers, supporting the combination of STCNNs and RNNs.
- Performance: 52.8% and 26.3% WER are obtained by Baseline-LSTM on unseen and overlapped speakers, respectively, the lowest performance among the compared systems.Its sentence-level performance is lower than the reported word-level result for the replicated architecture.
- Representation analysis: LipNet’s saliency maps emphasize phonologically important articulatory regions, while most erroneous phoneme predictions occur within visemes.Examples include attention to tongue and jaw movement in “lay” and articulatory changes in “please”; some viseme distinctions remain visually ambiguous.
5 CONCLUSIONS
The paper presents LipNet as an end-to-end model mapping mouth-image sequences directly to complete sentences without hand-engineered visual features or a separately trained sequence model. Its evaluation reports strong performance, while the authors identify larger datasets as an important direction for improvement.
- Conclusions: LipNet maps sequences of mouth images to entire sentences end-to-end, eliminating word segmentation and separate training of visual features or a sequence model.The model is presented as the first deep-learning approach in the paper to perform this sentence-level mapping end-to-end.
- Conclusions: LipNet’s evaluation highlights the importance of spatiotemporal feature extraction and temporal aggregation, with 4.8% WER on GRID.The reported WER is 2.8× lower than the word-level state of the art, and performance is 4.1× better than the human lipreading baseline.
- Future work: The authors suggest that more data may improve performance and propose applying LipNet to larger sentence-level datasets.They also aim to extend the approach to jointly trained audiovisual speech recognition for robustness in noisy environments.
A ARCHITECTURE DETAILS
The appendix provides additional details about LipNet’s implementation and architecture.
- Architecture details: The appendix contains supplementary implementation details.
- Architecture details: The appendix contains supplementary architecture details.
- Architecture details: These details are provided outside the main paper sections.
A.1 IMPLEMENTATION
The implementation uses Torch, CTC libraries, specified initialization and dropout choices, Adam optimization, and decoder settings with a character 5-gram language model.
- Software: LipNet is implemented with Torch, warp-ctc, and Stanford-CTC’s decoder.
- Optimization: The network uses He initialization, orthogonal initialization for square GRU matrices, and channel-wise dropout after pooling layers.The stated dropout rate is p = 0.5.
- Decoding: CTC beam search uses beam width 200, α = 1, β = 1.5, and a character 5-gram binarized language model.
A.2 LIPNET ARCHITECTURE
This section specifies LipNet’s architecture hyperparameters and conventions for representing time, channels, feature dimensions, and spatial dimensions. It also clarifies how spatiotemporal kernels are ordered and how layers after the Bi-GRU operate.
- Table 3 defines T as time, C as channels, F as feature dimension, H and W as height and width, and V as vocabulary size including the CTC blank.
- Spatiotemporal convolution kernels are ordered according to the input dimensions, with the three kernel dimensions depending on the number of channels and kernel specification.
- Layers following the Bi-GRU are applied independently at each timestep.
A.3 BASELINE-LSTM ARCHITECTURE
The Baseline-LSTM reproduces Wand et al.’s setup and training procedure while using a two-layer LSTM configuration with specified preprocessing, dropout, and initialization choices.
- Baseline-LSTM replicates Wand et al. (2016) and is trained in the same way as LipNet.
- The model uses two LSTM layers with 128 neurons, grayscale input frames, and 50×25px down-sampling.
- Dropout is set to p = 0, and parameters are initialized uniformly between [−0.05, 0.05].
B PHONEMES AND VISEMES
This section presents the phoneme-to-viseme clustering used in the analysis and provides LipNet’s full phoneme confusion matrix.
- Table 4 reports the phoneme-to-viseme clustering of Neti et al. (2000).
- The phoneme confusion matrix complements the viseme clustering used for analyzing LipNet’s prediction confusions.
- Figure 4 presents LipNet’s full phoneme confusion matrix.