Source-linked AI summary
Lipreading with Long Short-Term Memory
Michael Wand, Jan Koutník, Jürgen Schmidhuber
TL;DR
The paper addresses purely visual speech recognition by replacing conventional feature extraction and classification with an end-to-end neural network. Its stacked feed-forward and LSTM architecture achieves almost 80% word accuracy and improves over conventional feature-based SVM systems.
Problem
Purely visual speech recognition requires systems to recognize speech from visual information such as a speaker’s lips and face.
Method
A neural network chains feed-forward and LSTM layers to process raw mouth images, replacing manual feature extraction and conventional classification.
Results
11.6% improvement in word accuracy over the best conventional solution, HOG + SVM, was achieved on evaluation speakers 10–19.
Takeaways & Limitations
The LSTM lipreader consistently achieved almost 80% word accuracy in speaker-dependent lipreading while learning features automatically.
Takeaways & Limitations
The evaluation was speaker-dependent, while speaker-independent LSTM lipreading remained future work; CNN experiments also did not improve results.
Abstract
from arXiv · showhide
Lipreading, i.e. speech recognition from visual-only recordings of a speaker's face, can be achieved with a processing pipeline based solely on neural networks, yielding significantly better accuracy than conventional methods. Feed-forward and recurrent neural network layers (namely Long Short-Term Memory; LSTM) are stacked to form a single structure which is trained by back-propagating error gradients through all the layers. The performance of such a stacked network was experimentally evaluated and compared to a standard Support Vector Machine classifier using conventional computer vision features (Eigenlips and Histograms of Oriented Gradients). The evaluation was performed on data from 19 speakers of the publicly available GRID corpus. With 51 different words to classify, we report a best word accuracy on held-out evaluation speakers of 79.6% using the end-to-end neural network-based solution (11.6% improvement over the best feature-based solution evaluated).
1. INTRODUCTION
Lipreading research aims to recognize speech from visual information alone. This paper replaces manual feature extraction and conventional classification with a compact neural network using raw mouth images and LSTM sequence modeling.
- Lipreading enables purely visual speech recognition by processing visual information from a speaker’s lips and face.
- Conventional lipreading systems typically combine feature extraction with classification, often using SVMs or HMMs.
- The proposed compact neural network chains feed-forward and LSTM layers to replace the complete visual speech recognition pipeline.
- The network takes raw mouth images as input, eliminating the need for manual feature extraction.
2. RELATED WORK
Prior work used lipreading both alongside noisy audio recognition and for purely visual speech recognition, with applications including Silent Speech interfaces. Neural networks increasingly replace hand-designed processing stages, while CNNs have also been applied to lipreading.
- Lipreading supports both audio-visual recognition in noisy conditions and purely visual speech recognition.
- Silent Speech interfaces enable communication when an audible acoustic signal is unavailable, including for some speech impairments and confidential communication.
- Neural networks, particularly LSTMs, have progressively replaced larger portions of speech-recognition pipelines previously dominated by HMMs.
- CNNs trained by gradient descent dominate image recognition and have been used as preprocessors for HMM-based lipreading systems.
3. THE GRID DATA CORPUS
The study uses visual data from the GRID audiovisual corpus, whose fixed-format sentences provide 51 word classes. Speakers 1–19 are split into development and evaluation groups under speaker-dependent testing.
- The GRID corpus contains recordings of 34 speakers saying 1,000 fixed-structure sentences each, totaling 28 hours.
- Word-level segmentation produces 6,000 single-word samples per speaker across 51 different words.
- Each sentence lasts 3 seconds at 25 frames per second, yielding 75 frames; the acoustic recordings were not used.
- A 40 × 40 pixel mouth window is localized, contrast-normalized, and converted to grayscale for model input.
- Speakers 1–9 form the development set and speakers 10–19 the held-out evaluation set, but all experiments remain speaker-dependent.
4. METHODS
The method compares an LSTM lipreader with SVM classifiers using Eigenlips or HOG features. Neural networks process inputs through layered feed-forward and recurrent units, whereas SVM baselines require fixed-length sequence feature vectors.
- Baseline Feature Extraction and Classification: The neural lipreader is compared with SVM classifiers using Eigenlips and HOG conventional features.
- Baseline Feature Extraction and Classification: Eigenlips apply PCA to training images, while HOG accumulates local gradient-orientation histograms across spatial cells.
- Baseline Feature Extraction and Classification: SVM baselines convert variable-length word-frame sequences into fixed-length vectors by repeating or averaging frames.
- Results: Word accuracy is averaged over development speakers 1–9 and evaluation speakers 10–19 for the compared systems.
- Feed-forward networks pass signals forward, whereas recurrent networks provide temporal memory through cyclic connections.
5. EXPERIMENTS
Experiments compared Eigenlips- and HOG-based SVM systems with an LSTM lipreader, finding a significant held-out-speaker advantage for the neural approach while revealing much higher confusion on letters than longer words.
- Experimental Setup: The LSTM lipreader was compared with Eigenlips and HOG features, each combined with an SVM classifier.Parameters were optimized on development speakers 1–9, and errors were reported on speaker-dependent test sets.
- Experimental Setup: The best SVM configuration used a linear kernel and a sequence feature vector length of 6 frames; longer vectors did not improve accuracy.The authors hypothesize that a dedicated sequence classifier would not substantially improve accuracy on this corpus.
- Results: 82.0% total accuracy was obtained for speaker 7, with 69.8% on letters and 93.4% on non-letter words.The corresponding chance levels were 4% for letters and 3.8% for non-letter words.
- Results: Letters remain difficult because visually similar sounds are poorly distinguished and their sequences often contain only 3–4 frames.Adjacent-sound influence further complicates letter recognition, while independent sentence parts provide little helpful context.
- Results: Longer words were recognized with close to 100% accuracy, whereas letters produced the highest confusion across speakers and experimental setups.The word-length pattern was also visible in the speaker 7 confusion matrix.
6. CONCLUSION
The study concludes that a neural lipreading system using raw mouth-region images outperformed a conventional feature-extraction pipeline. Its LSTM classifier achieved almost 80% speaker-dependent word accuracy, while CNN alternatives did not improve results.
- Conclusion: Almost 80% word accuracy was consistently achieved by the LSTM lipreader in speaker-dependent lipreading.The system learned features automatically while training the LSTM sequence classifier.
- Conclusion: The neural system achieved significantly better word accuracy than a conventional pipeline using feature extraction and classification.The neural network operated directly on raw images of mouth regions.
- Conclusion: CNNs did not improve results when substituted for the fully connected feed-forward layer.These experiments were not described in detail in the paper.
- Conclusion: Future experiments will examine CNNs with larger image sizes and speaker-independent LSTM lipreading.A possible explanation offered for the lack of CNN improvement is that 40 × 40 pixels already contain enough classification information.