Source-linked AI summary
Combining Residual Networks with LSTMs for Lipreading
Themos Stafylakis, Georgios Tzimiropoulos
TL;DR
Word-level visual speech recognition must handle target words embedded in continuous, variable broadcast speech without word-boundary information. The paper addresses this with an end-to-end network combining spatiotemporal convolution, a ResNet, and bidirectional LSTMs, achieving 83.0% word accuracy and a 6.8-point absolute improvement over the current state of the art.
Problem
The LRW task requires recognizing 500 target words embedded in whole utterances, including confusable viseme pairs, without word boundaries.
Method
The paper combines a spatiotemporal convolutional front-end, a per-time-step ResNet, and a bidirectional LSTM back-end trained end-to-end with aggregated per-time-step loss.
Results
83.0% word accuracy, a 6.8 absolute improvement over the current state of the art, is achieved on LRW without word-boundary information.
Takeaways & Limitations
The proposed network demonstrates strong word-level visual speech recognition on an in-the-wild benchmark where target words occur within continuous utterances.
Abstract
from arXiv · showhide
We propose an end-to-end deep learning architecture for word-level visual speech recognition. The system is a combination of spatiotemporal convolutional, residual and bidirectional Long Short-Term Memory networks. We train and evaluate it on the Lipreading In-The-Wild benchmark, a challenging database of 500-size target-words consisting of 1.28sec video excerpts from BBC TV broadcasts. The proposed network attains word accuracy equal to 83.0, yielding 6.8 absolute improvement over the current state-of-the-art, without using information about word boundaries during training or testing.
1. Introduction
The paper presents an end-to-end deep architecture for word-level visual speech recognition, combining convolutional, residual, and bidirectional recurrent components. It targets recognition and implicit spotting of words embedded in fixed-duration utterances without word-boundary information.
- Motivation: Visual speech recognition complements audio-based recognition by supporting dictation in noisy environments and silent dictation in public or office settings.The field also has applications in hearing aids and biometric authentication.
- Scope: The paper situates its word-modeling approach as more pertinent to isolated-word recognition, classification, and detection than to sentence-level or large-vocabulary continuous recognition.The proposed system can support viseme-level recognition by replacing word labels at the SoftMax layer.
- Architecture: The proposed system combines spatiotemporal convolution, a Residual Network applied at each time step, and a two-layer bidirectional LSTM.A SoftMax layer produces outputs at all time steps, and the aggregated per-time-step loss supports end-to-end training.
- Task setting: The system performs implicit key-word spotting because target words occur inside whole 1.28sec utterances rather than isolated clips.Word boundaries are not used during training or evaluation.
- Paper scope: The paper evaluates the proposed model on the LRW database and presents its model, preprocessing, implementation, and experimental comparisons in subsequent sections.The paper’s later sections cover related work, the database, model details, and baseline and state-of-the-art results.
2. Related work
Lipreading research moved from hand-engineered features and HMM-based pipelines toward deep feature extractors and end-to-end architectures. Prior work includes fully recurrent and sentence-level systems, while WAS reports strong word accuracies on GRID and LRW.
- Traditional methods: Earlier lipreading systems primarily used hand-engineered features modeled with HMM-based pipelines.Spatiotemporal descriptors and SVM classifiers were also investigated before deep learning became dominant.
- Deep learning methods: Deep learning approaches have used deep feature extraction or end-to-end architectures for visual and audio-visual speech recognition.Examples include Deep Belief Networks, bottleneck features with an LSTM back-end, and fully LSTM-based models.
3. Database
The LRW database contains challenging in-the-wild audiovisual speech clips from BBC broadcasts, with 500 target words embedded in continuous utterances. Its variability, confusable visemes, and unknown word boundaries make recognition also a spotting problem.
- Database characteristics: LRW contains audiovisual speech segments extracted from BBC TV broadcasts, with high variability in speakers and pose.The database includes News and Talk Shows among its broadcast sources.
- Database characteristics: The database has 500 target words, approximately an order of magnitude more than other cited public databases such as GRID and CUAVE.Its target-word vocabulary is substantially larger than those comparison databases.
- Recognition difficulty: Many target-word pairs share most of their visemes, including singular-plural nouns and present-past verbs.Examples include benefit-benefits and allow-allowed.
- Recognition difficulty: Target words appear within utterances rather than in isolation, so the network must ignore irrelevant speech and spot targets without word boundaries.Example utterances include surrounding words before and after the target.
- Dataset construction: Each clip lasts 1.28sec and contains 31 frames sampled at 25fps.The training set contains up to 1000 occurrences per target word, while validation and evaluation each contain 50.
4. Deep Learning modeling and preprocessing
The model processes mouth-region frame sequences through spatiotemporal convolution, a per-time-step ResNet, and a bidirectional LSTM back-end. Training uses repeated word labels across time steps and an aggregated loss, with staged initialization followed by end-to-end optimization.
- Preprocessing: Facial landmarks guide cropping and resizing of every clip frame to a fixed 112×112 mouth-region representation.A common crop is applied across a clip using median landmark coordinates.
- Front-end: The front-end applies 3D convolution with 64 kernels of size 5×7×7 to capture short-term mouth-region dynamics.The convolutional layers are followed by batch normalization and ReLU activations.
- Residual network: A 34-layer identity-mapping ResNet processes the 3D feature maps separately at each time step.Skip connections facilitate information propagation while pooling progressively reduces spatial dimensionality.
- Recurrent back-end: The back-end stacks two LSTMs in each direction of a bidirectional network and concatenates the final directional outputs.The LSTM back-end contains approximately 2.4M parameters.
- Optimization criterion: Applying the word-level criterion at every time step improves word accuracy by about 3% absolute over applying it only at the final time step.The overall loss aggregates the negative logarithms of word posteriors across time steps.
- Training procedure: Training begins with a temporal-convolutional back-end, then attaches and briefly trains the Bi-LSTM before optimizing the entire network end-to-end.The staged procedure keeps the convolutional front-end and ResNet fixed during the initial Bi-LSTM training phase.
5. Experiments
Experiments compare progressively richer network configurations, showing gains from spatiotemporal modeling, residual representations, Bi-LSTM back-ends, and end-to-end training. The best configuration reaches 83.0% word accuracy, while errors remain associated with closely related phonetic or visemic content and coarticulation.
- Baseline and component comparisons: 8.5% absolute improvement over the VGG-M baseline is achieved by the simplest proposed system.The baseline is the multi-tower VGG-M network, and word accuracy is evaluated using Top-1 and Top-N measures.
- Baseline and component comparisons: 5.0% absolute improvement results from replacing 2D with 3D convolution, emphasizing short-term mouth dynamics.The comparison is between configurations N1 and N2.
- Baseline and component comparisons: 4.9% better word accuracy is obtained with the ResNet than with a similarly sized three-layer DNN.The DNN contains approximately 20M parameters, comparable to the ResNet comparison.
- LSTM back-end comparisons: Using a single-layer Bi-LSTM instead of temporal convolution improves accuracy by 3.8%, while a second Bi-LSTM layer adds 1.2%.The LSTM comparisons use configurations N4 and N5, with the back-end trained while the front-end and ResNet remain fixed.
- Discussion and error analysis: The most frequent errors involve words with closely related phonetic or visemic content, especially when coarticulation obscures boundary visemes.The clips contain preceding and succeeding words because they are excerpts from continuous speech.
6. Conclusions
The proposed end-to-end spatiotemporal network combines a 3D convolutional front-end, ResNet, and LSTM-based back-end for word-level visual speech recognition. On LRW, it achieved 83.0% work accuracy and a 6.8% absolute improvement over the state-of-the-art.
- 6.8% absolute improvement over the state-of-the-art 76.2% accuracy was attained.The comparison state of the art was an attentional encoder-decoder network.
- The network stacks a 3D convolutional front-end, a ResNet, and an LSTM-based back-end.It was trained using an aggregated per time step loss and evaluated through end-to-end training.
- The experiments demonstrated the importance of each network building block and the performance gain from end-to-end training.