Source-linked AI summary
Lipreading using Temporal Convolutional Networks
Brais Martinez, Pingchuan Ma, Stavros Petridis, Maja Pantic
TL;DR
The paper addresses limited robustness and cumbersome training in isolated-word lip-reading models. It replaces BGRUs with TCNs, simplifies training, and adds variable-length augmentation, achieving new state-of-the-art performance on LRW and LRW1000.
Problem
Existing isolated-word lip-reading models use BGRU-based architectures and do not generalize well to variations in sequence length.
Method
The paper replaces BGRU layers with TCNs, uses one-stage cosine-scheduled training, and applies variable-length augmentation.
Results
1.2% on LRW and 3.2% on LRW1000 absolute top-1 accuracy improvements establish new state-of-the-art performance.
Takeaways & Limitations
The proposed model combines improved benchmark performance with simpler training and greater robustness to varying sequence lengths.
Abstract
from arXiv · showhide
Lip-reading has attracted a lot of research attention lately thanks to advances in deep learning. The current state-of-the-art model for recognition of isolated words in-the-wild consists of a residual network and Bidirectional Gated Recurrent Unit (BGRU) layers. In this work, we address the limitations of this model and we propose changes which further improve its performance. Firstly, the BGRU layers are replaced with Temporal Convolutional Networks (TCN). Secondly, we greatly simplify the training procedure, which allows us to train the model in one single stage. Thirdly, we show that the current state-of-the-art methodology produces models that do not generalize well to variations on the sequence length, and we addresses this issue by proposing a variable-length augmentation. We present results on the largest publicly-available datasets for isolated word recognition in English and Mandarin, LRW and LRW1000, respectively. Our proposed model results in an absolute improvement of 1.2% and 3.2%, respectively, in these datasets which is the new state-of-the-art performance.
1 Samsung AI Research Center, Cambridge, UK 2 Computing Department, Imperial College London, UK
The paper concerns visual speech recognition, lip-reading, and Temporal Convolutional Networks.
- Visual speech recognition is identified as a central topic.
- Lip-reading is identified as a central topic.
- Temporal Convolutional Networks are identified as a central topic.
1. INTRODUCTION
The paper situates lip-reading within visual speech recognition and prior temporal modeling approaches, then proposes TCNs, simpler training, and variable-length augmentation to improve isolated-word recognition.
- Lip-reading recognizes speech from lip movements without using the audio stream.It is particularly useful when audio is corrupted and can complement acoustic recognizers.
- Earlier systems used separate feature extraction and temporal modeling, while later deep-learning work adopted end-to-end architectures.Examples include DCT with HMMs, deep autoencoders with LSTMs, and recurrent or attention-based models.
- The prior isolated-word state-of-the-art combines 3D convolution, an 18-layer ResNet, BGRU layers, and softmax classification.It achieves state-of-the-art performance on LRW and LRW1000.
- The paper replaces BGRUs with TCNs, trains in one stage with a cosine scheduler, and introduces variable-length augmentation.Training time is reduced from 3 weeks to 1 week GPU-time, while augmentation targets sequences of varying length.
2. DATABASES
The study evaluates lip-reading on LRW and LRW1000, large in-the-wild datasets covering English and Mandarin with substantial visual variability.
- LRW and LRW1000 are the largest publicly available in-the-wild lip-reading datasets for English and Mandarin, respectively.
- LRW contains 1.16-second BBC-program segments, more than 1000 speakers, 500 word classes, and varied head pose and illumination.These properties make LRW challenging for isolated-word recognition.
- LRW videos have a fixed length of 29 frames.
- LRW1000 contains 1000 word classes and 718,018 samples totaling approximately 57 hours.The dataset also varies substantially in scale, resolution, and background clutter.
3. BACKGROUND
The baseline processes grayscale video with a 3D-convolutional ResNet feature encoder, then models the resulting temporal features using a bidirectional GRU and softmax layer through three training stages.
- The baseline methodology retains a video feature encoder but changes sequence-classification layers and substantially revises training.
- The network accepts a B×T×H×W grayscale video tensor and produces B×C×T features using a 3D-convolutional ResNet18 encoder.No temporal downsampling is used, and global spatial average pooling follows the final convolution.
- The encoded feature sequence is classified by a two-layer bidirectional GRU followed by a dense softmax layer.
- Baseline training uses three sequentially trained models, with each model initializing the next.The supplied passage describes the first model as using a single-layer TCN head before later BGRU training and fine-tuning.
4. PROPOSED METHODOLOGY
The methodology replaces recurrent sequence modeling with temporal convolutions, including a multi-scale variant, and simplifies training to enable training from scratch. Variable-length augmentation addresses overfitting to fixed 29-frame sequences.
- Temporal Convolutional Networks: TCNs replace recurrent layers as a convolutional alternative for encoding temporal feature sequences without altering sequence length.Dilated convolutions expand the receptive field, and the model uses a non-causal design because it classifies the whole sequence at once.
- Temporal Convolutional Networks: The multi-scale TCN mixes short- and long-term information by combining branches with different kernel sizes in every temporal convolution.With n branches, each branch uses C/n kernels and outputs are concatenated.
- Training strategy: The full convolutional model can be trained successfully from scratch with a cosine scheduler in 1 week of GPU-time.The authors also pretrain on the 10% hardest LRW words, which provides faster training and a small performance improvement with minimal overhead.
- Variable length augmentation: Variable-length augmentation randomly crops training sequences before and after target-word boundaries to reduce dependence on LRW’s fixed-length, centered-word bias.LRW sequences contain 29 frames and fixed positioning can make models sensitive to small input changes.
5. EXPERIMENTAL RESULTS
Experiments show improved isolated-word recognition on LRW and LRW1000, greater robustness to removed frames, and stronger audio-visual performance under noise. These gains are obtained with simpler training and comparatively lighter modeling choices.
- Comparison with the current State-of-the-Art: 1.2% and 3.2% absolute top-1 accuracy improvements establish new state-of-the-art results on LRW and LRW1000, respectively.The model also achieves 1.9% better top-1 accuracy than baseline.
- Fixed Length VS Variable Length Training: Variable-length training improves robustness to random frame removal, reaching a 35.80% absolute improvement over fixed-length training when five frames are removed.Using the full sequence, robustness augmentation trades performance at 84.60% versus 82.40% for the compared setting.
- Fixed Length VS Variable Length Training: The TCN with variable-length augmentation matches BGRU robustness to frame removals while achieving superior performance on the full test sequence.This combines the robustness benefit of variable-length training with the stronger full-sequence result reported for TCN.
- Audio-visual experiments: Under clean audio, audio-visual error is 1.04% versus 1.54% for audio-only; at 0 dB, the rates are 2.92% and 8.57%, respectively.At -5 dB, audio-visual error is 6.53% compared with 26.21% for audio-only.
- Audio-visual experiments: Audio-visual models show a clear gain over the baseline across different noise levels, especially as babble noise increases.The comparison covers audio-only, video-only, and audio-visual models under varying Signal to Noise Ratio levels.
6. CONCLUSIONS
The paper presents an improved isolated-word lip-reading model that combines TCNs, simpler training, and variable-length augmentation. It reports new state-of-the-art performance on LRW and LRW1000.
- Contributions: The proposed model replaces BGRUs with TCNs, simplifies training, and uses variable-length augmentation to improve varying-length generalization.The training process is reduced to a quicker single-stage procedure.
- Results: The model achieves new state-of-the-art performance on the LRW and LRW1000 datasets.