Source-linked AI summary
Lip Reading Sentences in the Wild
Joon Son Chung, Andrew Senior, Oriol Vinyals, Andrew Zisserman
TL;DR
Lip reading remains difficult for unconstrained natural-language sentences in real-world video. This paper introduces WLAS and the LRS dataset, surpassing prior benchmark results, outperforming a professional lip reader, and improving speech recognition when audio is available.
Problem
The paper addresses open-world lip reading of unconstrained sentences in videos featuring talking faces in varied real-world conditions.
Method
The WLAS network uses dual attention across visual and audio inputs, curriculum learning, and the large-scale LRS dataset for character-level transcription.
Results
The model surpasses prior benchmark work, beats a professional lip reader, and reduces character error rate from 10.4% with audio alone to 7.9% with audio and lip reading.
Takeaways & Limitations
Visual speech information improves recognition both without audio and when noisy or clean audio is available.
Takeaways & Limitations
The sequence-to-sequence model decodes complete lip sequences in batch mode rather than operating online without future input.
Abstract
from arXiv · showhide
The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem - unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) a 'Watch, Listen, Attend and Spell' (WLAS) network that learns to transcribe videos of mouth motion to characters; (2) a curriculum learning strategy to accelerate training and to reduce overfitting; (3) a 'Lip Reading Sentences' (LRS) dataset for visual speech recognition, consisting of over 100,000 natural sentences from British television. The WLAS model trained on the LRS dataset surpasses the performance of all previous work on standard lip reading benchmark datasets, often by a significant margin. This lip reading performance beats a professional lip reader on videos from BBC television, and we also demonstrate that visual information helps to improve speech recognition performance even when the audio is available.
1. Introduction
The paper frames lip reading as an ambiguous but context-resolvable problem whose practical automation depends on deep sequence models and large-scale data. It introduces a character-level, dual-attention model and BBC-based sentence dataset for open-world visual speech recognition, while also studying audio-visual speech recognition.
- Motivation: Lip-reading predictions are inherently ambiguous because homophemes can share identical lip sequences, but sentence context and language models can resolve these ambiguities.This motivates character-level sequence modeling that incorporates linguistic context.
- Contributions: The LRS dataset enables open-world lip reading of unconstrained sentences from BBC broadcasts containing talking faces and subtitles.The dataset is built from thousands of hours of television, and performance is assessed on its test set and public benchmarks.
- Audio-visual speech recognition: The paper tests whether visual speech improves audio recognition by training on both modalities and disturbing audio or removing visual input.This addresses the established interest in audio-visual speech recognition, particularly in noisy environments.
- Contributions: The model outputs characters, learns a language model, and uses dual attention over visual input, audio input, or both.Its sequence-to-sequence encoder-decoder architecture follows approaches developed for speech recognition and machine translation [3] [5] [35].
- Contributions: The model surpasses prior work on public lip-reading benchmarks including LRW and GRID.The paper also demonstrates open-world sentence lip reading on the LRS dataset.
2. Architecture
The WLAS architecture combines visual and audio encoders with a character decoder using dual attention to transcribe talking-face videos with or without audio. Jointly trained modules support asynchronous or missing input streams, while attention is critical for reliable audio-visual recognition.
- Overall architecture: WLAS comprises Watch, Listen, and Spell, whose encoders produce states and output sequences that the decoder attends to for character prediction.The three modules are trained jointly, and Figure 1 shows the decoder producing separate visual and audio attention vectors at each output step.
- Watch and Listen: Watch uses a VGG-M-based convolutional network [6] followed by an LSTM, while Listen directly encodes 13-dimensional MFCC features in reverse time order.The visual ConvNet processes five grayscale mouth-centered frames, with its 512-dimensional fc6 vector feeding the LSTM.
- Spell: Spell extends an LSTM transducer [3] [5] [8] with independent visual and audio attention mechanisms whose fused context vectors drive character prediction.At the first output step, the final encoder states are concatenated to initialize context generation; a softmax MLP produces each character distribution.
- Discussion: Attention is critical: without it, the model largely forgets the input and produces outputs correlated with only the first one or two words.Those initial words are correct because they are the last words seen by the encoder.
- Spell: Dual attention extracts information from asynchronous visual and audio streams, including when one stream is absent or the streams are not time-aligned.Independent attention mechanisms address the different sampling rates of the lip and audio inputs and fuse their information into decoder context vectors.
- Discussion: Replacing the unidirectional Watch and Listen encoders with bidirectional LSTMs [5] [8] increased training time without obvious performance improvement.The authors attribute this to the decoder already being conditioned on the full input sequence.
3. Training strategy
The training strategy accelerates WLAS learning and improves generalisation by curriculum learning, scheduled sampling, balanced audio-visual inputs, supplementary unpaired sentences, and staged noise augmentation. Implementation details specify the multimodal inputs, recurrent architecture, optimisation, and training schedule.
- Curriculum learning: Curriculum learning starts with single-word segments and gradually increases sequence length, accelerating convergence, reducing overfitting, and substantially improving test performance.The short sequences are extracted from longer dataset sentences and act as natural data augmentation; the test improvement is reported in Section 5.
- Decoder training: Scheduled sampling [4] bridges teacher-forced training and inference by making the recurrent decoder tolerate erroneous previous predictions.During training, ground-truth tokens normally provide the next input, whereas inference must use the model’s own previous predictions.
- Multimodal training: Uniformly sampling audio-only, lip-only, or audio-and-lips inputs prevents the easier audio modality from dominating multimodal training.Audio recognition is easier than lip reading, motivating balanced exposure to the three input types.
- Supplementary data: Over 200,000 sentences without facetracks supplement training to boost audio recognition and enrich the language model for better generalisation.The recorded data contains over 300,000 sentences, but only around 100,000 have corresponding facetracks; the remainder provide monolingual supplementary data.
- Noise augmentation: The model is trained first on clean audio, then with additive white Gaussian noise at 10dB and 0dB SNR to improve noise tolerance.The stated SNRs correspond to signal-to-noise power ratios of 10:1 and 1:1, respectively.
- Implementation: Inputs use 120×120 lip images at 25Hz and MFCCs at 100Hz, with three-layer LSTMs and SGD training using batches of 64, dropout, and label smoothing.The Watch and Listen LSTMs have cell size 256, while the Spell LSTM has cell size 512; training lasts approximately 500,000 iterations, or 10 days.
4. Dataset
The LRS dataset is generated through a multi-stage pipeline that collects large-scale BBC audio-visual speech data with corresponding face tracks. The dataset also includes carefully prepared alignments, date-based partitions, and an auxiliary audio-only training set.
- Dataset construction: The pipeline collects thousands of hours of spoken BBC sentences and phrases with corresponding face tracks from programs recorded between 2010 and 2016.The selected programs are listed in Table 1 and shown in Figure 3.
- Dataset construction: Video preparation detects shot boundaries, identifies faces, tracks speakers across frames, and extracts facial landmarks before constructing mouth-motion data.The processing pipeline is summarized in Figure 4; the described methods include colour-histogram shot detection, HOG-based face detection, KLT tracking, and regression-tree landmark extraction.
- Dataset construction: Subtitle transcripts are force-aligned to audio and filtered against IBM Watson Speech to Text because broadcast subtitles are unsynchronized and not verbatim.The Penn Phonetics Lab Forced Aligner is used for alignment, while the IBM Watson service filters alignment errors.
- Dataset partitioning: Training, validation, and test sets are divided by broadcast date, and the dataset contains thousands of different speakers.The corresponding partition dates are provided in Table 2.
- Auxiliary audio-only data: An auxiliary audio-only dataset comprises BBC sentences without available face tracks and is used only for training, not testing.Its use is described in Section 3.3, with statistics reported in Table 4.
5. Experiments
Experiments evaluate WLAS and its training strategies on LRS and external lip-reading benchmarks using CER, WER, and unigram BLEU. The lips-only model outperforms professional lip reading, while fine-tuning exceeds the state of the art on LRW by a large margin.
- Evaluation protocol: The experiments report CER, WER, and unigram BLEU on an independent LRS test set, with decoding performed by beam search of width 4.Beam-width experiments found no observed benefit from increasing the width beyond 4.
- Results: All training methods from Section 3 improve performance, with their individual contributions broken down for the lips-only experiment in Table 5.Other experiments report results using the best training strategy.
- Decoding speed: The model decodes significantly faster than real time, taking approximately 0.5 seconds to process a 5-second sentence with beam width 4.The reported speed concerns reading and decoding a sentence.
- Human comparison: The lips-only WAS model correctly deciphers half of the spoken words, compared with less than one-quarter for a professional lip reader.The human comparison used a random sample of 200 test videos and a professional with around 10 years of experience.
- External benchmarks: On the LRW dataset, fine-tuning the network for one epoch on its 500-word lexicon exceeds the current state of the art by a large margin.LRW contains up to 1000 utterances of 500 isolated words from BBC television, spoken by over a thousand speakers.
6. Summary and extensions
The paper introduces WLAS, a dual-attention network that transcribes speech into characters from visual input, audio input, or both. It achieves strong lip-reading performance while motivating monotonic-attention and online-decoding extensions.
- Summary: WLAS uses dual attention to transcribe speech into characters from visual input only, audio input only, or both.The same architecture is reported to beat a professional lip reader on BBC television videos and surpass previous work on standard lip-reading benchmarks.
- Extensions: The unconstrained alignment attention consistently moves monotonically from left to right, suggesting soft or hard monotonicity constraints as an extension.The paper identifies explicit monotonicity as a possible future modeling constraint.
- Extensions: Because the sequence-to-sequence model decodes with the entire corresponding lip sequence in batch mode, a more online architecture is proposed for future work.The proposed online variant would change access to the lip sequence during decoding.
7. Appendix
The appendix specifies the LSTM, attention, encoder-decoder, ConvNet, and output-character components of the WLAS system. It also provides visual demonstrations of model predictions and video-to-text alignment generated by attention.
- Visualisation: A video demonstration shows predicted captions, with changing text colour indicating video-to-text alignment produced by the attention mechanism.The captions are predictions generated by the ‘Watch, Attend and Spell’ model.
- LSTM: The appendix documents the standard LSTM implementation, including input, forget, and output gates and cell activations matching the hidden-vector size.The sigmoid function is used for the gate equations.
- Architecture: Figure 8 details the encoder-decoder architecture, while Table 9 specifies the ConvNet architecture.Together, these visuals document the principal architectural components of the model.
- Attention: The attention mechanism follows Bahdanau et al. [3] and computes a video-stream attention vector, or alignment, using learnable weights and biases.Examples of attention outputs for both modalities appear in Figure 9.
- Output representation: The WLAS model generates output sequences from the character inventory listed in Table 10.The table provides the output characters used by the ‘Watch, Listen, Attend and Spell’ model.