Source-linked AI summary
Deep Audio-Visual Speech Recognition
Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, Andrew Zisserman
TL;DR
The paper addresses open-world sentence-level lip reading and asks how visual speech complements audio recognition, especially with noisy audio. It compares Transformer CTC and sequence-to-sequence models, introduces LRS2-BBC, and reports strong visual-only and noisy audio-visual recognition results. The work also shows practical differences between the two objectives in robustness, language-model dependence, and computational cost.
Problem
The paper seeks sentence-level lip reading in unconstrained videos and evidence about whether visual speech complements audio recognition when audio is noisy.
Method
The authors compare Transformer models trained with CTC or sequence-to-sequence losses and construct LRS2-BBC from thousands of British television videos with talking faces and subtitles.
Results
The best visual-only model surpasses prior LRS2-BBC lip-reading performance by a large margin, while combining audio and visual inputs significantly improves recognition under noisy audio.
Takeaways & Limitations
Visual information remains useful even with clean audio and becomes especially beneficial when the audio is noisy; CTC is more robust to loud babble noise than TM-seq2seq.
Takeaways & Limitations
TM-seq2seq is harder and slower to train than TM-CTC, taking approximately 8 days versus 5 days on the same hardware.
Abstract
from arXiv · showhide
The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem - unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) we compare two models for lip reading, one using a CTC loss, and the other using a sequence-to-sequence loss. Both models are built on top of the transformer self-attention architecture; (2) we investigate to what extent lip reading is complementary to audio speech recognition, especially when the audio signal is noisy; (3) we introduce and publicly release a new dataset for audio-visual speech recognition, LRS2-BBC, consisting of thousands of natural sentences from British television. The models that we train surpass the performance of all previous work on a lip reading benchmark dataset by a significant margin.
1 INTRODUCTION
The paper targets sentence-level lip reading in unconstrained videos and compares Transformer models using CTC and sequence-to-sequence losses. It also examines visual contributions to audio speech recognition under noise and introduces a large-scale training dataset.
- Motivation: Lip reading is ambiguous at the word level because different sounds can produce identical lip sequences.Context from neighboring words or a language model can resolve some ambiguities.
- Motivation: A machine that can lip read could support noisy-environment dictation, silent-film transcription, multi-talker speech separation, and improved speech recognition.
- Contributions: The paper compares Transformer-based lip-reading models trained with CTC and sequence-to-sequence losses using as much shared architecture as possible.The comparison is designed to assess the advantages and disadvantages of the two losses head-to-head.
- Contributions: The study investigates whether lip reading contributes to audio-based speech recognition, particularly when the audio signal is noisy.The authors train models on audio and visual inputs and systematically disturb the audio channel.
- Contributions: The models output characters: CTC outputs are independent, whereas sequence-to-sequence learning implicitly incorporates a language model through its architecture.Both models use beam search, optionally combined with an external language model.
- Contributions: The dataset contains talking faces with subtitles from thousands of hours of video and supports training and evaluating the models.The paper also reports over 22% absolute WER improvement over the earlier model through updated Transformer and ResNet architectures.
2 BACKGROUND
The background contrasts CTC and sequence-to-sequence approaches for sequence prediction and reviews their use in speech and lip-reading systems. Prior lip-reading datasets were limited in subjects and vocabulary, motivating larger-scale sentence recognition.
- CTC versus seq2seq: CTC predicts frame-wise labels and finds an alignment to the output sequence, but assumes independent labels and monotonic input-output ordering.A language model is therefore commonly used during post-processing.
- CTC versus seq2seq: Sequence-to-sequence models read the full input before predicting outputs conditioned on previous symbols, implicitly learning a language model.
- Related work: Earlier lip-reading work often recognized phonemes, visemes, words, or short phrases using CNNs, LSTMs, handcrafted features, or residual networks.A phoneme is a smallest distinguishable sound unit, while a viseme is its visual equivalent.
- Related work: Existing datasets had few subjects and limited vocabularies below 60 words, restricting progress toward full-sentence lip reading.
- Related work: The authors’ earlier WLAS system used dual attention over visual and audio streams and could transcribe characters from either or both modalities.
- Related work: Concurrent work trained a CTC lip-reading pipeline on a very large YouTube dataset and reported a 40.9% word error rate.
3 ARCHITECTURES
The proposed audio-visual systems share Transformer self-attention encoders but differ in their decoding objectives: one uses seq2seq attention, and the other uses CTC frame posteriors. They process video and audio as separate modalities and can operate with either modality alone.
- Model variants: The architecture explores Transformer encoder-decoder attention for seq2seq training and stacked self-attention blocks for CTC training.
- Common encoder: Video and audio enter separate modality encoders before being combined by the downstream model.The visual stream is processed by a spatio-temporal ResNet, while audio uses STFT-derived spectrograms.
- Common encoder: The common encoder uses stacked multi-head self-attention layers with fixed sinusoidal positional embeddings.
- TM-seq2seq: TM-seq2seq attends independently to video and audio on each decoder layer, concatenates their context vectors, and predicts character probabilities.
- TM-CTC: TM-CTC concatenates video and audio encodings and produces CTC posterior probabilities for every input frame.The network is trained with CTC loss.
- Inference: Both variants can run with only one modality by using the available modality’s encoding or attention vector.During inference, both use a character-level language model with left-to-right beam search.
4 DATASET
LRS2-BBC is generated through a multi-stage pipeline that extracts aligned talking-face sentences from varied British television broadcasts. The resulting dataset includes development, test, and pre-training partitions for audiovisual speech recognition.
- Dataset construction: The pipeline collects thousands of hours of spoken sentences and phrases with corresponding face tracks from diverse BBC programs.
- Video preparation: Face detection, shot-boundary analysis, and face tracking identify speaker appearances within broadcast videos.The SSD detector is used for face detection, while positions support face-track generation.
- Audio and text preparation: Subtitle text is force-aligned to audio and filtered against IBM Watson Speech to Text because broadcast subtitles are not verbatim.
- Synchronization: Audio-video synchronization and speaker detection identify the face whose lip movements match the audio and reject voice-over clips.Broadcast streams can be out of sync by up to around one second.
- Sentence extraction: Transcripts are split into sentences or phrases using punctuation and clipped to 100 characters or 10 seconds.No vocabulary-size restriction is imposed.
- Dataset splits: LRS2-BBC is divided by broadcast date into development and test sets, with an additional pre-train set containing excerpts and word-level alignment boundaries.The dataset also supports training external language models from a 26M-word subtitle corpus.
5 TRAINING STRATEGY
Training combines staged visual pretraining, frozen-feature sequence learning, and end-to-end fine-tuning. Curriculum learning lengthens sequences gradually, while noise augmentation prevents audio from dominating multimodal training.
- Training proceeds in four stages: visual-front-end training, feature generation, sequence-module training on frozen features, and end-to-end optimization.
- The visual front-end is pretrained on MV-LRS word excerpts with temporal convolution and augmentation through flips, frame removal, and spatial-temporal shifts.
- Curriculum learning starts with single-word examples and gradually increases sequence length, accelerating convergence and reducing overfitting.The longer examples are assembled from short subsequences of sentences.
- Sequence models are pretrained on frozen features from MV-LRS, LRS2-BBC, and LRS3-TED before separate end-to-end fine-tuning on the evaluation dataset.
- Babble noise at 0dB SNR is added during multimodal training because the easier audio task otherwise dominates the visual signal.Noise is added with probability p_n = 0.25 during training, followed by fine-tuning with noise always present for robustness assessment.
- The character-level output uses 40 tokens, with [sos] for seq2seq and [blank] for CTC, while decoding uses teacher forcing during training and prediction feedback during inference.Punctuation is not modeled because the dataset transcriptions contain none.
6 EXPERIMENTS
The experiments evaluate video-only lip reading, audio-visual recognition, noisy and unsynchronised inputs, and practical decoding and training trade-offs. TM-seq2seq leads on video-only lip reading, while combining audio and video is especially beneficial under noise; CTC offers efficiency and noise-robustness advantages.
- Lips only: 48.3% WER on LRS2-BBC makes TM-seq2seq the best-performing network, improving by over 22% absolute on the previous 70.4% state of the art.It also establishes a 58.9% WER baseline on LRS3-TED.
- Lips only: TM-seq2seq correctly predicts complex unseen sentences from a wide range of content using video only.Examples are provided in Table 3.
- Lips only: Video-only WER is relatively uniform across sentence lengths, although three-word samples appear more difficult, presumably because they provide less context.Sentence sizes with fewer than five samples are excluded from the analysis.
- Audio-visual speech recognition: 10.1% to 8.2% WER demonstrates an audio-visual TM-CTC improvement over audio-only recognition even with clean audio.The audio-visual TM-seq2seq gains over audio-only recognition are reported as similar.
- Audio-visual speech recognition: Mouth movements provide important speech-recognition cues under noisy audio, with audio-visual models outperforming either modality alone and reducing WER by up to 30%.The evaluation adds babble noise to the utterances and compares audio-only, video-only, and audio-visual systems.
- Discussion on seq2seq vs CTC: TM-seq2seq performs significantly better for lip reading without audio, whereas CTC models appear more robust to loud babble noise and decode faster.TM-seq2seq is harder and slower to train, while CTC decoding does not require autoregression and is faster for deployment.
7 CONCLUSION
The paper introduces the LRS2-BBC audiovisual dataset and shows that shared architectures support visual-only, audio-only, and audiovisual transcription. Visual information improves recognition with clean audio, while combining modalities provides a significant benefit under noise.
- Dataset: LRS2-BBC is a large-scale, unconstrained audiovisual dataset collected from thousands of British television videos.The dataset was formed by collecting and preprocessing thousands of videos from British television.
- Models: The same architectures transcribe speech from audio and video sequences and also operate when only one modality is present.
- Audiovisual recognition: Visual information improves speech recognition even when clean audio is available, with a significant improvement from combining modalities in noisy audio.
- Noise robustness: Under babble noise, audio-only attention appears more spread around the corrupted frames, while the last frames are not attended upon.The figure compares attention masks for clean and noisy utterances, with babble noise added to the 25 central audio frames.
APPENDIX A VISUAL FRONT-END ARCHITECTURE
The visual front end is specified as a spatio-temporal convolutional architecture whose detailed layer configuration is given in Table 5. Its residual blocks downsample aggressively while preserving shortcut connections.
- Architecture: Table 5 provides the architecture details for the spatio-temporal visual front end.
- Residual blocks: The residual 2D convolutional blocks apply their listed strides only to the first layer of each block.
- Downsampling: The visual front end has a total downsampling factor of 32, with a shortcut connection after every pair of 2D convolutions.
APPENDIX B TRANSFORMER ARCHITECTURE DETAILS
The models use multi-head attention blocks for both self-attention and encoder-decoder attention. Queries, keys, and values determine the context vectors, with their roles differing between self-attention and seq2seq decoding.
- Attention blocks: The same multi-head attention block is used in the models' self-attention and encoder-decoder attention layers.
- Inputs and outputs: Each block receives query, key, and value tensors and produces one context vector for every attention head.
- Dimensions: The tensors are organized using model dimension dmodel, with dk defining the size of each attention head.
- Tensor roles: For self-attention, Q = K = V; for TM-seq2seq encoder-decoder attention, K = V are encoder outputs and Q is the decoder input.The decoder input is the previous decoding step's network output in the first layer and the previous decoder layer's output thereafter.
APPENDIX C SEQ2SEQ DECODING WITH EXTERNAL LANGUAGE
TM-seq2seq decoding uses left-to-right beam search that combines visual-model probabilities with an external language model and a length normalization factor. The beam-search hyperparameters are selected on validation data.
- Beam search: TM-seq2seq uses left-to-right beam search with width W to score candidate hypotheses during decoding.
- Scoring: The decoding score combines log p(y|x) from the visual model with α log pLM(y), together with a length normalization factor LP(y).
- Hyperparameters: Without an external language model, validation selected W = 6, α = 0.0, and β = 0.6; with one, it selected W = 35, α = 0.1, and β = 0.7.
- CTC comparison: The CTC decoder also uses beam search, with W = 100, α = 0.5, and β = 0.1 producing the best validation results.
APPENDIX E PRECISION AND RECALL FROM EDIT DISTANCE
The appendix computes word-level precision, recall, and F1 from minimum edit-distance operations aggregated across the evaluation set. It also specifies CTC beam-search decoding with blank and non-blank path probabilities.
- Edit-distance evaluation: Word-wise precision, recall, and F1 are calculated from the minimum substitutions, insertions, and deletions needed to transform ground-truth transcriptions into predictions.These edit operations are computed for every evaluation sample and then aggregated before calculating average measures per word.
- CTC decoding: CTC beam-search decoding maintains separate probabilities for partial transcriptions ending in blank and non-blank tokens, combining them into each transcription’s total probability.The search uses CTC probabilities, a word dictionary, beam width, and hyperparameters while extending candidate transcriptions across time.
- CTC decoding: At each time step, candidate character extensions are added to the beam when new partial transcriptions are formed, and the highest-probability beam output is returned.The algorithm updates non-blank probabilities for candidate extensions and returns the maximum-probability transcription in the beam.
- Edit-distance evaluation: The aggregate counts distinguish substitutions of word j with word w from matches, deletions, and insertions involving word w.The notation defines ns(j,w), nm(w), ni(w), and nd(w) as totals over the evaluation set.