PaperlayerView exact revision on arXiv

Source-linked AI summary

Sub-word Level Lip Reading With Visual Attention

K R Prajwal, Triantafyllos Afouras, Andrew Zisserman

arXiv:2110.07603v2cs.CVcs.CL

TL;DR

Lip reading must recover speech from complex visual input with inherently ambiguous lip shapes, while many prior approaches adapt ASR techniques without accounting for vision-specific challenges. This paper combines attention-based visual pooling, sub-word tokens, and a visual speech detection model. It reports state-of-the-art lip-reading results on public benchmarks and strong visual speech detection performance, including results beyond several audio-visual methods.

  • Problem

    Lip reading must handle high-dimensional video with spatial and temporal complexity and ambiguous visual speech, while many prior methods adapt ASR without tailoring it to vision.

    Source: §1 · p. 1

  • Method

    The paper combines attention-based pooling of visual representations, sub-word tokenization, and a visual speech detection model trained on the lip-reading encoder.

    Source: §1 · pp. 1–2

  • Results

    The models achieve state-of-the-art results on public-data lip-reading benchmarks and outperform industrial models trained on an order of magnitude more data; the VSD model also surpasses several audio-visual baselines.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The results show that a lip-reading system can also support visual speech detection, with performance reported beyond several audio-visual baselines.

    Source: §6 · p. 8

    The VSD model struggles with segment lengths beyond the training limit of 8 seconds, so long-video inference uses short overlapping clips.

    Source: D. Limitations of our Visual Speech… · p. 12

Abstract

from arXiv · show

The goal of this paper is to learn strong lip reading models that can recognise speech in silent videos. Most prior works deal with the open-set visual speech recognition problem by adapting existing automatic speech recognition techniques on top of trivially pooled visual features. Instead, in this paper we focus on the unique challenges encountered in lip reading and propose tailored solutions. To this end, we make the following contributions: (1) we propose an attention-based pooling mechanism to aggregate visual speech representations; (2) we use sub-word units for lip reading for the first time and show that this allows us to better model the ambiguities of the task; (3) we propose a model for Visual Speech Detection (VSD), trained on top of the lip reading network. Following the above, we obtain state-of-the-art results on the challenging LRS2 and LRS3 benchmarks when training on public datasets, and even surpass models trained on large-scale industrial datasets by using an order of magnitude less data. Our best model achieves 22.6% word error rate on the LRS2 dataset, a performance unprecedented for lip reading models, significantly reducing the performance gap between lip reading and automatic speech recognition. Moreover, on the AVA-ActiveSpeaker benchmark, our VSD model surpasses all visual-only baselines and even outperforms several recent audio-visual methods.

1. Introduction

Lip reading combines high-dimensional visual processing with ambiguity in lip shapes, while much prior work adapts ASR methods without addressing vision-specific challenges. The paper proposes visual attention pooling, sub-word tokens, and visual speech detection, reporting strong benchmark performance.

1. Introduction

  • Lip reading must process high-dimensional video with spatial and temporal complexity, and visually indistinguishable sounds such as “pa,” “ba,” and “ma” make speech recognition ambiguous.The paper notes that humans perform relatively poorly at lip reading compared with understanding speech under noise or across accents [7].Source: §1 · p. 1
  • Most prior methods adapt ASR and machine-translation techniques without tailoring them to visual input, motivating models designed around visual-domain challenges.The paper conjectures that such design can improve accuracy and data efficiency by changing visual encoding and text tokenisation.Source: §1 · p. 1
  • The proposed visual backbone uses end-to-end attention pooling to track and aggregate lip-movement representations instead of relying on potentially suboptimal landmark pipelines.Landmarks may fail in profile views, and the useful facial region can vary in scale, aspect ratio, and location [72].Source: §1 · pp. 1–2
  • The paper uses semantically meaningful sub-word tokens and trains a visual speech detection model on top of its lip-reading encoder.Word-pieces shorten output sequences, reduce runtime and memory requirements, and provide language information; VSD identifies speech segments from video alone.Source: §1 · p. 2
  • The best models exceed prior work trained on public data and outperform industrial models trained on an order of magnitude more data; the VSD model also surpasses previous methods.The introduction describes the VSD result as far superior on a standard evaluation benchmark.Source: §1 · p. 2

2. Related Work

Prior lip-reading work progressed from handcrafted systems to deep-learning sequence models, while most pipelines rely on pooled CNN features and character outputs. The paper situates its alternatives among visual attention methods and video-only speech detection research.

Lip reading

  • Lip reading progressed from handcrafted visual and statistical systems to deep models using LSTM sequence-to-sequence [15] or CTC [7], with later Transformer and Conformer variants.Other work has used proprietary training data or distilled ASR models into lip-reading systems.Source: §2 · p. 2

Mouth ROI selection, registration and tracking; Text tokenization

  • Most pipelines use face crops or landmark-based mouth regions [7], then global average pooling; most output characters, whereas this paper proposes sub-word tokens.Prior approaches also include phoneme outputs and viseme sequences [21, 33]; sub-word tokens shorten output sequences and encode language information.Source: §2 · p. 3

Speech Detection

  • Prior visual speech detection methods include handcrafted statistical models, optical flow, and CNN-LSTM systems, often evaluated on constrained or non-public datasets.WildVVAD’s test set was unavailable at submission, preventing its use for benchmarking.Source: §2 · p. 3

3. Method

The method converts silent-video frames into compact visual features, predicts sub-word sequences with a Transformer, and trains the model in two stages: short-phrase end-to-end training followed by encoder-decoder training on longer subsequences.

3.1. Visual backbone

  • A spatio-temporal residual CNN extracts framewise feature maps, and Visual Transformer Pooling adds spatial encodings and self-attention before a learned query weights locations into compact frame representations.Stacking the frame representations yields a sequence that captures a compact spatio-temporal representation for each input frame.Source: §3.1 · p. 3, Figure 1 · p. 4
  • An encoder-decoder Transformer predicts output tokens autoregressively from the visual sequence, using WordPiece tokenization and beam-search decoding.The method also decodes horizontally flipped frames and can rescore hypotheses with an external language model.Source: §3.1 · p. 4

3.2. Training

  • Training first optimizes the whole network end-to-end on two-word phrases, then freezes the visual backbone and trains the encoder-decoder on consecutive-word subsequences of length two or more.The second stage matches the performance of the more complicated curriculum used in prior work while requiring less training time and manual effort.Source: §3.2 · p. 5

4. Experiments

The experiments evaluate the model on LRS2 and LRS3, add TEDxext training data, and compare implementation choices and ablations. The best model reaches 22.6% WER on LRS2 and 30.7% on LRS3, while the ablations examine component gains, training protocol, and visual attention.

4.1. Data

  • LRS2 provides approximately 224 hours and LRS3 475 hours of sentence-level video; both include word-aligned pretraining partitions and sentence-level test sets.The experiments also use random visual augmentations during training and horizontal flipping at inference.Source: §4.1 · p. 5, §4.2 · p. 5
  • TEDxext adds 1,204 hours across 318,459 visual speech tracks from 13,211 TEDx talks, with word boundaries force-aligned from automatically generated captions.The captions provide noisier supervision, while avoiding manual transcription makes the data collection scalable.Source: §4.1 · p. 5

4.2. Implementation details

  • For the best model, VTP uses six layers and eight heads per layer; pooling operates first at H/4 × W/4, with reduced feature dimensions and later downsampling to manage compute.The implementation uses a Linear Transformer for efficiency without a reported drop in recognition performance.Source: §4.2 · p. 5

4.3. State-of-the-art lipreading

  • 22.6% and 30.7% WER on LRS2 and LRS3, respectively; the model outperforms the strongest public-data baseline by 9% on LRS2 and 2.7% on LRS3.With MV-LRS and TEDxext, it also outperforms while using 10× less training data.Source: Table 1 · p. 6, §4.3 · p. 6

4.4. Ablations

  • WordPiece improves LRS2 by 3.8% absolute and VTP decreases WER by 6.3%; beam rescoring and test-time flipping add 1.1% and 0.9% improvements, respectively.The ablation compares successive design changes against an improved TM-seq2seq baseline.Source: §4.4 · p. 6, Table 2 · p. 7
  • Pooling at 24 × 24 feature resolution is more effective than pooling at 12 × 12 or 6 × 6, with approximately equal visual-front-end parameter counts.The comparison varies spatial resolution while keeping the number of Transformer layers per stage set to approximately match parameter counts.Source: §4.4 · p. 6, Table 3 · p. 7
  • Training on all n-gram subsequences at once yields WER 30.92, comparable to 30.91 with curriculum training, while requiring less manual tuning.The authors attribute the benefit to data augmentation from training on subsequences of varying lengths, rather than gradual length increases.Source: §4.4 · p. 7

4.5. Visual attention visualization

  • VTP attention tracks the mouth accurately as speakers turn their heads, including in extreme profile views.The visualization uses random LRS3 clips and shows the attention masks overlaid on input frames.Source: §4.5 · p. 7, Figure 2 · p. 7

5. Visual Speech Detection Application

The paper builds Visual Speech Detection (VSD) on a pretrained lip-reading encoder and evaluates it on AVA ActiveSpeaker. Its VTP model outperforms video-only and several audio-visual methods, and also beats a GAP ablation.

Model

  • The VSD head adds a fully connected layer and sigmoid activation to frame-level outputs of the pretrained lip-reading transformer encoder.It is trained on top of the encoder using binary cross-entropy for each training sample.Source: §5 · pp. 7–8

Dataset and evaluation

  • Training and evaluation use AVA ActiveSpeaker, with visual speaking labels formed by combining the speaking-and-audible and speaking-but-not-audible classes.The dataset has 120 training videos, 33 validation videos, and 109 test videos; evaluation uses mAP.Source: §5 · p. 8

Results

  • VTP outperforms the video-only baseline by over 17 mAP and several audio-visual methods [13] on AVA ActiveSpeaker.The authors also report an 8 mAP-point advantage over an otherwise identically evaluated GAP ablation.Source: Table 4 · p. 8, §5 · p. 8

6. Conclusion

The conclusion presents attention-based aggregation, sub-word tokenisation, and a VSD model built on the lip-reading system. The authors report state-of-the-art results for lip reading and VSD, including comparisons with industrial and audio-visual systems.

Conclusion

  • The best lip-reading models achieve state-of-the-art results, outperforming prior work trained on public data and industrial models trained on orders of magnitude more data.The conclusion describes the margin over prior public-data work as significant.Source: §6 · p. 8
  • The improved lip-reading architecture uses attention-based aggregation of visual representations and includes sub-word tokenisation among its training-protocol enhancements.Source: §6 · p. 8
  • The VSD model built on the lip-reading system achieves state-of-the-art results and outperforms several audio-visual baselines.Source: §6 · p. 8

A. Architecture Details

The visual CNN backbone uses batch normalization and ReLU after each convolution. The architecture details distinguish layers used only by the TM-seq2seq baseline from those in the best model.

A. Architecture Details

  • Batch normalization and ReLU activation follow every convolutional layer in the visual CNN backbone.Source: Table 5 · p. 12
  • Shortcut connections are added at each layer except the first layer of every residual block, which has stride greater than 1.Source: Table 5 · p. 12
  • Gray layers in the architecture table are used only by the TM-seq2seq baseline and are absent from the best model.The accompanying description says these baseline-only lines are removed when adding VTP layers after a desired feature-map resolution.Source: Table 5 · p. 12, A. Architecture Details · p. 12

B. Applications and Ethical Considerations

The paper describes possible applications of lip reading alongside privacy risks and conditions that bound the surveillance concern. It also argues that public code and models can support broader research and transparency.

Applications and Ethical Considerations

  • Potential applications include improving recognition when audio is corrupted, silent dictation, transcribing silent films, and communication support for speech-impaired people and people with aphonia.The paper also lists medical applications, including helping people with Lou Gehrig’s disease speak.Source: B. Applications and Ethical… · p. 12
  • The authors identify privacy risks, including possible surveillance using public CCTV footage to eavesdrop on private conversations.Source: B. Applications and Ethical… · p. 12
  • The surveillance-risk discussion is bounded by benchmarks using professional, high-resolution, high-frame-rate, well-lit footage with speakers generally aware of filming and facing the camera.The paper reports that lip-reading performance greatly deteriorates at lower resolutions or frame rates and at non-frontal viewpoints.Source: B. Applications and Ethical… · p. 12
  • The authors say releasing code and pretrained models can accelerate research and enable research on defenses against adversarial attacks.They also describe public availability as supporting transparency and democratizing research.Source: B. Applications and Ethical… · p. 12
  • The authors conclude that positive applications, including medical ones, outweigh the risks of malevolent uses and support continued transparent research.Source: B. Applications and Ethical… · p. 12

C. Lip reading error analysis

The error analysis describes the model’s mistakes as meaningful and identifies additional context as a possible direction for improvement.

C. Lip reading error analysis

D. Limitations of our Visual Speech Detection model

The VSD model can mistake emotional expressions for speech and struggles with clips longer than its training segments; overlapping short clips provide a workaround for long-video inference.

D. Limitations of our Visual Speech Detection model

Loading 2110.07603v2…