Deep Lip Reading: a comparison of models and an online application

1. Introduction

• The paper frames recent deep-learning advances and large-scale datasets as drivers of improved visual speech recognition.Source: §1 · p. 1
• The paper proposes recurrent, fully convolutional, and Transformer lip-reading models, compares their accuracy and training time, and reports a new LRS2 benchmark state of the art.It also investigates modifications for online transcription, with outputs available immediately rather than only utterance-in, utterance-out.Source: §1 · p. 1
• Online transcription is proposed as a route to applications such as real-time speech captioning in noisy environments.The paper considers modifications that make transcriptions available immediately rather than utterance-in, utterance-out.Source: §1 · p. 1

1.1. Related works

• Prior character-level visual recognition work includes CTC models such as LipNet [14] and sequence-to-sequence attention models such as, later extended with wider pose variety and multi-view models.Source: §1.1 · p. 1
• Related work extends convolutional architectures across translation and speech tasks, including fully convolutional ASR trained with CTC [25] or simplified variants.Earlier online methods include monotonic or hard attention [31], online CTC approaches [32] [33], and dynamic programming over partially observed input.Source: §1.1 · p. 1

2. Architectures

• A shared visual frontend processes loosely cropped lip-region image sequences into per-frame features before one of three sequence modules produces characters.Source: §2 · p. 2

2.1. Vision Module (VM)

• The visual frontend applies a five-frame 3D convolution followed by a 2D ResNet, preserving temporal resolution and yielding a 512-dimensional feature vector per input frame.Source: §2.1 · p. 2

2.2. Bidirectional LSTM (BL)

• BL uses three stacked bidirectional LSTM layers to emit framewise character probabilities and is trained with CTC, with beam-search decoding that incorporates an external language model.The output alphabet includes the CTC blank character.Source: §2.2 · p. 2

2.3. Fully Convolutional (FC)

• The fully convolutional model uses depth-wise separable temporal convolutions, shortcut connections, batch normalization, and ReLU; its FC-10 and FC-15 variants use CTC and external-language-model beam search.Source: §2.3 · p. 2

2.4. Transformer model (TM)

• The Transformer uses an encoder-decoder with multi-head attention and sinusoidal positional embeddings, trains character predictions with cross-entropy, and learns an implicit language model.The described base model has six encoder and six decoder layers, model size 512, eight attention heads, and dropout p = 0.1.Source: §2.4 · p. 2

2.5. External Language Model (LM)

• Inference combines character-level language-model probabilities with model outputs through shallow fusion; Table 1 reports CER, WER, parameter count, batch processing time, and curriculum training time.The table excludes the two-week visual-frontend training time.Source: §2.5 · p. 2, Table 1 · p. 3

3.1. Datasets and evaluation measures

• LRW provides about 489K fixed-length, single-word clips, while challenging LRS2 contains sentence videos, over 2M training word instances, and a vocabulary exceeding 40K.Source: §3.1 · p. 3

3.2. Training protocol

• Training proceeds from visual-front-end pretraining through feature extraction to sequence-model training, using curriculum learning and LRS2 fine-tuning.Source: §3.2 · p. 3

3.3. Results and Model Comparison

• 50% WER with the Transformer and T2 improved on the previous 70.4% state-of-the-art on LRS2.Source: §3.3 · p. 3
• 55% WER for the fully convolutional model; adding five layers reduced WER by 2%, and depth-separable convolutions doubled training speed without reducing accuracy.Source: §3.3 · p. 3
• T2 decoding improved WER by 0.7–1.3% over T1 across all models; FC-15 completed training in 3.5 days versus 13 days for the Transformer.Source: §3.3 · p. 4
• FC and BL generalized to longer sequences, while Transformer generalization was weaker beyond training lengths; FC accuracy gains diminished beyond 80 frames.Source: §3.3 · p. 4

• With K = 5, the 11- and 16-layer models peek 22 and 32 future frames, respectively, including the front-end 3D convolutions.Source: §4 · p. 4
• Applying convolutions with a lag of r frames and incrementally running beam search yields the same final prediction as offline decoding.Source: §4 · p. 4
• Variable-length training permits partial-sentence predictions at every time step, with overall decoding complexity O(TrW|A|).Source: §4 · p. 4
• The language model can also complete unfinished words in the current beam to predict further ahead.Source: §4 · p. 4

• The authors propose and compare three neural network architectures for lip reading, exceeding the previous state of the art by a large margin.Source: §5 · p. 4
• The paper's online lip-reading investigation is preliminary and includes a proposed decoding algorithm.Source: §5 · p. 4
• Future work could test alternative activations and phoneme or byte-pair outputs instead of characters.Source: §5 · p. 4

• MV-LRS(w) consists of isolated clips for the 480 most frequent words, each with at least 1,000 samples.Source: Table 3 · p. 6
• The MV-LRS and LRS2 statistics include their noisy pre-train sets alongside the main datasets.Source: Table 3 · p. 6
• T1 contains LRS2 sample transcriptions, whereas T2 contains full subtitles from the shows used to create LRS2.Source: Table 3 · p. 6

A.2. Visual front-end architecture

• The visual front-end has a total down-sampling factor of 32.The residual 2D convolutional block strides apply only to each block’s first layer.Source: Table 4 · p. 6
• Residual shortcut connections are added after every pair of 2D convolutions.Source: Table 4 · p. 6
• The 2D convolutions are applied separately to each time-frame.Source: Table 4 · p. 6

A.3. Seq2Seq decoding with external language model

• Left-to-right beam search scores Transformer hypotheses using visual and language-model probabilities and a length penalty.The search uses beam width W.Source: §A.3 · p. 6
• For decoding without the external language model (T1), the study determined the best hyperparameter values by conducting a grid search on the validation set.Source: §A.3 · p. 6
• W=15, α=0.1, and β=0.7 were selected for T2 decoding with the language model.The values were determined by grid search on the validation set.Source: §A.3 · p. 6

A.4. CTC decoding algorithm with external language model

• CTC beam search combines the language model with a length penalty to score candidate transcriptions.The best reported settings are W=100, α=0.5, and β=0.1.Source: §A.4 · p. 6
• The search bookkeeping handles repeated and blank character emissions.The length normalization is LP(y)=|y|^β.Source: §A.4 · p. 6
• The decoder tracks partial transcription probabilities for paths ending in blank and non-blank tokens.The total probability for a partial transcription is the sum of those two probabilities.Source: §A.4 · p. 6

A.5. Online CTC decoding algorithm

• O(TrW|A|) time is required for online CTC decoding, with decoding steps lagging real time by r frames.Here T is input length, r is half the network’s total receptive field, W is beam width, and |A| is alphabet size.Source: §A.5 · p. 7

A.6. Confusion Matrix

• Greedy CTC predictions frequently confuse B with M or P and V with F, while Q is often emitted instead of W.The figure reports normalized substitution counts averaged over the dataset; the authors hypothesize visual similarity explains the Q/W confusion.Source: Figure 2 · p. 7, §A.6 · p. 7

A.7. Decoding examples

• Online decoding examples show predictions becoming correct after initial frames with insufficient context for confident output.The table gives sequences started at the utterance beginning and middle; red marks language-model word completions.Source: §A.7 · p. 7, Table 5 · p. 8