Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecture

Prior work

• Traditional audiovisual systems extract audio and visual features separately before combining them for joint classification.Deep learning has enabled direct visual feature extraction from mouth-region pixels, complementing established audio features.Source: §1 · p. 1
• Recent visual speech models use either fully connected layers with recurrent networks or 3D convolutions with convolutional and recurrent networks.These approaches model temporal dynamics from mouth-region inputs using LSTMs, GRUs, or related architectures.Source: §1 · p. 1

Proposed approach

• The proposed model uses ResNets for mouth-ROI features and a hybrid CTC/attention architecture for audio-visual continuous speech recognition in-the-wild.Attention provides encoder-decoder alignment, while CTC imposes strictly monotonic alignment.Source: §1 · pp. 1–2
• The architecture uses a BLSTM encoder and joint CTC/attention decoding with an external language model.The considered architectures combine stacked BLSTMs with joint decoding.Source: §1 · p. 1, Figure 1 · p. 2

Contribution

• This is presented as the first use of a hybrid CTC/attention architecture for audio-visual speech recognition in-the-wild, evaluated on LRS2.LRS2 is described as the largest publicly available database of continuous audio-visual speech in-the-wild.Source: §1 · p. 2

Results

• The audio-visual model improves over audio-only recognition by 1.3% absolute WER in clean audio and reaches 7% WER on LRS2.The proposed system also reports a 6.9% absolute WER decrease for visual-only recognition over the LRS2 state of the art.Source: §1 · p. 2

Dataset scope

• LRS2 is described as the largest publicly available dataset for lip-reading sentences in-the-wild.The database contains short segments from BBC news and talk programmes.Source: §2 · p. 2

Dataset characteristics

• LRS2 is challenging because it includes thousands of speakers and substantial head-pose and illumination variation.Head pose ranges from frontal to profile, with an example shown in Fig. 2.Source: §2 · p. 2, Figure 2 · p. 3

Dataset scale

• The dataset contains more than 2 million words and more than 140K utterances.It includes training, validation, test, and pre-training sets.Source: §2 · p. 2

• The system maps audio or video input sequences to label sequences using BLSTM encoders and a joint decoder with RNN-LM, attention, and CTC mechanisms.Source: §3.2 · p. 3
• The hybrid architecture combines attention’s non-independent modeling with CTC’s monotonic alignment and blank-token transcription.Source: §3.2 · p. 3
• Training uses a shared encoder with separate CTC and attention decoder mechanisms, optimizing a weighted combination of their objectives.Source: §3.2 · p. 3
• Decoding combines attention and CTC probabilities to address non-monotonic alignment and end-of-sentence detection issues.Source: §3.2 · p. 3
• A character-level RNN-LM is incorporated through shallow fusion, with β controlling its relative weight.Source: §3.2 · p. 3

Fusion Types

• Early fusion concatenates audio and visual features inside the encoder, whereas late fusion combines probabilities from separate audio and video encoder-decoder systems.Source: §3.3 · pp. 3–4

Pre-processing

• Mouth ROIs use a fixed 130 by 80 bounding box, resize to 122 by 122, and undergo grayscale conversion and mean-variance normalization.Source: §4.1 · p. 4
• Video training uses random cropping, while testing uses the central patch; audio features are normalized per utterance.Source: §4.1 · p. 4

Dataset

• LRS2 is divided into training, validation, and test sets, with a pre-training set containing part-sentences and multiple sentences.Source: §4.2 · p. 4
• The training set contains only single full sentences, unlike the pre-training set.Source: §4.2 · p. 4

Training procedure

• Training first pre-trains the visual extractor on LRW, fine-tunes it on LRS2, trains the hybrid model, and then trains an external language model on two text corpora.Source: §5 · p. 4

Visual training

• The visual model uses ResNet pre-training, a 2-layer BLSTM, Adam optimization, mini-batches of 36 sequences, learning rate 0.0003, and early stopping.Source: §5.1 · p. 4

Augmentation

• Video augmentation applies random cropping and horizontal flips with probability 50% to every frame in a clip.Source: §5.1 · p. 4

Hybrid training

• The hybrid CTC/attention model trains for 20 epochs with Adadelta and audio augmentation using babble noise at 0 dB, 5 dB, and 10 dB SNR.Source: §5.2 · p. 4

Language model

• The language model combines LibriSpeech transcriptions containing 9.4 million words with more than 2 million words from the LRS2 pre-training set.Source: §5.3 · p. 4

Evaluation

• The study evaluates audio-only, visual-only, and audiovisual models under multiple fusion settings and reports WER and CER on LRS2.Source: Figure 3 · p. 5, Table 2 · p. 6

• 6.9% absolute WER improvement over establishes the visual-only system as state of the art when trained only on LRS2.The comparison excludes systems trained with external databases.Source: §6 · p. 6
• 8.3% WER and 4.4% CER are achieved by the audio-only model.Source: §6 · p. 6
• 1.3% WER and 0.8% CER improvements over audio-only result from early audiovisual fusion.Source: §6 · p. 6
• 8.5% WER from late fusion is worse than early fusion, possibly because it cannot directly model audio–visual feature correlation.Source: §6 · p. 6
• 84.5%–93.7% audio-only WER occurs at -5 dB across noise types, compared with 8.3%–10.3% at 20 dB.Car noise is an exception, producing 30.9% WER at -5 dB.Source: §6 · p. 6
• 32.9% absolute WER improvement is reached over audio-only at -5 dB for construction drilling noise.Across noise types, audiovisual gains generally increase as SNR decreases; tested noise ranges from -5 dB to 20 dB.Source: §6 · p. 6

• The work presents a joint CTC/attention hybrid architecture for audiovisual speech recognition.Source: §7 · p. 6
• On LRS2, the audiovisual model significantly outperforms audio-only recognition, especially at high noise levels.Source: §7 · p. 6
• The audiovisual model achieves new state-of-the-art performance on LRS2 across the different noise types considered.Source: §7 · p. 6