Source-linked AI summary
Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecture
Stavros Petridis, Themos Stafylakis, Pingchuan Ma, Georgios Tzimiropoulos, Maja Pantic
TL;DR
The paper addresses the tension between CTC's conditional independence and attention models' nonsequential alignments in speech recognition. It applies a hybrid CTC/attention architecture to audio-visual speech recognition on LRS2, achieving 7% WER and stronger noise robustness than audio-only recognition.
Problem
CTC assumes conditional independence between characters, whereas attention-based models can produce nonsequential alignments.
Method
The paper uses a hybrid CTC/attention architecture with ResNet-based visual features for audio-visual continuous speech recognition in-the-wild.
Results
The audio-visual model achieves 7% WER on LRS2, improves over audio-only recognition by 1.3% absolute WER in clean audio, and improves WER by up to 32.9% under noise.
Takeaways & Limitations
On LRS2, audio-visual recognition significantly outperforms audio-only recognition, especially at high noise levels, across the noise types considered.
Abstract
from arXiv · showhide
Recent works in speech recognition rely either on connectionist temporal classification (CTC) or sequence-to-sequence models for character-level recognition. CTC assumes conditional independence of individual characters, whereas attention-based models can provide nonsequential alignments. Therefore, we could use a CTC loss in combination with an attention-based model in order to force monotonic alignments and at the same time get rid of the conditional independence assumption. In this paper, we use the recently proposed hybrid CTC/attention architecture for audio-visual recognition of speech in-the-wild. To the best of our knowledge, this is the first time that such a hybrid architecture architecture is used for audio-visual recognition of speech. We use the LRS2 database and show that the proposed audio-visual model leads to an 1.3% absolute decrease in word error rate over the audio-only model and achieves the new state-of-the-art performance on LRS2 database (7% word error rate). We also observe that the audio-visual model significantly outperforms the audio-based model (up to 32.9% absolute improvement in word error rate) for several different types of noise as the signal-to-noise ratio decreases.
1. INTRODUCTION
This work combines visual feature extraction with a hybrid CTC/attention architecture for audio-visual speech recognition in-the-wild. On LRS2, the proposed system improves visual-only and audio-visual recognition while remaining more robust to noise than audio-only recognition.
- Prior work: Traditional audiovisual systems extract audio and visual features separately before combining them for joint classification.Deep learning has enabled direct visual feature extraction from mouth-region pixels, complementing established audio features.
- Prior work: Recent visual speech models use either fully connected layers with recurrent networks or 3D convolutions with convolutional and recurrent networks.These approaches model temporal dynamics from mouth-region inputs using LSTMs, GRUs, or related architectures.
- Proposed approach: The proposed model uses ResNets for mouth-ROI features and a hybrid CTC/attention architecture for audio-visual continuous speech recognition in-the-wild.Attention provides encoder-decoder alignment, while CTC imposes strictly monotonic alignment.
- Proposed approach: The architecture uses a BLSTM encoder and joint CTC/attention decoding with an external language model.The considered architectures combine stacked BLSTMs with joint decoding.
- Contribution: This is presented as the first use of a hybrid CTC/attention architecture for audio-visual speech recognition in-the-wild, evaluated on LRS2.LRS2 is described as the largest publicly available database of continuous audio-visual speech in-the-wild.
- Results: The audio-visual model improves over audio-only recognition by 1.3% absolute WER in clean audio and reaches 7% WER on LRS2.The proposed system also reports a 6.9% absolute WER decrease for visual-only recognition over the LRS2 state of the art.
2. LRS2 DATABASE
The study uses LRS2, a large in-the-wild lip-reading dataset built from BBC programme segments. Its challenging variation spans speakers, head pose, illumination, and utterance duration.
- Dataset scope: LRS2 is described as the largest publicly available dataset for lip-reading sentences in-the-wild.The database contains short segments from BBC news and talk programmes.
- Dataset characteristics: LRS2 is challenging because it includes thousands of speakers and substantial head-pose and illumination variation.Head pose ranges from frontal to profile, with an example shown in Fig. 2.
- Dataset scale: The dataset contains more than 2 million words and more than 140K utterances.It includes training, validation, test, and pre-training sets.
3. ARCHITECTURE
The architecture combines audio and visual features with a hybrid CTC/attention encoder-decoder. It uses joint CTC-attention training and decoding, with early and late fusion alternatives.
- The system maps audio or video input sequences to label sequences using BLSTM encoders and a joint decoder with RNN-LM, attention, and CTC mechanisms.
- The hybrid architecture combines attention’s non-independent modeling with CTC’s monotonic alignment and blank-token transcription.
- Training uses a shared encoder with separate CTC and attention decoder mechanisms, optimizing a weighted combination of their objectives.
- Decoding combines attention and CTC probabilities to address non-monotonic alignment and end-of-sentence detection issues.
- A character-level RNN-LM is incorporated through shallow fusion, with β controlling its relative weight.
- Fusion Types: Early fusion concatenates audio and visual features inside the encoder, whereas late fusion combines probabilities from separate audio and video encoder-decoder systems.
4. EXPERIMENTAL SETUP
Pre-processing extracts normalized mouth-region video inputs and normalized audio features from LRS2. The dataset provides separate training, validation, test, and pre-training partitions with different utterance structures.
- Pre-processing: Mouth ROIs use a fixed 130 by 80 bounding box, resize to 122 by 122, and undergo grayscale conversion and mean-variance normalization.
- Pre-processing: Video training uses random cropping, while testing uses the central patch; audio features are normalized per utterance.
- Dataset: LRS2 is divided into training, validation, and test sets, with a pre-training set containing part-sentences and multiple sentences.
- Dataset: The training set contains only single full sentences, unlike the pre-training set.
5. TRAINING
Training proceeds in three phases: visual pre-training and fine-tuning, hybrid CTC/attention training, and external language-model training. The setup includes augmentation, tuned decoding parameters, and reported evaluation configurations.
- Training procedure: Training first pre-trains the visual extractor on LRW, fine-tunes it on LRS2, trains the hybrid model, and then trains an external language model on two text corpora.
- Visual training: The visual model uses ResNet pre-training, a 2-layer BLSTM, Adam optimization, mini-batches of 36 sequences, learning rate 0.0003, and early stopping.
- Augmentation: Video augmentation applies random cropping and horizontal flips with probability 50% to every frame in a clip.
- Hybrid training: The hybrid CTC/attention model trains for 20 epochs with Adadelta and audio augmentation using babble noise at 0 dB, 5 dB, and 10 dB SNR.
- Language model: The language model combines LibriSpeech transcriptions containing 9.4 million words with more than 2 million words from the LRS2 pre-training set.
- Evaluation: The study evaluates audio-only, visual-only, and audiovisual models under multiple fusion settings and reports WER and CER on LRS2.
6. RESULTS
The proposed audiovisual system improves recognition over audio-only and prior visual-only systems, with stronger gains under severe noise. Early fusion performs best among the tested fusion strategies.
- 6.9% absolute WER improvement over establishes the visual-only system as state of the art when trained only on LRS2.The comparison excludes systems trained with external databases.
- 8.3% WER and 4.4% CER are achieved by the audio-only model.
- 1.3% WER and 0.8% CER improvements over audio-only result from early audiovisual fusion.
- 8.5% WER from late fusion is worse than early fusion, possibly because it cannot directly model audio–visual feature correlation.
- 84.5%–93.7% audio-only WER occurs at -5 dB across noise types, compared with 8.3%–10.3% at 20 dB.Car noise is an exception, producing 30.9% WER at -5 dB.
- 32.9% absolute WER improvement is reached over audio-only at -5 dB for construction drilling noise.Across noise types, audiovisual gains generally increase as SNR decreases; tested noise ranges from -5 dB to 20 dB.
7. CONCLUSIONS
The paper presents a joint CTC/attention architecture for audiovisual speech recognition and evaluates it on LRS2. The audiovisual model achieves state-of-the-art performance and remains especially advantageous under high noise across noise types.
- The work presents a joint CTC/attention hybrid architecture for audiovisual speech recognition.
- On LRS2, the audiovisual model significantly outperforms audio-only recognition, especially at high noise levels.
- The audiovisual model achieves new state-of-the-art performance on LRS2 across the different noise types considered.