Source-linked AI summary
End-to-end Audiovisual Speech Recognition
Stavros Petridis, Themos Stafylakis, Pingchuan Ma, Feipeng Cai, Georgios Tzimiropoulos, Maja Pantic
TL;DR
End-to-end audiovisual speech recognition had limited prior work using raw multimodal inputs. The paper proposes separate ResNet-BGRU streams with recurrent fusion for LRW word recognition. It reports a 0.3% absolute accuracy gain over audio-only and MFCC-based systems in clean conditions, and up to 14.1% improvement over audio-only models under high noise.
Problem
End-to-end audiovisual speech recognition research was very limited, with prior systems relying on MFCCs or smaller datasets.
Method
The model extracts features from raw mouth images and waveforms using separate ResNet-BGRU streams, then fuses them with another 2-layer BGRU.
Results
The audiovisual model improves accuracy by 0.3% over the end-to-end audio-only model and MFCC-based system, and by up to 14.1% over audio-only models under high noise.
Takeaways & Limitations
Audiovisual fusion provides small benefits in clean or low-noise audio and substantially improves robustness under high noise.
Abstract
from arXiv · showhide
Several end-to-end deep learning approaches have been recently presented which extract either audio or visual features from the input images or audio signals and perform speech recognition. However, research on end-to-end audiovisual models is very limited. In this work, we present an end-to-end audiovisual model based on residual networks and Bidirectional Gated Recurrent Units (BGRUs). To the best of our knowledge, this is the first audiovisual fusion model which simultaneously learns to extract features directly from the image pixels and audio waveforms and performs within-context word recognition on a large publicly available dataset (LRW). The model consists of two streams, one for each modality, which extract features directly from mouth regions and raw waveforms. The temporal dynamics in each stream/modality are modeled by a 2-layer BGRU and the fusion of multiple streams/modalities takes place via another 2-layer BGRU. A slight improvement in the classification rate over an end-to-end audio-only and MFCC-based model is reported in clean audio conditions and low levels of noise. In presence of high levels of noise, the end-to-end audiovisual model significantly outperforms both audio-only models.
1. INTRODUCTION
End-to-end audiovisual speech recognition remained relatively underexplored, with prior systems often relying on engineered features or smaller datasets. This work introduces a raw-input audiovisual model and reports strong performance, especially under high noise.
- Prior work: Prior audiovisual systems typically extracted features separately before combining them for classification.Deep approaches often replaced hand-crafted extraction with bottleneck features, then used classifiers such as SVMs or HMMs.
- Research gap: End-to-end audiovisual speech recognition had very limited prior research, with existing systems using MFCCs or smaller OuluVS data.One prior system used mouth ROIs and MFCCs, while another used raw images and spectrograms on OuluVS.
- Contribution: The proposed model uses separate ResNet-BGRU streams for raw mouth images and waveforms, followed by BGRU-based audiovisual fusion.Each modality is modeled temporally before another two-layer BGRU fuses the streams.
- Results: 0.3% absolute accuracy improvement over the end-to-end audio-only model and MFCC-based system is reported on 500-word LRW classification.The model achieves state-of-the-art audiovisual fusion performance on LRW.
- Results: Up to 14.1% absolute improvement over audio-only models is reported under high noise.The audiovisual model significantly outperforms both audio-only baselines in these conditions.
2. LRW DATABASE
The LRW database is a large, challenging in-the-wild lipreading resource containing short BBC-program segments and substantial speaker, pose, illumination, and lexical variation.
- Dataset scope: LRW is the largest publicly available in-the-wild lipreading dataset used here for within-context word recognition.Its clips are 1.16 seconds long and come mainly from BBC news and talk shows.
- Dataset scope: The dataset contains 500 words and more than 1000 speakers, exceeding the vocabulary size of typical word-recognition datasets.Existing lipreading databases commonly contain only 10 to 50 words.
- Challenges: LRW includes large variation in head pose and illumination, making recognition challenging.Words also occur mid-utterance, with co-articulation from preceding and subsequent words.
- Challenges: Visually similar word forms, including singular-plural pairs and related forms such as America and American, create additional ambiguity.
3. END-TO-END AUDIOVISUAL SPEECH RECOGNITION
The system learns audiovisual word-recognition representations directly from raw mouth images and audio waveforms using modality-specific ResNet-BGRU streams and recurrent fusion.
- Feature extraction: Two streams extract features directly from raw images and audio waveforms, with ResNets serving as modality-specific feature extractors.The visual stream uses a 34-layer ResNet, while the audio stream uses an 18-layer ResNet with one-dimensional kernels.
- Temporal modeling: Each modality uses a 2-layer BGRU to model the temporal dynamics of its extracted features.The visual and audio streams both feed their ResNet outputs into two-layer BGRU back-ends.
- Visual stream: The visual stream combines spatiotemporal convolution with ResNet-34 and is trained without pretrained models.Its initial convolution captures short-term mouth-region dynamics before residual feature extraction.
- Audio stream: The audio stream uses a 5 ms temporal kernel, 0.25 ms stride, and average pooling to produce 29 frames aligned with video.Deeper one-dimensional residual layers extract longer-term speech characteristics.
- Fusion and classification: Concatenated stream outputs feed another 2-layer BGRU that fuses modalities and jointly models their temporal dynamics.A softmax labels each frame, and the sequence label is selected using the highest average probability.
4. EXPERIMENTAL SETUP
Training uses standardized mouth-video and audio preprocessing, data augmentation, staged stream initialization, and subsequent end-to-end audiovisual optimization.
- Preprocessing: Mouth videos use fixed 96 × 96 crops, grayscale conversion, and normalization by overall mean and variance.A fixed bounding box is possible because the mouth ROIs are already centered.
- Preprocessing: Audio segments are z-normalized to zero mean and unit standard deviation to account for speaker loudness variation.
- Augmentation: Training data include random mouth-ROI cropping and horizontal flips, while babble noise is added at levels from -5 dB to 20 dB.The same video transformation is applied to every frame in a clip, with horizontal flips sampled at 50% probability.
- Training procedure: The two modality streams are first trained independently before being used to initialize the multi-stream network.Direct end-to-end stream training was reported to give suboptimal performance, motivating staged initialization.
- Training procedure: The fusion BGRU is initially trained with the stream weights fixed, after which the entire audiovisual network is trained jointly.This staged procedure precedes end-to-end optimization with Adam.
5. RESULTS
On LRW, the audiovisual model slightly improves clean-condition classification over audio-only and MFCC systems, while showing stronger robustness as audio noise increases.
- Table 1 compares Classification Rate (CR) for audio-only, video-only, and audiovisual models on the LRW database.
- The end-to-end audio system performs similarly to the MFCC-based system despite using only the raw waveform.Training the end-to-end audio system requires significantly more effort than training the 2-layer BGRU with MFCCs.
- 0.3% absolute improvement: the audiovisual system exceeds the audio-only models under clean audio conditions and low noise.
- The end-to-end audio model outperforms the MFCC classifier by 0.9% at 5 dB, 3.5% at 0 dB, and 7.5% at -5 dB.
- The audiovisual model significantly outperforms the end-to-end audio-only model by 1.3% at 5 dB, 3.9% at 0 dB, and 14.1% at -5 dB.The noise experiments vary babble-noise conditions from -5 dB to 20 dB SNR.
6. CONCLUSION
The paper presents an end-to-end audiovisual fusion system that learns from pixels and audio waveforms for within-context word recognition. It slightly outperforms audio baselines in clean or low-noise conditions and substantially outperforms them under high noise.
- The system jointly learns features from image pixels and audio waveforms and performs classification using BGRUs.
- On the LRW database, audiovisual fusion slightly outperforms the standard MFCC-based system under clean conditions and low noise.
- Under high noise, the audiovisual model significantly outperforms both the end-to-end and MFCC-based audio models.Future work includes sentence recognition and adaptive modality weighting based on noise levels.