Source-linked AI summary
End-to-end Audio-visual Speech Recognition with Conformers
Pingchuan Ma, Stavros Petridis, Maja Pantic
TL;DR
The paper addresses limitations of audiovisual speech recognition systems that rely on staged processing or pre-computed features. It proposes an end-to-end hybrid CTC/attention architecture using raw pixels and waveforms, conformers, and a transformer language model, and reports state-of-the-art gains across audio-only, visual-only, and audio-visual settings on LRS2 and LRS3.
Problem
Prior audiovisual systems often use pre-computed features, while few truly end-to-end systems operate on raw pixels and audio waveforms.
Method
The model jointly trains ResNet-18 audio and visual front-ends, conformer back-ends, MLP fusion, hybrid CTC/attention decoding, and a transformer-based language model.
Results
The proposed models raise state-of-the-art performance by a large margin in audio-only, visual-only, and audio-visual experiments on LRS2 and LRS3.
Takeaways & Limitations
Audio-visual recognition particularly outperforms audio-only recognition at high noise levels.
Takeaways & Limitations
LRS3 contains overlapped identities in both its pre-training and test sets.
Abstract
from arXiv · showhide
In this work, we present a hybrid CTC/Attention model based on a ResNet-18 and Convolution-augmented transformer (Conformer), that can be trained in an end-to-end manner. In particular, the audio and visual encoders learn to extract features directly from raw pixels and audio waveforms, respectively, which are then fed to conformers and then fusion takes place via a Multi-Layer Perceptron (MLP). The model learns to recognise characters using a combination of CTC and an attention mechanism. We show that end-to-end training, instead of using pre-computed visual features which is common in the literature, the use of a conformer, instead of a recurrent network, and the use of a transformer-based language model, significantly improve the performance of our model. We present results on the largest publicly available datasets for sentence-level speech recognition, Lip Reading Sentences 2 (LRS2) and Lip Reading Sentences 3 (LRS3), respectively. The results show that our proposed models raise the state-of-the-art performance by a large margin in audio-only, visual-only, and audio-visual experiments.
1. INTRODUCTION
AVSR transcribes speech from audio and visual streams, with visual information offering robustness as background noise increases. This work addresses limits of prior systems by training end-to-end from raw inputs and improving the model with conformers and a transformer language model.
- AVSR transcribes text from audio and visual streams and can improve over audio-only recognition as noise increases.
- Prior approaches commonly separate feature extraction from recognition or rely on pre-computed audio or visual features.
- Few prior audiovisual systems train truly end-to-end from raw pixels and audio waveforms, with limited task or environmental scope.
- The proposed model jointly integrates feature extraction with hybrid CTC/attention training, replaces recurrent networks with conformers, and uses a transformer-based language model.
- The architecture is evaluated on LRS2 and LRS3, where the authors report large state-of-the-art gains in audio-only, visual-only, and audio-visual experiments.
2. DATASETS
The study uses two large-scale public audiovisual speech datasets with substantial variation in recording conditions. LRS3 is approximately twice the size of LRS2.
- LRS2 contains 224.1 hours and 144,482 video clips from BBC programs.Its splits include pre-training, training, validation, and testing sets.
- Both datasets are challenging because they contain large variations in head pose and illumination.
- LRS3 contains 151,819 utterances totaling 438.9 hours from TED and TEDx talks.The dataset includes pre-training, training-validation, and test sets.
3. ARCHITECTURE
The architecture processes raw visual and acoustic inputs through modality-specific front-ends and conformer back-ends, fuses their representations with an MLP, and decodes with hybrid CTC/attention objectives.
- The encoder consists of front-end, back-end, and fusion modules processing pixels and raw audio waveforms.
- Front-end: The visual front-end uses a modified ResNet-18 with an initial 3D convolution, while the acoustic front-end uses a 1D-convolutional ResNet-18.Acoustic features are down-sampled to 25 frames per second to match visual features.
- Back-end: The back-end uses conformer blocks to model temporal information with attention and convolutional modules.The convolutional module is described as capturing local and global temporal information.
- Front-end: The acoustic and visual front-end backbones contain 3.85 M and 11.18 M parameters, respectively.
- Fusion Layers: Acoustic and visual back-end features are concatenated and projected to d_k-dimensional space by an MLP.
- Decoder and Loss functions: The decoder uses masked self-attention and encoder-decoder attention, while the hybrid objective combines CTC and attention losses.The loss weight α controls the relative contribution of the two mechanisms.
4. EXPERIMENTS
The experiments align and normalize mouth-region video with raw audio, apply modality-specific augmentation, and use decoding weights for CTC and language-model contributions.
- Pre-processing: Video preprocessing tracks 68 facial landmarks, aligns faces, crops 96 × 96 mouth regions, and normalizes grayscale inputs.
- Data augmentation: Audio waveforms are normalized, while augmentation adds babble noise, time masking, band rejection, and speed perturbation in audio-only experiments.Babble noise uses SNR levels from -5 dB to 20 dB.
- Experimental settings: At decoding, λ weights the CTC term and β weights the language model, with λ set to 0.1 and β set to 0.6.
5. RESULTS
The experiments evaluate ablations and audio-visual speech recognition on LRS2 and LRS3, including robustness to babble noise. End-to-end training, conformers, and multimodal inputs improve reported performance, with audio-visual gains increasing under heavy noise.
- Ablation studies: 12.6% absolute improvement results from training the visual model end-to-end rather than using the two-stage approach.The visual front-end is trained jointly with the hybrid CTC/Attention back-end.
- Ablation studies: 3.8% absolute improvement follows replacing LSTM encoders and decoders with a conformer encoder and transformer decoder.
- Ablation studies: 37.9% WER is achieved after replacing the RNN-based language model with a transformer-based language model.This change provides a 4.5% absolute improvement.
- Results on LRS2: 4.3% WER is achieved by the end-to-end audio-only model on LRS2, matching the log-Mel filter-bank model.The audio-only result improves over the current state of the art by 2.4% absolute.
- Results on LRS3: 43.3% WER is reported for the best visual-only LRS3 model, while audio-only and audio-visual models each achieve 2.3% WER on LRS3v0.4.The model uses 595 hours of training data versus 31,000 hours for the compared method.
- Noise robustness: The audio-visual model outperforms the audio-only model as babble-noise levels rise, with the gap becoming larger under high noise.Audio-only, visual-only, and audio-visual experiments vary the SNR of babble noise.
6. CONCLUSIONS
The paper presents an end-to-end encoder-decoder attention architecture for audio-visual speech recognition. It reports state-of-the-art results on LRS2 and LRS3, with especially strong audio-visual gains at high noise levels.
- The proposed encoder-decoder attention architecture is trained end-to-end and achieves state-of-the-art results on LRS2 and LRS3.
- Audio-visual models significantly outperform audio-only models, especially at high noise levels.
- Future work could investigate adaptive fusion that weighs each modality according to noise levels.