PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-end Audio-visual Speech Recognition with Conformers

Pingchuan Ma, Stavros Petridis, Maja Pantic

arXiv:2102.06657v1cs.CVeess.AS

TL;DR

AVSR transcribes audio and visual streams and is studied for noise robustness; this paper jointly learns features from raw audio and pixels in an end-to-end recognition model. It reports state-of-the-art results on LRS2 and LRS3, with audio-visual models performing especially well against audio-only models at high noise levels.

  • Problem

    The paper addresses end-to-end AVSR that jointly learns features from raw audio and pixels rather than separating feature extraction from the recognition backend.

    Source: §1 · p. 1

  • Method

    The paper jointly trains a hybrid CTC/attention model with ResNet-18 and Conformer encoders for raw audio and video, then fuses their features through an MLP.

    Source: §1 · p. 1, §3 · p. 2

  • Results

    The models achieve state-of-the-art results on LRS2 and LRS3, and audio-visual models significantly outperform audio-only models especially at high noise levels.

    Source: §6 · p. 4, §5 · p. 4

  • Takeaways & Limitations

    Audio-visual recognition is particularly beneficial when background noise heavily corrupts the audio modality.

    Source: §5 · p. 4

    Pre-training and test sets contain overlapping identities.

    Source: §5 · p. 4

Abstract

from arXiv · show

In this work, we present a hybrid CTC/Attention model based on a ResNet-18 and Convolution-augmented transformer (Conformer), that can be trained in an end-to-end manner. In particular, the audio and visual encoders learn to extract features directly from raw pixels and audio waveforms, respectively, which are then fed to conformers and then fusion takes place via a Multi-Layer Perceptron (MLP). The model learns to recognise characters using a combination of CTC and an attention mechanism. We show that end-to-end training, instead of using pre-computed visual features which is common in the literature, the use of a conformer, instead of a recurrent network, and the use of a transformer-based language model, significantly improve the performance of our model. We present results on the largest publicly available datasets for sentence-level speech recognition, Lip Reading Sentences 2 (LRS2) and Lip Reading Sentences 3 (LRS3), respectively. The results show that our proposed models raise the state-of-the-art performance by a large margin in audio-only, visual-only, and audio-visual experiments.

1. INTRODUCTION

The paper presents an end-to-end AVSR model trained from raw pixels and waveforms, using a hybrid CTC/attention architecture with conformers and a transformer language model. It reports large state-of-the-art gains on LRS2 and LRS3 across audio-only, visual-only, and audio-visual experiments.

1. INTRODUCTION

  • Because visual input is unaffected by noise, AVSR can outperform audio-only recognition as noise increases; raw-audio models perform similarly in clean conditions and slightly better in noisy ones.The paper compares audio-only models trained on log-Mel filter-bank features and raw waveforms.Source: §1 · p. 1
  • Earlier AVSR methods often used precomputed features [1, 19, 25, 29, 34], while few trained from raw pixels and waveforms; addressed word classification and a constrained environment.Traditional systems separated feature extraction from recognition.Source: §1 · p. 1
  • The proposed models raise state-of-the-art performance by a large margin on LRS2 and LRS3 in audio-only, visual-only, and audio-visual experiments, even surpassing methods trained on much larger external datasets.The paper evaluates the architecture on the largest in-the-wild audio-visual speech datasets.Source: §1 · p. 1
  • The model jointly learns feature extraction from raw pixels and audio waveforms with a hybrid CTC/attention backend, then improves performance using conformers and a transformer-based language model.The authors replace recurrent networks with conformers and an RNN-based language model with a transformer-based one.Source: §1 · p. 1

2. DATASETS

The study uses two large-scale public audio-visual datasets, LRS2 and LRS3, which pose challenges from variation in head pose and illumination. LRS3, collected from TED and TEDx talks, is twice the size of LRS2, collected from BBC programs.

2. DATASETS

  • LRS2 is a large-scale publicly available audio-visual dataset comprising 224.1 hours across 144 482 video clips from BBC programs, with utterances assigned to pre-training, training, validation, and testing.Source: §2 · p. 1
  • Both public datasets are challenging because they contain substantial variation in head pose and illumination.LRS2 is sourced from BBC programs, while LRS3 [3] is collected from TED and TEDx talks.Source: §2 · p. 1
  • LRS3 [3] contains 151 819 utterances (438.9 hours), with 118 516 utterances for pre-training, 31 982 for training-validation, and 1 321 for testing.The sets comprise 408, 30, and 0.9 hours, respectively.Source: §2 · p. 1

3. ARCHITECTURE

The model processes raw pixels and audio waveforms through visual and acoustic front-ends, Conformer back-ends, and feature fusion, then predicts output symbols with a transformer decoder trained using a hybrid CTC/attention objective.

Architecture

  • The end-to-end architecture has visual and acoustic streams whose encoded features are fused before decoding.The architecture takes pixels and raw audio waveforms as inputs and comprises front-end, back-end, and fusion modules.Source: §3 · p. 1, Figure 1 · p. 2

Front-end

  • The visual front-end uses a modified ResNet-18 with a 3D first convolution, while the acoustic front-end uses a 1D-convolution ResNet-18 and downsamples features to 25 frames per second.Visual features are globally average-pooled across spatial dimensions; acoustic features are downsampled to match the visual frame rate.Source: §3 · p. 2

Back-end

  • Conformer back-ends model temporal features using feed-forward, self-attention, and convolutional modules, capturing local and global temporal information.The convolutional module’s sequence includes point-wise and temporal depth-wise convolutions; this combination is reported to improve ASR performance over the transformer architecture.Source: §3 · p. 2

Fusion Layers

  • The acoustic and visual back-end features are concatenated and projected to d_k dimensions by an MLP.The MLP uses a 4×d_k linear layer, batch normalization, ReLU, and a final d_k-dimensional linear layer.Source: §3 · p. 2

Decoder

  • The transformer decoder embeds target prefixes with absolute positional encoding and uses masked self-attention plus encoder-attending cross-attention.Its feed-forward module is the same as the encoder’s.Source: §3 · p. 2

Loss functions

  • The hybrid CTC/attention objective combines monotonic alignment constraints with attention’s removal of CTC’s conditional-independence assumption.The loss weights CTC and attention log-probabilities by α and 1−α, respectively.Source: §3 · p. 2

4. EXPERIMENTS

Experiments preprocess and augment audio and video inputs, train the encoder-decoder end to end, and incorporate a transformer language model through shallow fusion.

Pre-processing

  • Video is aligned using 68 tracked facial landmarks and cropped to 96 × 96 grayscale mouth regions; waveforms are normalized by mean and standard deviation.Video crops are normalized using training-set mean and variance.Source: §4 · pp. 2–3

Data augmentation

  • Video augmentation uses random 88 × 88 crops and horizontal flips, while audio augmentation adds noise, masks time spans, and rejects frequency bands.Audio experiments also use babble noise at SNRs from −5 to 20 dB or clean audio, and audio-only experiments add speed perturbation between 0.9 and 1.1.Source: §4 · p. 3

Experimental settings

  • The models use randomly initialized encoders, 12 Conformer blocks, and a six-block transformer decoder, trained end to end for 50 epochs.Front-end modules are sometimes initialized from LRW pre-trained models; training uses Adam with a peak learning rate of 0.0004, and excludes pre-training utterances longer than 600 frames.Source: §4 · p. 3

Language Model

  • A transformer language model trained on 16.2 million words is incorporated through shallow fusion, with λ = 0.1 and β = 0.6.Its training text combines LibriSpeech transcriptions with the LRS2 and LRS3 pre-training and training sets.Source: §4 · p. 3

5. RESULTS

The results report strong audio-only, visual-only, and audio-visual performance on LRS2 and LRS3, alongside gains from end-to-end training, Conformers, transformer language modeling, and visual input under noise.

Ablation Studies

  • End-to-end training improves the baseline by 12.6% absolutely, LRW visual initialization adds 4.7%, Conformer and transformer decoder replacements add 3.8%, and a transformer language model reaches 37.9% WER, a 4.5% absolute improvement.These are ablation results on LRS2.Source: §5 · p. 3, Table 2 · p. 3

Results on LRS2

  • Under babble noise, raw-audio slightly outperforms log-Mel across noise levels, with a maximum absolute margin of 7.5% at -5 dB; audio-visual also beats audio-only, increasingly at high noise.The growing gap indicates audio-visual input is particularly beneficial when audio is heavily corrupted by background noise.Source: Figure 2 · p. 3, §5 · pp. 3–4
  • On LRS2, visual-only WER falls from 48.3% to 39.1% using 6× fewer training data, and to 37.9% with LRW initialization; audio-only WER is 4.3%, 2.4% absolutely better than prior state of the art.LRW initialization lowers audio-only WER further to 3.9%; the audio-visual model gives a small improvement over audio-only.Source: §5 · p. 3, Table 3 · p. 4

Results on LRS3

  • On LRS3v0.4, visual-only WER is 43.3%, while audio-only and audio-visual WERs are both 2.3%, beating by 2.5% and 2.2%, respectively.The model was trained on 595 hours, a dataset 52× smaller than ’s 31,000 hours.Source: §5 · p. 4, Table 4 · p. 4

Results on LRS3v0.0

  • On older LRS3v0.0, audio-only, visual-only, and audio-visual WERs are 1.3%, 30.4%, and 1.2%, respectively, with 1.2% the new state of the art.The paper attributes the gains over LRS3v0.4 mainly to overlapping identities between pretraining and test sets.Source: §5 · p. 4

6. CONCLUSIONS

The paper presents an end-to-end encoder-decoder attention architecture for audio-visual speech recognition that achieves state-of-the-art results on LRS2 and LRS3. Audio-visual recognition outperforms audio-only recognition especially at high noise levels; adaptive modality weighting is proposed as future work.

6. CONCLUSIONS

  • The end-to-end encoder-decoder attention architecture achieves state-of-the-art audio-visual speech recognition results on LRS2 and LRS3.Audio-visual experiments also show that the model significantly outperforms the audio-only model, especially at high noise levels.Source: §6 · p. 4
  • Future work could investigate adaptive fusion that learns to weigh each modality according to noise levels.Source: §6 · p. 4

Related papers

Loading 2102.06657v1…