PaperlayerView exact revision on arXiv

Source-linked AI summary

Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels

Pingchuan Ma, Alexandros Haliassos, Adriana Fernandez-Lopez, Honglie Chen, Stavros Petridis, Maja Pantic

arXiv:2303.14307v3cs.CVcs.SDeess.AS

TL;DR

The paper investigates whether automatically generated transcriptions can expand audio-visual speech recognition training data without costly manual labelling. It transcribes unlabelled datasets with publicly available ASR models and trains ASR, VSR, and AV-ASR systems on the augmented data. The approach achieves state-of-the-art results, including 0.9% WER for AV-ASR on LRS3, while audio-visual models show stronger noise robustness than audio-only models.

  • Problem

    Accurate transcription of large speech datasets requires manual labelling that is time-consuming and prohibitively expensive.

    Source: §1 · p. 1

  • Method

    The method uses publicly available pre-trained ASR models to transcribe unlabelled audio-visual datasets and incorporates those transcriptions into ASR, VSR, and AV-ASR training.

    Source: §1 · p. 1

  • Results

    The proposed system achieves state-of-the-art performance on LRS2 and LRS3, including 0.9% WER for AV-ASR on LRS3.

    Source: §1 · p. 1, §5 · p. 4

  • Takeaways & Limitations

    Increasing the amount of automatically transcribed training data reduces WER, while the resulting audio-visual model is more robust to noise than its audio-only counterpart.

    Source: §4.2 · p. 3, §5 · p. 4

Abstract

from arXiv · show

Audio-visual speech recognition has received a lot of attention due to its robustness against acoustic noise. Recently, the performance of automatic, visual, and audio-visual speech recognition (ASR, VSR, and AV-ASR, respectively) has been substantially improved, mainly due to the use of larger models and training sets. However, accurate labelling of datasets is time-consuming and expensive. Hence, in this work, we investigate the use of automatically-generated transcriptions of unlabelled datasets to increase the training set size. For this purpose, we use publicly-available pre-trained ASR models to automatically transcribe unlabelled datasets such as AVSpeech and VoxCeleb2. Then, we train ASR, VSR and AV-ASR models on the augmented training set, which consists of the LRS2 and LRS3 datasets as well as the additional automatically-transcribed data. We demonstrate that increasing the size of the training set, a recent trend in the literature, leads to reduced WER despite using noisy transcriptions. The proposed model achieves new state-of-the-art performance on AV-ASR on LRS2 and LRS3. In particular, it achieves a WER of 0.9% on LRS3, a relative improvement of 30% over the current state-of-the-art approach, and outperforms methods that have been trained on non-publicly available datasets with 26 times more training data.

1. INTRODUCTION

The paper addresses the cost of manually labelling audio-visual speech datasets by using publicly available ASR models to generate transcriptions for unlabelled data. These transcriptions are incorporated directly into training, supporting large-scale ASR, VSR, and AV-ASR models.

  • Manual transcription is time-consuming and prohibitively expensive, motivating the use of large unlabelled audio-visual datasets.Source: §1 · p. 1
  • The proposed approach uses publicly available pre-trained ASR models to automatically annotate unlabelled audio-visual datasets.Source: §1 · p. 1
  • Automatically transcribed data are incorporated into training rather than used solely for knowledge distillation.Source: §1 · p. 1
  • More than 2 000 hours of videos are automatically transcribed, enabling ASR, VSR, and AV-ASR training on the augmented data.Source: §1 · p. 1
  • The approach achieves state-of-the-art performance on LRS2 and LRS3, including 0.9% WER for AV-ASR on LRS3.Source: §1 · p. 1

2. AUTO-AVSR

Auto-AVSR scales audio-visual speech recognition by generating labels for unlabelled clips with pre-trained ASR models, then training models on the expanded data. It evaluates several ASR label generators and uses a fused audio-visual architecture.

  • Unlabelled VoxCeleb2 and AVSpeech clips are transcribed by a pre-trained ASR model and added to the training sources.Source: §2.1 · p. 2
  • Four ASR models—Whisper, wav2vec 2.0, HuBERT, and Conformer-Transducer—provide the automatic transcriptions for comparison.Source: §2.2 · p. 2
  • The architecture uses separate audio and visual encoders whose outputs are fused through an MLP before projection and Transformer decoding.Source: §2.3 · p. 2

3. EXPERIMENTAL SETUP

Experiments use the public LRS2 and LRS3 benchmarks, with additional implementation and preprocessing choices documented for audio, visual, and audio-visual models. Table 1 evaluates how the transcription source affects downstream models.

  • LRS2 contains 144 482 clips totaling 225 hours, while LRS3 contains 151 819 clips from TED talks totaling 439 hours.Source: §3.1 · p. 2
  • Visual inputs are mouth-region crops normalized using training-set statistics, while audio inputs receive per-utterance z-normalization.Source: §3.2 · p. 2
  • Table 1 reports WER on Librispeech test-clean and LRS3 test for visual-only and audio-only models trained with 3 448 hours of data.Source: Table 1 · p. 3
  • Audio- and visual-only models use ResNet-based front ends, 12-layer Conformer encoders, and Transformer decoders.Source: §3.3 · p. 3
  • Visual-only models use a transformer-based language model, whereas ASR and AV-ASR models do not because no further improvements are observed.Source: §3.3 · p. 3

4. RESULTS

The experiments show that automatically transcribed data improves recognition performance across modalities, while benchmark ASR quality is not a strong predictor of downstream results. The models achieve state-of-the-art results on LRS2 and LRS3 and retain an advantage under increasing acoustic noise.

4.1. Do better Librispeech ASR models provide better transcriptions for VSR?

  • Conformer-Transducer transcriptions lead to the best average performance for the trained ASR and VSR models, despite Librispeech WER not strongly predicting downstream performance.Wav2vec 2.0, Whisper, and Conformer-Transducer produce similar WERs, while HuBERT is slightly worse despite strong Librispeech performance.Source: §4.1 · p. 3

4.2. Impact of the number of hours of unlabelled data

  • 1.3% audio-only WER and 26.6% visual-only WER are obtained on LRS3 after adding 526 hours of AVSpeech and VoxCeleb2.Further increases in training hours especially reduce VSR WER, although ASR gains become marginal beyond 1 578 hours.Source: §4.2 · p. 3, Table 2 · p. 3

4.3. Comparison with the state-of-the-art

  • 14.6% visual-only, 1.5% audio-only, and 1.5% audio-visual WER are achieved on LRS2, establishing state-of-the-art results for the reported models.The LRS2 comparison uses models trained with the listed labelled and automatically transcribed datasets.Source: §4.3 · p. 3, Table 3 · p. 4
  • 19.1% visual-only WER on LRS3 is surpassed only by a method using 26× more training data, while the audio-only model reaches 1.0% WER with 1 921 training hours.Introducing AVSpeech beyond this setting yields no further audio-only improvement, suggesting saturation.Source: §4.3 · pp. 3–4

4.4. Noise experiments

  • Audio-visual models show a growing performance advantage over audio-only models as acoustic noise increases.With babble noise, both model types achieve WER below 10% at -7.5 dB, likely because the noise type overlaps between training and testing.Source: §4.4 · p. 4

5. CONCLUSIONS

The paper scales audio-visual speech-recognition training with automatic transcriptions from publicly available ASR models and evaluates data scale and noise robustness. It achieves state-of-the-art AV-ASR on LRS2 and LRS3, including stronger robustness than audio-only recognition under varying noise levels.

5. CONCLUSIONS

  • The method scales audio-visual speech-recognition data by using publicly available ASR models to generate transcriptions for unlabelled videos.The study examines the effect of unlabelled training-data size on LRS3.Source: §5 · p. 4
  • State-of-the-art AV-ASR performance is achieved on both LRS2 and LRS3 using automatically generated transcriptions.The conclusion identifies both datasets as publicly available audio-visual benchmarks.Source: §5 · p. 4, Table 4 · p. 4
  • The audio-visual model is more robust to different noise levels than its audio-only counterpart.Noise performance is evaluated on LRS3 using the conditions described for Table 5.Source: §5 · p. 4, Table 5 · p. 4

Related papers

Loading 2303.14307v3…