Source-linked AI summary
Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels
Pingchuan Ma, Alexandros Haliassos, Adriana Fernandez-Lopez, Honglie Chen, Stavros Petridis, Maja Pantic
TL;DR
The paper investigates whether automatically generated transcriptions can expand audio-visual speech recognition training data without costly manual labelling. It transcribes unlabelled datasets with publicly available ASR models and trains ASR, VSR, and AV-ASR systems on the augmented data. The approach achieves state-of-the-art results, including 0.9% WER for AV-ASR on LRS3, while audio-visual models show stronger noise robustness than audio-only models.
Problem
Accurate transcription of large speech datasets requires manual labelling that is time-consuming and prohibitively expensive.
Method
The method uses publicly available pre-trained ASR models to transcribe unlabelled audio-visual datasets and incorporates those transcriptions into ASR, VSR, and AV-ASR training.
Results
The proposed system achieves state-of-the-art performance on LRS2 and LRS3, including 0.9% WER for AV-ASR on LRS3.
Takeaways & Limitations
Increasing the amount of automatically transcribed training data reduces WER, while the resulting audio-visual model is more robust to noise than its audio-only counterpart.
Abstract
from arXiv · showhide
Audio-visual speech recognition has received a lot of attention due to its robustness against acoustic noise. Recently, the performance of automatic, visual, and audio-visual speech recognition (ASR, VSR, and AV-ASR, respectively) has been substantially improved, mainly due to the use of larger models and training sets. However, accurate labelling of datasets is time-consuming and expensive. Hence, in this work, we investigate the use of automatically-generated transcriptions of unlabelled datasets to increase the training set size. For this purpose, we use publicly-available pre-trained ASR models to automatically transcribe unlabelled datasets such as AVSpeech and VoxCeleb2. Then, we train ASR, VSR and AV-ASR models on the augmented training set, which consists of the LRS2 and LRS3 datasets as well as the additional automatically-transcribed data. We demonstrate that increasing the size of the training set, a recent trend in the literature, leads to reduced WER despite using noisy transcriptions. The proposed model achieves new state-of-the-art performance on AV-ASR on LRS2 and LRS3. In particular, it achieves a WER of 0.9% on LRS3, a relative improvement of 30% over the current state-of-the-art approach, and outperforms methods that have been trained on non-publicly available datasets with 26 times more training data.
1. INTRODUCTION
The paper addresses the cost of manually labelling audio-visual speech datasets by using publicly available ASR models to generate transcriptions for unlabelled data. These transcriptions are incorporated directly into training, supporting large-scale ASR, VSR, and AV-ASR models.
- Manual transcription is time-consuming and prohibitively expensive, motivating the use of large unlabelled audio-visual datasets.
- The proposed approach uses publicly available pre-trained ASR models to automatically annotate unlabelled audio-visual datasets.
- Automatically transcribed data are incorporated into training rather than used solely for knowledge distillation.
- More than 2 000 hours of videos are automatically transcribed, enabling ASR, VSR, and AV-ASR training on the augmented data.
- The approach achieves state-of-the-art performance on LRS2 and LRS3, including 0.9% WER for AV-ASR on LRS3.
2. AUTO-AVSR
Auto-AVSR scales audio-visual speech recognition by generating labels for unlabelled clips with pre-trained ASR models, then training models on the expanded data. It evaluates several ASR label generators and uses a fused audio-visual architecture.
- Unlabelled VoxCeleb2 and AVSpeech clips are transcribed by a pre-trained ASR model and added to the training sources.
- Four ASR models—Whisper, wav2vec 2.0, HuBERT, and Conformer-Transducer—provide the automatic transcriptions for comparison.
- The architecture uses separate audio and visual encoders whose outputs are fused through an MLP before projection and Transformer decoding.
3. EXPERIMENTAL SETUP
Experiments use the public LRS2 and LRS3 benchmarks, with additional implementation and preprocessing choices documented for audio, visual, and audio-visual models. Table 1 evaluates how the transcription source affects downstream models.
- LRS2 contains 144 482 clips totaling 225 hours, while LRS3 contains 151 819 clips from TED talks totaling 439 hours.
- Visual inputs are mouth-region crops normalized using training-set statistics, while audio inputs receive per-utterance z-normalization.
- Table 1 reports WER on Librispeech test-clean and LRS3 test for visual-only and audio-only models trained with 3 448 hours of data.
- Audio- and visual-only models use ResNet-based front ends, 12-layer Conformer encoders, and Transformer decoders.
- Visual-only models use a transformer-based language model, whereas ASR and AV-ASR models do not because no further improvements are observed.
4. RESULTS
The experiments show that automatically transcribed data improves recognition performance across modalities, while benchmark ASR quality is not a strong predictor of downstream results. The models achieve state-of-the-art results on LRS2 and LRS3 and retain an advantage under increasing acoustic noise.
- 4.1. Do better Librispeech ASR models provide better transcriptions for VSR?: Conformer-Transducer transcriptions lead to the best average performance for the trained ASR and VSR models, despite Librispeech WER not strongly predicting downstream performance.Wav2vec 2.0, Whisper, and Conformer-Transducer produce similar WERs, while HuBERT is slightly worse despite strong Librispeech performance.
- 4.2. Impact of the number of hours of unlabelled data: 1.3% audio-only WER and 26.6% visual-only WER are obtained on LRS3 after adding 526 hours of AVSpeech and VoxCeleb2.Further increases in training hours especially reduce VSR WER, although ASR gains become marginal beyond 1 578 hours.
- 4.3. Comparison with the state-of-the-art: 14.6% visual-only, 1.5% audio-only, and 1.5% audio-visual WER are achieved on LRS2, establishing state-of-the-art results for the reported models.The LRS2 comparison uses models trained with the listed labelled and automatically transcribed datasets.
- 4.3. Comparison with the state-of-the-art: 19.1% visual-only WER on LRS3 is surpassed only by a method using 26× more training data, while the audio-only model reaches 1.0% WER with 1 921 training hours.Introducing AVSpeech beyond this setting yields no further audio-only improvement, suggesting saturation.
- 4.4. Noise experiments: Audio-visual models show a growing performance advantage over audio-only models as acoustic noise increases.With babble noise, both model types achieve WER below 10% at -7.5 dB, likely because the noise type overlaps between training and testing.
5. CONCLUSIONS
The paper scales audio-visual speech-recognition training with automatic transcriptions from publicly available ASR models and evaluates data scale and noise robustness. It achieves state-of-the-art AV-ASR on LRS2 and LRS3, including stronger robustness than audio-only recognition under varying noise levels.
- 5. CONCLUSIONS: The method scales audio-visual speech-recognition data by using publicly available ASR models to generate transcriptions for unlabelled videos.The study examines the effect of unlabelled training-data size on LRS3.
- 5. CONCLUSIONS: State-of-the-art AV-ASR performance is achieved on both LRS2 and LRS3 using automatically generated transcriptions.The conclusion identifies both datasets as publicly available audio-visual benchmarks.
- 5. CONCLUSIONS: The audio-visual model is more robust to different noise levels than its audio-only counterpart.Noise performance is evaluated on LRS3 using the conditions described for Table 5.