Source-linked AI summary
Robust Self-Supervised Audio-Visual Speech Recognition
Bowen Shi, Wei-Ning Hsu, Abdelrahman Mohamed
TL;DR
ASR can fail under noise, while AVSR’s reliance on labeled data limits its reach in low-resource settings. This paper adapts self-supervised AV-HuBERT for robust AVSR, reporting lower babble-noise WER than prior work with substantially fewer labels.
Problem
ASR degrades under noise, and labeled-data demands constrain AVSR’s reach in low-resource settings.
Source: §1 · p. 1Method
The framework adapts self-supervised AV-HuBERT, using unlabeled audio-visual speech for pretraining and noise augmentation during pretraining and fine-tuning.
Source: §1 · p. 1, §2.2 · p. 2Results
With 30 hours of labels, the model achieves 49.6% lower WER (28.0%→14.1%) than prior SOTA on average across babble-noise SNR ratios, using ten times fewer labels.
Source: §3.2 · p. 3Takeaways & Limitations
The model achieves high recognition accuracy and robustness across noise categories even with few hours of labeled data.
Source: §4 · p. 4A direct comparison with was constrained because its LRS2 audio was restricted and unavailable to the authors.
Source: §3.3.1 · p. 3
Abstract
from arXiv · showhide
Audio-based automatic speech recognition (ASR) degrades significantly in noisy environments and is particularly vulnerable to interfering speech, as the model cannot determine which speaker to transcribe. Audio-visual speech recognition (AVSR) systems improve robustness by complementing the audio stream with the visual information that is invariant to noise and helps the model focus on the desired speaker. However, previous AVSR work focused solely on the supervised learning setup; hence the progress was hindered by the amount of labeled data available. In this work, we present a self-supervised AVSR framework built upon Audio-Visual HuBERT (AV-HuBERT), a state-of-the-art audio-visual speech representation learning model. On the largest available AVSR benchmark dataset LRS3, our approach outperforms prior state-of-the-art by ~50% (28.0% vs. 14.1%) using less than 10% of labeled data (433hr vs. 30hr) in the presence of babble noise, while reducing the WER of an audio-based model by over 75% (25.8% vs. 5.8%) on average.
1. Introduction
The section motivates robust AVSR by contrasting noisy ASR’s vulnerability with visual speech cues, then identifies labeled-data dependence as a barrier and presents a self-supervised approach.
1. Introduction
- ASR can degrade drastically when recordings are noisy, motivating efforts to improve reliability across scenarios.Source: §1 · p. 1
- AVSR combines audio with noise-invariant lip movement information and supports scenarios where both audio and visual streams are accessible.Source: §1 · p. 1
- Modern AVSR relies on costly labeled data, limiting low-resource use, including for most of the approximately 7,000 spoken languages.Source: §1 · p. 1
- The proposed framework pretrains on unlabeled audio-visual speech and fine-tunes on limited transcribed data, evaluating low- and mid-resource settings and multiple noise types.Source: §1 · p. 1
2. Method
The method adapts AV-HuBERT’s self-supervised audio-visual representations for speech recognition, then adds noise during fine-tuning and pretraining to improve robustness.
2. Method
- The method uses AV-HuBERT pretraining for joint audio-visual representations and adapts the model for robust AVSR.Source: §2 · p. 1
2.1. AV-HuBERT for Audio-Visual Speech Recognition
- AV-HuBERT processes synchronized audio and video, iterating between frame-level feature clustering and masked prediction of cluster assignments.For AVSR fine-tuning, both modalities remain active in a sequence-to-sequence model; the pretrained encoder is initially frozen, then all weights are updated.Source: §2.1 · p. 2
2.2. Noise-Augmented AV-HuBERT
- Noise-augmented fine-tuning adds sampled noise to clean audio, while noise-augmented pretraining randomly adds noise to the audio input.Pretraining retains cluster assignments inferred from clean audio-visual speech because the associated phonetic information should be noise-invariant.Source: §2.2 · p. 2
- Unlike WavLM’s audio-only utterance mixing, the approach uses diverse speech and non-speech noise and allows flexible mixing because video distinguishes the primary speaker.Source: §2.2 · p. 2
3. Experiments
Experiments on LRS3 evaluate self-supervised AVSR with 30h and 433h of labeled data, comparing input modalities and pretraining conditions. The results show gains over prior AVSR and audio-only systems, with noise-augmented pretraining and visual input especially beneficial under noisy conditions.
3.1. Data and Experimental Setup
- LRS3 provides about 433h of labeled speech from over 5,000 speakers; VoxCeleb2 supplies 1,326h of English video for self-supervised pretraining.The experiments also used 30h and 433h labeled-data settings and evaluated noise at five SNR levels across noise types.Source: §3.1 · p. 2
3.3. Analysis
- The analysis compares six combinations of audio or audio-visual input with no, clean-audio, or noise-augmented pretraining.Figure 2 breaks performance down by SNR and noise type.Source: §3.3 · p. 3
3.3.1. Effect of the visual modality
- Under noisy settings, AVSR achieved relative WER reductions over audio-only ASR of 53.0% (42.6%→20.0%) with 30 hours and 70.8% (31.3%→9.2%) with 433 hours of labeled data, averaged across pre-training configurations.AVSR outperformed audio-only ASR across SNR and noise settings except when trained from scratch with 30 hours.Source: §3.3.1 · p. 3
3.2. Main Results
- 49.6% lower WER (28.0%→14.1%) than prior SOTA was achieved with 30h—ten times fewer labels—and the gain reached 55.7% (28.0%→12.4%) with 433h.The comparison is on babble noise averaged across SNR ratios; the authors note potential domain mismatch between training and test babble noise.Source: §3.2 · p. 3, Table 2 · p. 4
3.3.2. Effect of pre-training
- 78.3% (42.9%→9.3%) and 53.4% (14.8%→6.9%) average relative improvements followed AV-HuBERT pretraining with 30h and 433h of labeled data, respectively.Gains were larger in the low-resource setting.Source: §3.3.2 · p. 3, Table 3 · p. 4
3.3.3. Effect of noise-augmented pre-training
- 16.1% and 15.9% average WER reductions over clean-audio pretraining were reported with noise-augmented pretraining at 30h and 433h, respectively.Gains were largest at low SNR and peaked around 0dB; noise-augmented pretraining was especially effective with overlapping speech, and its gains generalized across architectures and finetuning strategies.Source: Table 4 · p. 4, §3.3.3 · p. 4
4. Conclusion
The paper presents an AV-HuBERT-based AVSR model trained with large volumes of unlabeled audio-visual speech data. It reports robust recognition with few labeled hours and performance about 50% better than prior state of the art using less than 10% of labeled data.
4. Conclusion
- With less than 10% of labeled data, the model outperforms prior state of the art by approximately 50%.The paper also reports high recognition accuracy and robustness across different noise categories with a few hours of labeled data.Source: §4 · p. 4
- The model is based on AV-HuBERT and is the paper’s first attempt to build AVSR using a large volume of unlabeled audio-visual speech data.The paper describes AV-HuBERT as an approach for multimodal speech representation learning.Source: §4 · p. 4
- Future work includes applying AVSR in real-world low-resource and multilingual settings.Source: §4 · p. 4
6. Appendix
Tables 5 and 6 report WERs for LARGE and BASE AV-HuBERT under different noise types and levels.
6.1. Full results
- Tables 5 and 6 report WERs for LARGE and BASE AV-HuBERT under different noise types and SNR levels.Source: §6.1 · p. 5
- Table 5 gives test WER (%) for LARGE AV-HuBERT across noise types and levels, with lower values indicating better performance.Its noise abbreviations are B for babble, S for speech, M for music, and N for natural noise.Source: Table 5 · p. 6
- Table 6 gives test WER (%) for BASE AV-HuBERT fine-tuned with 30 hours of labeled data across noise types and levels.Lower WER is better; the abbreviations are B for babble, S for speech, M for music, and N for natural noise.Source: Table 6 · p. 7