Source-linked AI summary
MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation
Mohamed Anwar, Bowen Shi, Vedanuj Goswami, Wei-Ning Hsu, Juan Pino, Changhan Wang
TL;DR
Multilingual AVSR benchmarks are predominantly English, and labeled audio-visual speech translation data has been scarce. MuAViC assembles a large, transcribed nine-language corpus with translation directions, and its baselines show improved AVSR performance in noisy conditions.
Problem
Method
MuAViC aligns TED and TEDx audio and video with transcriptions and translation labels, building a 1,200-hour corpus across nine languages.
Source: §3 · p. 2Results
In noisy conditions, audio-visual monolingual AVSR models reduce average WER by 32% relative to their audio-only modes.
Source: §4.2.2 · p. 4Takeaways & Limitations
Abstract
from arXiv · showhide
We introduce MuAViC, a multilingual audio-visual corpus for robust speech recognition and robust speech-to-text translation providing 1200 hours of audio-visual speech in 9 languages. It is fully transcribed and covers 6 English-to-X translation as well as 6 X-to-English translation directions. To the best of our knowledge, this is the first open benchmark for audio-visual speech-to-text translation and the largest open benchmark for multilingual audio-visual speech recognition. Our baseline results show that MuAViC is effective for building noise-robust speech recognition and translation models. We make the corpus available at https://github.com/facebookresearch/muavic.
1. Introduction
MuAViC addresses the shortage of large multilingual audio-visual datasets for recognition and translation. It provides a nine-language corpus and benchmark for both tasks.
1. Introduction
- Audio-visual speech recognition uses visual lip movements alongside audio, which can improve robustness because visual information is immune to acoustic noise.State-of-the-art systems can lower WER by 3 times versus audio-only counterparts under 0 dB babble noise.Source: §1 · p. 1
- Most AVSR benchmarks are monolingual and English-focused, while non-English datasets are much smaller and AVST has relied on synthetic data.CMU-MOSEAS provides less than 20 hours for each of four non-English languages, at least 20 times smaller than LRS3-TED.Source: §1 · p. 1
- MuAViC provides 1,200 hours of transcribed audio-visual speech from over 8,000 speakers in nine languages, with translation labels for six directions each way.It is the largest open multilingual AVSR benchmark and the first publicly available AVST corpus.Source: §1 · p. 1
2. Related Works
Earlier AVSR resources were often small, controlled, or English-dominant, while multilingual translation-ready audio-visual data remained scarce. Large unlabeled collections do not directly fill that gap.
2. Related Works
- Early AVSR datasets such as GRID used controlled settings and had limited perspectives, scale, vocabulary, and speaker counts; later resources include LRW and LRS3-TED.LRW contains 173 hours of isolated-word clips, while LRS3-TED contains approximately 437 hours of English sentence-level videos.Source: §2 · p. 2
- Non-English AVSR datasets remained small: CMLR has 86 hours from 11 speakers, and CMU-MOSEAS has 68 hours across four languages.CMU-MOSEAS also lacks cross-lingual translations for AVST.Source: §2 · p. 2
- Large resources such as VoxCeleb2 and AV-Speech are unlabeled and thus not directly applicable to AVSR or AVST; labeled audio-visual translation remained largely uncharted.Most speech-translation corpora include audio only, even when video exists in the original source recordings.Source: §2 · p. 2
3. Corpus Creation
MuAViC combines TED and TEDx audio and video with human transcriptions and translation labels. Its scale and labeled test sets support multilingual recognition and translation evaluation.
3. Corpus Creation
- MuAViC collects TED and TEDx recordings, aligning audio and video with human transcriptions and text translations.Talks usually feature one native or non-native speaker on stage, with cameras switching viewpoints.Source: §3 · p. 2
- For English, the corpus reuses LRS3-TED and matches TED2020 translations; non-English data reuses mTEDx, with M2M-100 pseudo-labels for untranslated training examples.English development and test matches use exact text matching; training uses fuzzy matching to improve recall before pseudo-translation labels are added.Source: §3 · p. 2
- MuAViC contains 1,200 hours of transcribed audio-visual speech from more than 8,000 speakers across nine languages.It provides translations into or out of English for six languages.Source: §3 · pp. 2–3
- Every AVSR and AVST test set is human-labeled and contains at least 0.5 hours of speech from at least seven speakers.Source: §3 · p. 3
4. Experiments
In noisy conditions, audio-visual models outperform their audio-only counterparts in both recognition and translation. In clean conditions, multilingual AVSR slightly outperforms monolingual AVSR, while end-to-end AVST is on par with cascaded systems in the constrained-data setup.
4.1. Experimental Setup
- AVSR and AVST use pre-trained English AV-HuBERT; training includes noise augmentation, and evaluation uses WER for recognition and SacreBLEU for translation.A quarter of input samples are randomly augmented with additive noise at SNR 0; utterances shorter than 0.2 seconds or longer than 20 seconds are removed.Source: §4.1 · p. 3
4.2.1. Clean Setup
- Multilingual AVSR slightly outperforms monolingual AVSR on average in clean conditions, with gains on Arabic and German but degradation on Greek and Russian.Monolingual audio-visual AVSR reduces WER by an average 52% versus monolingual Transformer ASR, but has 14% higher WER than hybrid LF-MMI ASR.Source: §4.2.1 · pp. 3–4
4.2.2. Noisy Setup
- 32% average WER reduction for monolingual AVSR in audio-visual mode; multilingual AVSR outperforms monolingual counterparts on every non-English language except Greek in both modes.The test environment adds multilingual babble noise to clean speech at SNR 0.Source: §4.2.2 · p. 4, Table 4 · p. 4
4.3.1. Clean Setup
- End-to-end AVST is on par with cascaded systems in the constrained-data setup, and multilingual X-to-English AVST performs slightly better on average than bilingual AVST.The comparison covers both English-to-X and X-to-English directions.Source: §4.3.1 · p. 4
4.3.2. Noisy Setup
- Audio-visual AVST improves average BLEU over audio-only AVST by 3.2 for bilingual and 3.3 for multilingual models in noisy conditions.Audio-only bilingual and multilingual models exceed Whisper by 7.8 and 6.6 average BLEU, respectively; Whisper averages 0.3 BLEU across six directions.Source: §4.3.2 · p. 4, Table 4 · p. 4
5. Conclusion
MuAViC is an open multilingual audio-visual speech corpus intended for robust recognition and speech-to-text translation. It provides 1200 hours across nine languages and serves as an open benchmark for both tasks.
5. Conclusion
- MuAViC provides 1200 hours of audio-visual speech across nine languages.The corpus is introduced for robust speech recognition and speech-to-text translation.Source: §5 · p. 4
- MuAViC is the first open benchmark for audio-visual speech-to-text translation and the largest open benchmark for multilingual audio-visual speech recognition.Source: §5 · p. 4
- The corpus is available on GitHub.Source: §5 · p. 4
A.1. Performance of AV-HuBERT without pre-training
AV-HuBERT models without pre-training perform worse on speech recognition and speech-to-text translation than their pre-trained counterparts. This finding is consistent with prior English-only AVSR results.
A.1. Performance of AV-HuBERT without pre-training
- Randomly initialized AV-HuBERT models perform significantly worse than pre-trained counterparts on speech recognition and speech-to-text translation.Source: §A.1 · p. 6
- The comparison concerns AV-HuBERT models trained without pre-training on unlabeled audio-visual speech data.Source: §A.1 · p. 6
- The finding is consistent with results previously reported for English-only AVSR.Source: §A.1 · p. 6
A.2. Results for AVSR and AVST models in noisy environments with other noise types and SNRs
Audio-visual models consistently outperform audio-only models across multilingual babble, speech, and music noise at SNRs of -10, 0, and 10.
A.2. Results for AVSR and AVST models in noisy environments with other noise types and SNRs
- Audio-visual mode consistently outperforms audio-only mode across all tested noise types and SNRs.The reported sweep covers multilingual babble, speech, and music noise at SNRs of -10, 0, and 10.Source: §A.2 · p. 6, Table 6 · p. 7, Table 7 · p. 7
- Tables 6–8 cover multilingual babble, speech, and music noise at SNRs of -10, 0, and 10.Source: §A.2 · p. 6, Table 6 · p. 7, Table 7 · p. 7
- Table 5 reports randomly initialized AV-HuBERT performance for non-English speech recognition and X-En speech-to-text translation.Source: Table 5 · p. 7