Source-linked AI summary
VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation
Changhan Wang, Morgane Rivière, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, Emmanuel Dupoux
TL;DR
Multilingual speech data is limited beyond a few high-resource languages, while open speech-to-speech interpretation resources are scarce. VoxPopuli addresses these gaps by assembling multilingual unlabeled, transcribed, and aligned interpretation speech from European Parliament recordings. In-domain unlabeled pre-training reduces average test WER by 59%, with especially large gains for low-resource languages.
Problem
Multilingual speech data is limited beyond a few high-resource languages, while open speech-to-speech interpretation resources are scarce.
Method
VoxPopuli combines European Parliament recordings with processing pipelines for multilingual unlabeled speech, transcriptions, and speech-to-speech alignment.
Results
59% average test WER reduction results from in-domain unlabeled pre-training, with the largest gains on low-resource languages.
Takeaways & Limitations
VoxPopuli provides open multilingual speech resources for unsupervised pre-training, self-training, and speech-to-speech interpretation.
Takeaways & Limitations
The evaluation uses only part of VoxPopuli’s unlabeled data, leaving validation on more data for future work.
Abstract
from arXiv · showhide
We introduce VoxPopuli, a large-scale multilingual corpus providing 100K hours of unlabelled speech data in 23 languages. It is the largest open data to date for unsupervised representation learning as well as semi-supervised learning. VoxPopuli also contains 1.8K hours of transcribed speeches in 16 languages and their aligned oral interpretations into 5 other languages totaling 5.1K hours. We provide speech recognition baselines and validate the versatility of VoxPopuli unlabelled data in semi-supervised learning under challenging out-of-domain settings. We will release the corpus at https://github.com/facebookresearch/voxpopuli under an open license.
1 Introduction
Speech pre-training and semi-supervised learning have advanced speech-to-text, but multilingual resources remain concentrated in a few high-resource languages. VoxPopuli addresses this gap with a large multilingual corpus supporting representation learning, semi-supervised learning, and interpretation.
- Multilingual speech datasets remain scarce beyond a few high-resource languages, despite recent progress in multilingual ASR and speech translation.
- Interpretation data differs from written translation, and open speech-to-speech resources remain limited in size or unavailable.
- VoxPopuli introduces a large-scale multilingual speech corpus for representation learning, semi-supervised learning, and interpretation.
- The corpus includes unlabeled speech, transcribed speech, and aligned oral interpretations across many languages.
- The paper describes corpus creation, analyzes the resulting data, and evaluates ASR and multilingual unlabeled or weakly labeled data in non-English settings.
2 Corpus Creation
VoxPopuli is constructed from European Parliament recordings by collecting, segmenting, filtering, transcribing, and aligning multilingual source and target speech. Its processing pipelines produce unlabeled, transcribed, and speech-to-speech resources, including weakly labeled reversed-direction translation data.
- Data Acquisition: VoxPopuli uses 2009–2020 European Parliament recordings containing multilingual speeches, transcripts, speaker information, and interpretations.
- Unlabeled Speech: Unlabeled speech is segmented into 15–30-second clips with voice activity detection, and silent clips are discarded.Around 16% of the data is removed after silence filtering.
- Transcribed Speech: Transcribed speech is aligned using diarization, ASR force alignment, punctuation or silence boundaries, and filtering of potentially incorrect segments.Long speech paragraphs are further divided into utterances of at most 20 seconds.
- Transcribed Speech: Train, development, and test sets are split by speaker groups using an 18:1:1 target duration ratio to increase speaker diversity in evaluation sets.
- Speech-To-Speech Alignment: Speech-to-speech alignment removes identical audio, segments target speech, decodes it with ASR and a language model, and force-aligns decoded words.
- Speech-To-Speech Alignment: Candidate target segments are selected by maximizing cosine similarity between LASER representations of source text and decoded target text.
- Speech-To-Speech Alignment: Human transcription of 400 hours of English target speech provides ground-truth speech-to-speech alignments, while reversed-direction alignments create weakly labeled speech-to-text data.
- Language Modeling: Language models combine VoxPopuli transcripts with EuroParl text after normalization and punctuation processing.
3 Data Analysis
VoxPopuli provides broad multilingual coverage across unlabeled, transcribed, and aligned interpretation data, while analyses characterize alignment quality and interpretese. Its ASR and representation-learning evaluations show benefits from multilingual and unlabeled data, especially for lower-resource settings.
- Unlabeled Speech: 400K hours of unlabeled speech are distributed across 23 EU languages, with 8K–24K hours per language.The distribution supports multilingual training with limited data imbalance.
- Transcribed Speech: 1.8K hours of transcribed speech cover 16 languages and 4.3K speakers, while seven languages lack transcribed coverage because of limited data or unavailable processing pipelines.
- Speech-to-Speech Alignment: Aligned target speech covers more than 70% of source sentences in five languages, more than 50% in five others, and at least 40% for the remainder.
- Speech-to-Speech Alignment: Human-transcription alignment data is 1.1 times larger than ASR-based alignment data, with approximately 12% of low-quality alignments filtered because of ASR errors.ASR-based and ground-truth boundaries differ by an average of 0.75 seconds.
- Interpretese vs. Translationese: Interpretese is more general and summarizing than verbatim translationese, dropping unimportant details as interpreters trade off quality and latency.
- Unsupervised Representation Learning: Multilingual representations generalize better and are more robust on unseen languages according to cross-language phoneme discriminability evaluation.
4 Experiments & Results
VoxPopuli data supports multilingual representation learning, ASR, speech translation, and semi-supervised learning across in-domain, out-of-domain, and out-of-language settings. Experiments show gains from unsupervised pre-training, self-training, and weakly labeled speech-to-speech data, while multilingual representations generalize across languages.
- Unsupervised Representation Learning: Multilingual representations show lower cross-language score variation and generalize better to unseen languages than monolingual representations.The multilingual model’s variance is comparable to Fr-500 while its average score is lower than Fr-500.
- ASR with Unsupervised Pre-Training: 59% average test WER reduction results from in-domain pre-training across languages, with the largest gains on low-resource languages.Sl improves from nearly 100% test WER to around 30%.
- ASR with Unsupervised Pre-Training: VP-100K outperforms XLSR-10 on 8 (9) of 10 Common Voice languages and performs competitively with XLSR-53 in out-of-domain phoneme recognition.VP-100K also outperforms XLSR-53 on Zh.
- ASR with Unsupervised Pre-Training: VP-50K outperforms the supervised DeepSpeech-Polyglot baseline using the standard Common Voice training set, even without language-model decoding.The baseline uses an extended Common Voice set, several other corpora, and a language model.
- ASR with Unsupervised Pre-Training: VP-100K outperforms in-domain in-language XLSR baselines on most uncovered languages and has lower PER variance across all 10 languages.The model lacks pre-training coverage for Ky, Ru, Tr, Tt, and Zh but uses data from 18 additional EU languages.
- Self-Training for ASR and ST: Self-training improves both in-domain and out-of-domain ASR and ST performance, while narrowing the gap between end-to-end and cascaded ST models without additional expensive labeled data.The labeled data combines EuroParl-ST and CoVoST 2, and VoxPopuli unlabeled data is pseudo-labeled.
- Weakly Supervised ST: Joint training with weakly labeled data yields substantial in-domain and out-of-domain gains and outperforms self-training.The weakly labeled set contains 0.4K hours, compared with 3K hours of pseudo-labeled data for self-training.
5 Related Work
Prior multilingual speech corpora span several languages and domains, but their scale, openness, or availability remains limited. Existing interpretation resources are likewise smaller or unavailable for download.
- LibriLight offers the largest-scale unlabeled speech corpus, but it is limited to English.
- MLS and MAILABS provide multilingual read speech, while VoxForge and Common Voice collect crowdsourced speech at smaller scales.MLS covers 8 languages, MAILABS 9 languages, and VoxForge about 300 total hours.
- IARPA Babel covers 24 languages but is not released under an open license and provides 25–65 labeled hours per language.
- Existing interpretation corpora include EuroParl-ST, EPIC, CIAIR, STC, and MaSS, with some unavailable and others totaling only about 23–180 hours.
6 Conclusion
VoxPopuli is introduced as a large-scale multilingual speech corpus for representation learning, semi-supervised learning, and interpretation. The paper releases open unlabeled speech data, interpretation data, ASR baselines, and semi-supervised validation.
- VoxPopuli is a large-scale multilingual speech corpus for representation learning, semi-supervised learning, and interpretation.
- The corpus provides the largest open unlabeled speech data to date and the first large-scale open speech-to-speech interpretation data.
- The paper provides ASR baselines and validates unlabeled-data versatility for semi-supervised learning under challenging out-of-domain settings.
8 Ethical Considerations
The authors acknowledge the European Union for creating and publishing the materials used by VoxPopuli. The corpus includes European Parliament speeches from 2009–2020 without topic or speaker selection.
- The authors acknowledge the European Union for creating and publishing the materials used by VoxPopuli.
- VoxPopuli includes all available speeches from 2009–2020 European Parliament events without selecting topics or speakers.The speeches represent the standpoints of speakers, many of whom are EU officials.