Source-linked AI summary
MLS: A Large-Scale Multilingual Dataset for Speech Research
Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, Ronan Collobert
TL;DR
Multilingual speech research lacks a large, openly available, transcribed resource spanning multiple languages. The paper constructs MLS from LibriVox audiobooks using alignment and post-processing, and releases language models and ASR baselines. MLS provides 36.5K hours of training data across eight languages and is intended to support multilingual ASR and TTS research.
Problem
Existing speech benchmarks are often English-only, low-scale, scattered, or unavailable under an open license, limiting openly available multilingual research resources.
Method
The paper constructs MLS from LibriVox audiobooks by segmenting audio, generating pseudo-labels, retrieving source transcripts, and applying post-processing, while releasing language models and ASR baselines.
Results
36.5K hours of training data are provided across eight languages, and 5-gram language-model decoding improves WER for all languages except Polish.
Takeaways & Limitations
MLS is intended to promote open research in large-scale and multilingual ASR and to support multilingual TTS research.
Takeaways & Limitations
Development and test splits are designed without speaker overlap, but the dataset retains book overlap across partitions.
Abstract
from arXiv · showhide
This paper introduces Multilingual LibriSpeech (MLS) dataset, a large multilingual corpus suitable for speech research. The dataset is derived from read audiobooks from LibriVox and consists of 8 languages, including about 44.5K hours of English and a total of about 6K hours for other languages. Additionally, we provide Language Models (LM) and baseline Automatic Speech Recognition (ASR) models and for all the languages in our dataset. We believe such a large transcribed dataset will open new avenues in ASR and Text-To-Speech (TTS) research. The dataset will be made freely available for anyone at http://www.openslr.org.
1 Introduction
MLS addresses the lack of a large, openly available multilingual counterpart to the English-only LibriSpeech benchmark. It introduces an 8-language read-speech dataset derived from public-domain LibriVox and Project Gutenberg data, alongside models and research resources.
- MLS extends the freely available LibriSpeech benchmark from English-only speech recognition to a multilingual setting at larger scale.
- The dataset contains 44.5K hours of English and 6K hours distributed across seven other languages.
- The paper covers dataset creation, data splits, language models, and baseline ASR experiments.
2 Related Work
Prior multilingual speech datasets vary widely in scale, language coverage, domain, and openness. MLS is situated among these efforts as a large-scale, openly distributable audiobook-based resource.
- LibriSpeech provides about 1000 hours of openly licensed labeled English audio, whereas MLS targets multilingual coverage at larger scale.
- Babel covers 24 mostly conversational telephone-speech languages, but its data is not released under an open license.
- CommonVoice offers more than 30 languages and 4500 validated hours, while M-AILABS provides about 1000 hours across 9 LibriVox languages.
3 Data processing pipeline
The MLS pipeline converts long multilingual LibriVox audiobooks into supervised speech segments by combining segmentation, pseudo-label generation, source-text retrieval, alignment, and transcript post-processing. It uses language-specific resources and filters to improve label quality.
- 3.1 Downloading audiobooks: The pipeline selects eight languages, downloads LibriVox audio, and downsamples it from 48kHz to 16kHz.
- 3.2 Audio segmentation: Long audiobook files are segmented into 10–20 second utterances using streaming acoustic-model inference and silence-aware split points.
- 3.3 Pseudo-label generation: Beam-search decoding with a 4-gram language model generates pseudo labels for the segmented audio.
- 3.4 Text extraction: Source texts are obtained through automated parsers and manual recovery of alternate sources, especially for non-English audiobooks.
- 3.5 Transcript retrieval: Transcript retrieval matches pseudo labels to overlapping source-text windows with TF-IDF and Smith-Waterman alignment, rejecting candidates whose WER exceeds 40%.
- 3.5.1 Post processing of numbers, hyphens, apostrophe: Post-processing handles number ambiguity and removes or replaces problematic hyphenated and apostrophe-containing word forms.
4 Putting it together
MLS constructs train, development, and test partitions across eight languages using speaker-based assignment, gender balancing, and quality controls. The section also reports partition statistics, limited-supervision sets, transcript verification, and segment-duration characteristics.
- Creating train, development and test splits: MLS partitions speakers into training, development, and test sets without speaker overlap, while balancing gender and duration in development and test.The split procedure selects shorter-duration speakers for training and balances selected speakers across genders for development and test.
- Creating train, development and test splits: The dataset excludes recordings with multiple speakers and removes books with corrupted metadata before transcript retrieval.Only valid books and their speakers are retained for subsequent processing.
- Creating train, development and test splits: Gender labels are assigned with an SVM classifier whose LibriSpeech test accuracy is 95%, with manually checked Dutch and Polish accuracies of 96% and 94%.The classifier uses 40-dimensional log-filterbank features averaged over time.
- Creating train, development and test splits: English MLS training is exclusive to LibriSpeech development and test sets, although the two training sets have speaker and book overlap.The development and test sets are completely independent from one another, while book overlap can occur across partitions because chapters may have different speakers.
- Limited supervision train data: MLS provides 10-hour, 1-hour, and six 10-minute limited-supervision sets for each language.The six 10-minute sets together form the 1-hour set, which is included in the 10-hour set.
- High quality development and test sets: Human transcribers evaluate development and test transcripts by listening to audio and correcting detected mistakes, with transcript differences summarized using WER.Figure 6 illustrates corrections, while Table 3 reports WER between generated and human-rated transcripts.
5 Language models
MLS releases language-model training data and pre-built models for all dataset languages. The models use filtered and normalized book text, with corpus statistics, OOV rates, and perplexities reported for evaluation.
- Language-model resources: MLS releases language-model training data and pre-built language models alongside the speech dataset.The training text comes from Project Gutenberg books and is normalized using the dataset’s text-processing steps.
- Corpus preparation: The language-model corpus filters books to reduce overlap with development and test sets using title similarity and 5-gram matching.Books with title edit distance below 2 are removed, and candidate books are checked against an inverted index of development and test 5-grams.
- Language-model resources: MLS trains 3-gram and 5-gram language models for every dataset language using KenLM.Table 4 reports corpus statistics, while Table 5 reports development-set OOV rates and perplexities.
6 Experiments and results
The paper trains baseline ASR systems and evaluates decoding across the eight languages, including comparisons between language-model-assisted and language-model-free decoding. Using MLS English training data also improves performance on LibriSpeech under the same model setup.
- Monolingual Baselines: Baseline acoustic models use a convolutional frontend followed by 36 four-head Transformer blocks and are trained end-to-end with CTC loss.The models use 80-channel log-mel filterbanks and SpecAugment during training.
- Monolingual Baselines: Viterbi WER is reported on development and test sets for all dataset languages.Table 6 presents the monolingual baseline results under different decoding strategies.
- Monolingual Baselines: WER improves with 5-gram language-model beam-search decoding for every language except Polish.The paper attributes the Polish exception to its 13% OOV rate and constrained-lexicon decoding.
- Comparison with LibriSpeech: Using MLS English training data improves LibriSpeech development and test performance with the same model architecture.The comparison uses the LibriSpeech 4-gram LM for decoding and a transformer LM trained on the LibriSpeech corpus and training data.
7 Conclusions
MLS is presented as a large-scale multilingual speech dataset with 36.5K hours of training data across eight languages. The authors position it for open ASR, multilingual ASR, and expanded multilingual TTS research.
- Conclusions: MLS contains 36.5K hours of training data spread over eight languages.The authors present the dataset as a resource for large-scale and multilingual speech research.
- Conclusions: The dataset is intended to promote open research in large-scale ASR and multilingual ASR.The authors also propose extending LibriTTS with MLS for larger and multilingual TTS research.