PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Speech Technology to 1,000+ Languages

Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, Michael Auli

arXiv:2305.13516v1cs.CLcs.SDeess.AS

TL;DR

Speech technology covers only a small fraction of the world’s more than 7,000 languages. MMS combines multilingual datasets with self-supervised learning to scale speech models across tasks, reducing Whisper’s word error rate by 58% on 54 FLEURS languages while using less labeled data.

  • Problem

    Speech technology supports only a small fraction of the world’s over 7,000 languages, limiting coverage relative to global linguistic diversity.

  • Method

    MMS builds multilingual labeled and unlabeled speech datasets, pre-trains wav2vec 2.0 models, and fine-tunes systems for recognition, identification, and synthesis.

  • Results

    MMS reduces Whisper’s word error rate by a relative 58% while supporting over 11 times as many languages, using 44.7K versus 680K hours of labeled data.

  • Takeaways & Limitations

    MMS scales major speech tasks by 10-40x, including recognition and synthesis for 1,107 languages and language identification for 4,017 languages.

  • Takeaways & Limitations

    MMS still covers only about 1,100-4,000 of over 7,000 languages, and dialects remain inadequately represented in training data.

Abstract

from arXiv · show

Expanding the language coverage of speech technology has the potential to improve access to information for many more people. However, current speech technology is restricted to about one hundred languages which is a small fraction of the over 7,000 languages spoken around the world. The Massively Multilingual Speech (MMS) project increases the number of supported languages by 10-40x, depending on the task. The main ingredients are a new dataset based on readings of publicly available religious texts and effectively leveraging self-supervised learning. We built pre-trained wav2vec 2.0 models covering 1,406 languages, a single multilingual automatic speech recognition model for 1,107 languages, speech synthesis models for the same number of languages, as well as a language identification model for 4,017 languages. Experiments show that our multilingual speech recognition model more than halves the word error rate of Whisper on 54 languages of the FLEURS benchmark while being trained on a small fraction of the labeled data.

1 Introduction

Speech technology remains unavailable for most of the world’s languages, while MMS uses new labeled and unlabeled datasets plus self-supervised learning to expand coverage across major speech tasks.

  • Over 7,000 languages are spoken worldwide, but prominent recent speech technology supports only about 100 languages.
  • Self-supervised speech representations reduce the labeled data needed to build speech systems, although coverage remains limited.
  • MMS combines labeled data for 1,107 languages with unlabeled speech in 3,809 languages.
  • The project pre-trains wav2vec 2.0 models and fine-tunes systems for speech recognition, language identification, and text-to-speech.
  • MMS aims to expand speech technology to more people and contribute to preserving the world’s language diversity.

2 Related Work

Prior multilingual speech datasets and systems generally cover roughly 100 languages or fewer, whereas MMS scales speech resources and models across substantially more languages and tasks.

  • Existing multilingual speech datasets with transcriptions are generally limited to about 100 languages and may focus mainly on European languages.
  • MMS covers 58% more languages than MMS-lab’s New Testament predecessor and over five times more than its unlabeled counterpart.
  • Unlike prior work focused on speech synthesis, MMS uses higher-quality aligned data to train self-supervised, recognition, identification, and synthesis models.
  • Multilingual Automatic Speech Recognition (ASR): Whisper transcribes 99 languages from 680K hours of web-mined data, while other recent systems scaled multilingual ASR to 100 languages.
  • Multilingual Automatic Speech Recognition (ASR): MMS uses actual paired speech and text for over 1,100 languages, contrasting with phoneme mapping from an eight-language model.
  • Spoken Language Identification (LID): Prior spoken language identification systems covered 10, 26, or 107 languages, whereas MMS scales identification to over 4,000 languages.
  • Multilingual Text-To-Speech (TTS): Prior multilingual text-to-speech efforts generally cover few languages, with examples scaling to 46 or 101 languages.
  • Multilingual NLP: Multilingual NLP systems have scaled beyond speech to 200 and even 1,000 languages in machine translation.

3 Dataset Creation

The MMS dataset-creation pipeline combines labeled and unlabeled speech resources with scalable forced alignment, quality filtering, and iterative alignment improvement across thousands of languages. The resulting datasets provide broad coverage while producing higher-quality ASR data than CMU Wilderness, though MMS-lab remains narrow-domain and largely single-speaker.

  • Dataset Scope: MMS creates labeled speech-text data for 1,107 languages and unlabeled speech data for 3,809 languages, with MMS-lab-U spanning 1,362 languages.The labeled MMS-lab dataset contains 44.7K hours, while the unlabeled MMS-unlab dataset contains 7.7K hours.
  • Dataset Scope: The source data consists primarily of New Testament recordings with corresponding text, usually spoken by a single speaker.The recordings come from Faith Comes By Hearing, goto.bible, and bible.com; single-speaker recordings support speech synthesis but may introduce speaker bias.
  • Scalable Forced Alignment: GPU-based forced alignment segments long recordings into trainable units while reducing Viterbi alignment memory from O(T × L) to O(L).The implementation chunks audio for posterior computation and transfers backtracking matrices to CPU memory.
  • Improved Data Alignment: The pipeline iteratively improves alignment models by selecting high-confidence samples using the difference between constrained alignment and unconstrained greedy-decoding probabilities.Samples with scores greater than −0.2 train an improved alignment model covering 31K hours in 1,130 languages.
  • Data Filtering and Splits: After filtering 3,837 recordings with development-set CER above 5%, the final labeled data retains 1,239 recordings covering 1,107 languages.The final labeled split contains 44.7K hours: 36.8K training, 3.5K development, and 4.4K testing.
  • Dataset Evaluation: MMS-lab alignment improves CER by 2.1%-4.7% over CMU Wilderness and retains more data, including 26.2 versus 11.1 hours for Telugu.For English, MMS-lab retains 17 hours versus 10.6 hours for CMU Wilderness.

4 Cross-lingual Self-supervised Speech Representation Learning

MMS uses self-supervised wav2vec 2.0 pre-training on multilingual speech data to extend cross-lingual speech representations beyond prior coverage. The resulting models improve ASR performance over XLS-R on 61 FLEURS languages, especially for several low-resource languages.

  • Motivation and approach: MMS pre-trained wav2vec 2.0 models on 1,406 languages, over four times the coverage of known prior work.The models provide representations for downstream speech tasks and improve ASR and LID performance compared with XLS-R, which covered 128 languages.
  • Pre-training data: MMS balances languages and datasets during pre-training using sampling factors βL = βD = 0.5.The sampling strategy controls the trade-off between high- and low-resource languages and balances the contributing datasets.
  • Model training: The models use roughly 300M and 1B parameters and are trained with wav2vec 2.0 on speech audio without transcriptions.Training follows prior cross-lingual wav2vec 2.0 work and uses large-scale optimization with one million updates on A100 GPUs.
  • Pre-training data: The pre-training data covers about 491K hours across six corpora, including MMS-lab-U, MLS, CommonVoice, VoxLingua-107, BABEL, and VoxPopuli.The data combines multilingual unlabeled speech sources with different domains and language coverage.
  • Comparison to XLS-R: 0.6 lower CER for MMS at 300M parameters and 0.7 lower CER at 1B parameters than XLS-R on 61 FLEURS languages.Both models are fine-tuned on MMS-lab data and evaluated without a language model on FLEURS development data.
  • Comparison to XLS-R: MMS improves performance particularly on low-resource languages, including Amharic, Lao, Malayalam, Chewa, Fulah, and Oromo.The passage also reports small degradations for some high-resource languages, such as English and Spanish.

5 Automatic Speech Recognition

MMS scales multilingual ASR to 1,107 languages using fine-tuned wav2vec 2.0 models, with language-specific parameters preserving performance as coverage expands. It outperforms Whisper on FLEURS while using substantially less labeled data and remains competitive across multiple domains.

  • 5 Automatic Speech Recognition: Fine-tuning the MMS (1B) model with CTC and a shared character vocabulary produces multilingual ASR models, while LSAH adds language-specific adapters and output layers.LSAH uses separate adapter weights for each language and a language-specific output layer.
  • 5.2 Scaling Multilingual ASR to 1,107 Languages: 5.1 CER points are added when dense ASR scales from 61 to 1,107 languages, compared with 0.4 CER on FLEURS-61 and 0.2 CER on CV-49 for LSAH.Dense models show 5.1 and 2.1 CER increases on FLEURS-61 and CV-49, respectively.
  • 5.2 Scaling Multilingual ASR to 1,107 Languages: Scaling ASR to over 1,000 languages is feasible when language-specific parameters are used, with little performance degradation.The authors summarize the scaling experiment as showing feasibility with limited degradation under LSAH.
  • 5.3.2 Google USM: MMS performs very competitively against USM, although the systems differ substantially in architecture and their use of labeled and unlabeled data.The comparison supports competitiveness of a CTC model paired with n-gram models against more elaborate approaches.
  • 5.4 Robust Multilingual ASR Models: A multi-domain MMS model supporting 1,162 languages outperforms prior work and single-domain baselines on FLEURS and CommonVoice, but is slightly worse on VoxPopuli and MLS.Overall, the combined model is competitive across several benchmarks, while other systems can benefit from focusing on fewer languages.
  • 5.5 Evaluation on 1,107 Languages: 96% of 1,107 languages meet the CER threshold, though the metric applies one threshold across languages and many recordings use a single speaker.The threshold is CER less than five, and the single-speaker setup creates matching voices between some training and test utterances.

6 Language Identification

MMS extends language identification from roughly 100 languages to 4,017 using labeled and unlabeled multilingual speech. Its data performs competitively out of domain, and accuracy degrades only modestly as language coverage expands.

  • 6 Language Identification: LID determines the spoken language of an utterance and supports routing speech to the correct recognition system and mining web speech data.Publicly available training corpora for LID previously covered no more than about 100 languages.
  • 6.1 Methodology: MMS-lab-U and MMS-unlab enable LID for many more languages by combining speech data from labeled and unlabeled sources.The LID models fine-tune the MMS (1B) model with a linear classifier over possible languages.
  • 6.3 Scaling Language Identification to 4,017 Languages: 40x more languages are targeted than existing LID systems by scaling from about 100 languages to 4,017.The expansion combines MMS-lab-U, MMS-unlab, FLEURS, and VoxLingua-107 data.
  • 6.2 Comparison to Existing Datasets: 2.1% and 1.6% are the out-of-domain accuracy gaps for MMS-lab-U+unlab on FLEURS and VoxLingua-107, respectively.In-domain models trained on FLEURS or VoxLingua-107 perform best, while combining MMS-lab-U and MMS-unlab benefits from greater variation.
  • 6.3 Scaling Language Identification to 4,017 Languages: 0.3% accuracy loss occurs on FLEURS and no loss on VoxLingua-107 when LID scales from 126 to 4,017 languages.Out-of-domain drops are 3.6% on BABEL and 0.2% on VoxPopuli.

7 Speech Synthesis

The paper scales VITS-based speech synthesis to 1,107 languages using paired speech and text, language-specific models, and preprocessing suited to low-resource recordings. Evaluation shows generally strong content preservation and out-of-domain robustness, while reduced training choices and noisy data affect quality.

  • Model and training: VITS is scaled to 1,107 languages using separate models trained for each language.The models predict raw speech waveforms from text and are trained with a reduced 100K-step setup to make large-scale training feasible.
  • Data and representations: Character inputs replace phoneme representations for languages lacking grapheme-to-phoneme lexicons.This choice supports scaling TTS beyond one thousand languages where manually annotated pronunciation resources are unavailable.
  • Ablations: Each design choice in the reduced setup, including fewer updates, MMS-lab data, and character inputs, lowers quality relative to highly optimized VITS training.The degradation is most pronounced for out-of-domain CER and smaller on the in-domain MMS-lab development set.
  • Data preprocessing: Denoising and removing multi-speaker utterances improve performance on recordings with background music.Together, both steps reduce the CER gap to clean-recording models by 69-87% relative to unprocessed background-music data.
  • Evaluation: TTS models retain most content out-of-domain, with about 85% of 1,107 languages meeting the CER quality threshold.Synthesized speech has only slightly higher out-of-domain CER, while MOS scores are nearly identical across domains; however, reference comparisons are affected by noisy FLEURS audio.

8 Bias Analysis and Ethical Considerations

The paper examines gender differences, religiously biased language, and ethical issues arising from training on religious texts. Results indicate similar average gender performance and only slight output bias, while the automatic bias-word procedure has limited precision and religious-data use remains context-dependent.

  • Gender bias: Average CER is very similar across genders for MMS and FLEURS-trained ASR models on 27 FLEURS languages.Within individual languages, male speakers have higher error rates in 14 languages and female speakers in 13.
  • Language bias: MMS ASR outputs contain biased words at a rate 0.7% absolute higher than FLEURS outputs.The rate is much lower than in MMS-lab training data, and for many languages matches the FLEURS model rate.
  • Language bias: Native-speaker checks found that most flagged Mongolian words were general language rather than specifically biased.For Persian, two religious words were over-predicted, while the MMS model also over-predicted the word for hand relative to FLEURS.
  • Methodological caveats: The automatic procedure has low precision but captures words likely to occur more often in religious contexts.In English, it also flags common words such as “you,” “that,” and “they,” limiting the specificity of the analysis.
  • Ethical considerations: Ethical acceptability of religious training data varies across traditions, and the data may bias models toward a particular worldview.Christian ethicists did not generally regard New Testament translations as too sacred for machine learning, but the paper notes that this does not hold for all religious texts.

9 Conclusions and Open Problems

The study presents speech technology covering over one thousand languages across major tasks, enabled by self-supervised representations and expanded datasets. It identifies remaining gaps in language and dialect coverage, unified multitask models, and additional speech applications.

  • Conclusions: The study scales speech technology to over one thousand languages across automatic speech recognition, language identification, and speech synthesis.The paper reports a 10-40x increase in supported languages for several major speech tasks.
  • Open problems: More than 7,000 spoken languages and underrepresented dialects remain beyond current system coverage.The paper notes that inadequate dialect representation can produce undesirable performance biases, including for high-resource languages.
  • Open problems: Future work could train single models for multiple downstream speech tasks across many languages.The paper also proposes extending coverage to speech translation, keyword spotting, intent classification, and other tasks.

A Forced Alignment

The forced-alignment procedure selects the most probable CTC alignment path that collapses to a target label, then computes it efficiently with dynamic programming and GPU backtracking. It performs global alignment over the full sequence rather than segment-by-segment alignment.

  • CTC formulation: CTC alignment maps input audio frames to output distributions over an alphabet plus a blank token, with paths collapsing to the target label.The model assumes conditional independence among outputs given the input.
  • Forced alignment: Forced alignment chooses the highest-probability path among all CTC paths that collapse to the target label.This path is determined under the acoustic model’s posterior distribution.
  • Implementation: Equation 2 is computed with dynamic programming and backtracking, using a CUDA implementation with CPU memory offloading for long audio.The GPU algorithm outputs an alignment path from posterior probabilities and a target label.
  • Comparison: Unlike segment-by-segment alternatives, the forced-alignment process performs a global alignment over the entire audio sequence.The resulting alignments support segmentation into verse-level audio segments.

B n-gram Language Models

The paper uses word- and character-level n-gram language models with different data sources and decoding configurations. These models support CTC beam-search decoding and are tuned on validation data.

  • Language-model construction: 5-gram language models are trained on Common Crawl data for each FLEURS language.Seven languages without space-separated words use 20-gram character-level models instead.
  • Language-model construction: Word-level models use up to 40GB of data and a 250K-word vocabulary, while character-level models use up to 6GB.
  • Language-model construction: 15-gram character-level models without pruning are trained on FLEURS transcriptions for all FLEURS languages.These models are used in comparisons involving FLEURS training-data transcriptions.
  • Comparison setup: MMS uses Common Crawl language models for comparison with Whisper.
  • Decoding: CTC beam-search decoding uses lexicon-based decoding for word-level models and lexicon-free decoding for character-level models.Language-model weight and word insertion penalty are tuned on the validation set.
  • Implementation details: Smaller datasets use reduced pruning settings, with discount fallback additionally enabled below 1GB.

C Comparison to Whisper

The appendix compares MMS with Whisper on the FLEURS test set, reporting language-level and aggregate error rates across the shared 54-language subset. The comparison uses normalization and language-model settings intended to make the systems more comparable.

  • Results organization: Tables A1 and A2 provide language-level and aggregate breakdowns of the comparison results.
  • Evaluation setup: 54 shared FLEURS languages are compared between MMS and Whisper using language-level error rates.Thai, Lao, Burmese, and Khmer use CER; the other languages use WER.
  • Evaluation setup: Table A1 reports WER or CER for each of the 54 languages supported by both MMS and Whisper.Whisper normalization is applied to both references and hypotheses.
  • Aggregate comparison: Average WER is reported on FLEURS-54 for MMS and Whisper.The comparison uses n-gram web-data language models with MMS because Whisper uses a neural sequence-model decoder.
  • Aggregate comparison: MMS is evaluated as a CTC-based model against Whisper with n-gram language models used to support a fairer decoder comparison.

D Comparison to USM

The supplied passages describe MMS results on all 102 FLEURS languages and a separate gender-bias analysis across 27 languages. They specify language-specific model components and per-gender CER reporting.

  • FLEURS-102 results: 102 FLEURS languages are evaluated using CER on development and test sets after fine-tuning MMS on labeled FLEURS data.
  • FLEURS-102 results: Language-specific adapters and heads are used for the FLEURS-102 evaluation.
  • FLEURS-102 results: Inference selects between Common Crawl word-based and FLEURS-transcription character-based language models using development-set accuracy.
  • Gender-bias analysis: Gender-bias analysis compares MMS-lab-trained and FLEURS-trained ASR models across 27 FLEURS languages.
  • Gender-bias analysis: The gender analysis reports development CER separately by speaker gender for languages with at least 50 samples per gender.
Loading 2305.13516v1…