Source-linked AI summary
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale
Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Conneau, Michael Auli
TL;DR
Cross-lingual speech representation learning lacked the scale and language coverage achieved in multilingual NLP. XLS-R scales wav2vec 2.0 pretraining to 436K hours across 128 languages and evaluates it broadly. It reaches strong or state-of-the-art results across translation, recognition, and language identification, including English-to-other-language translation where monolingual pretraining is favored.
Problem
Cross-lingually pretrained speech models remained much smaller than large-scale NLP and monolingual speech models, limiting the scale of multilingual speech representation learning.
Method
XLS-R scales cross-lingual wav2vec 2.0 pretraining using public speech corpora across 128 languages and fine-tunes the resulting models for multilingual speech tasks.
Results
XLS-R achieves state-of-the-art or competitive results across speech translation, recognition, and language identification, including outperforming English-only pretraining for English →X translation with sufficient capacity.
Takeaways & Limitations
Cross-lingually trained models with sufficient capacity can match or outperform specialized monolingual models and support speech technology across many languages, tasks, and data regimes.
Takeaways & Limitations
MLS is a notable exception, attributed to prior work having pretrained on large amounts of in-domain data.
Abstract
from arXiv · showhide
This paper presents XLS-R, a large-scale model for cross-lingual speech representation learning based on wav2vec 2.0. We train models with up to 2B parameters on nearly half a million hours of publicly available speech audio in 128 languages, an order of magnitude more public data than the largest known prior work. Our evaluation covers a wide range of tasks, domains, data regimes and languages, both high and low-resource. On the CoVoST-2 speech translation benchmark, we improve the previous state of the art by an average of 7.4 BLEU over 21 translation directions into English. For speech recognition, XLS-R improves over the best known prior work on BABEL, MLS, CommonVoice as well as VoxPopuli, lowering error rates by 14-34% relative on average. XLS-R also sets a new state of the art on VoxLingua107 language identification. Moreover, we show that with sufficient model size, cross-lingual pretraining can outperform English-only pretraining when translating English speech into other languages, a setting which favors monolingual pretraining. We hope XLS-R can help to improve speech processing tasks for many more languages of the world.
1 INTRODUCTION
Self-supervised multilingual models improve access to language technology, but cross-lingually pretrained speech models remained much smaller than their NLP counterparts. XLS-R addresses this gap by scaling cross-lingual wav2vec 2.0 pretraining to publicly available speech across 128 languages.
- Multilingual self-supervised models can improve low-resource language performance by leveraging data from high-resource languages while sharing one model.
- Cross-lingual speech pretraining extends wav2vec 2.0 and uses transfer from high-resource languages to improve representations for languages with little unlabeled data.
- XLS-R pretrains a large multilingual wav2vec 2.0 Transformer on 436K hours of unannotated speech in 128 languages from public speech corpora.
- Speech scaling had focused on supervised multilingual or monolingual self-supervised models, while cross-lingually pretrained speech models remained much smaller.
2 BACKGROUND
XLS-R builds cross-lingual speech representations with wav2vec 2.0 by encoding audio, quantizing training targets, masking inputs, and training across multiple languages. Language sampling uses an upsampling factor to balance high- and low-resource data.
- wav2vec 2.0 maps raw audio to latent representations with a convolutional encoder, then uses a Transformer to produce contextual representations.
- Training discretizes feature-encoder representations into quantized targets using a Gumbel-softmax codebook module.
- The model solves a masked contrastive task by identifying each true quantized latent among K = 100 distractors.
- A codebook diversity penalty encourages the model to use all codebook entries.
- Multilingual batches produce cross-lingual representations, with α controlling the trade-off between high- and low-resource languages during pretraining.
3 DATA AND EVALUATION
XLS-R is pretrained on a large, diverse public corpus spanning 436K hours and 128 languages, then evaluated across speech translation, recognition, language identification, and speaker identification settings.
- 3.1 TRAINING DATA: The pretraining corpus combines VoxPopuli, MLS, CommonVoice, VoxLingua107, and BABEL for 436K hours of publicly available speech.
- 3.1 TRAINING DATA: The 128-language dataset contains 24 high-resource languages, 17 mid-resource languages, and 88 low-resource languages defined by progressively smaller data ranges.
- 3.2 DOWNSTREAM EVALUATION: Evaluation covers diverse downstream tasks, data regimes, domains, and languages to assess pretrained-model generalization.
- 3.2.1 SPEECH TRANSLATION: CoVoST-2 evaluates speech translation from English into 15 languages and from 21 languages into English.
- 3.2.2 AUTOMATIC SPEECH RECOGNITION (ASR): BABEL ASR uses noisy conversational telephone speech in five languages with 30–76 hours of annotated training data per language.
- 3.2.2 AUTOMATIC SPEECH RECOGNITION (ASR): Recognition is also evaluated on MLS, CommonVoice, VoxPopuli, and LibriSpeech under labeled-data regimes ranging from minutes to hundreds of hours.
- 3.2.3 SPEECH CLASSIFICATION (LID AND SPEAKER ID): VoxLingua107 and VoxCeleb1 provide language-identification and speaker-identification evaluations using short speech clips from YouTube-derived or interview-video datasets.
4 EXPERIMENTAL SETUP
The experiments use wav2vec 2.0-based XLS-R models spanning 0.3B to 2B parameters, with multilingual pretraining and task-specific fine-tuning across speech translation and recognition.
- Pretraining and architectures: XLS-R architectures range from 0.3B to 2B parameters and are compared with prior model architectures.The architecture table covers languages, pretraining data, blocks, hidden states, feed-forward dimensions, attention heads, and parameter counts.
- Pretraining and architectures: Training data from 128 languages is balanced by upsampling languages within each corpus and then balancing the corpora with α = 0.5.The two-stage balancing treats each corpus as a language during corpus-level resampling.
- Speech recognition: Speech recognition fine-tuning adds a linear vocabulary-prediction layer and trains with Connectionist Temporal Classification.Characters are used for all recognition benchmarks except CommonVoice, which uses phonemes.
- Evaluation: The evaluation reports average BLEU for 21 X →English CoVoST-2 directions grouped by high-, mid-, and low-resource labeled-data conditions.The broader setup also specifies decoding choices for MLS and BABEL versus CommonVoice and VoxPopuli.
5 RESULTS
The paper next analyzes the pretrained models on downstream tasks.
- The pretrained models are evaluated on downstream tasks.
- The results analysis follows the pretrained-model experiments.
- The next section examines model performance beyond pretraining.
5.1 SPEECH TRANSLATION
XLS-R improves speech translation across resource conditions, with larger models producing stronger CoVoST-2 results and multilingual methods especially helping X →English directions.
- X →English: 7.4 BLEU separates XLS-R (2B) from the previous best result across 21 X →English directions, improving from 14.7 BLEU to 22.1 BLEU.The largest gains occur in mid-resource (+7.5 BLEU) and low-resource (+9.2 BLEU) directions.
- X →English: 6.1 BLEU and 2.8 BLEU are the average gains from scaling XLS-R from 0.3B to 1B and from 1B to 2B parameters, respectively.
- X →English: With two hours of labeled data, XLS-R (2B) reaches 29.6 BLEU for Swedish-English, 16.5 for Indonesian-English, and 17.1 for Arabic-English.The corresponding XLS-R (0.3B) scores are 10.3, 1.4, and 3.0 BLEU.
- English →X: 27.8 average BLEU for XLS-R (2B) exceeds 26.6 BLEU for a 60K-hour English-pretrained wav2vec 2.0 model on English →X translation.The result supports cross-lingual pretraining matching or exceeding English-only pretraining with sufficient capacity.
- Ablations: 3.3 BLEU is the average improvement from multilingual over bilingual fine-tuning on X →English directions, increasing from 20.9 BLEU to 24.2 BLEU.The labeled-data range spans 264 hours for French →English to 10 hours for Chinese →English.
5.2 SPEECH RECOGNITION
Across noisy, read-speech, parliamentary, and multilingual recognition benchmarks, XLS-R benefits from broader pretraining data and increased capacity, especially in low-data settings.
- BABEL: 1.4 WER is the average improvement of XLS-R (0.3B) over equally sized XLSR-53 across BABEL languages.WER decreases from 44.1 to 42.9 on Assamese, 26.5 to 24.3 on Swahili, and 31.1 to 28.0 on Georgian.
- BABEL: 2.5 WER is the average gain from increasing BABEL model capacity from XLS-R (0.3B) to XLS-R (1B).On Tagalog and Lao, XLS-R (1B) reduces WER by more than 10 compared with results from three years earlier.
- CommonVoice: 1.1 PER is the average improvement of XLS-R (0.3B) over XLSR-53 on CommonVoice with one hour of labeled data per language.
- CommonVoice: Kyrgyz PER falls from 6.1 to 5.1 with XLS-R (0.3B) and to 4.1 with XLS-R (1B), despite both models using only about 11 hours of Kyrgyz pretraining data.The result is presented as evidence of cross-lingual transfer and the importance of model capacity.
- Multilingual LibriSpeech: 1 WER and 2.9 WER are the average gains over XLSR-53 at equal capacity and from scaling XLS-R to 1B on Multilingual LibriSpeech, respectively.XLS-R rivals supervised models trained with substantially more labeled data in the ten-hour setup.
- LibriSpeech: XLS-R (0.3B) trails the English wav2vec 2.0 baseline on LibriSpeech, but XLS-R (1B) surpasses it in the ten-minute setting and remains 0.7 WER worse in the ten-hour setting.The comparison illustrates a capacity trade-off between multilingual transfer and English-focused performance.
5.3 SPEECH CLASSIFICATION
XLS-R is evaluated on language and speaker identification, including VoxLingua107’s 33-language development set and VoxCeleb1. It outperforms prior work on language identification and performs strongly for speaker identification.
- TASK SETUP: The smallest XLS-R model is used for utterance classification because these tasks require less capacity than speech recognition and speech translation.The evaluated classification tasks are language identification and speaker identification.
- LANGUAGE IDENTIFICATION: XLS-R improves the best known prior work on VoxLingua107 language identification by 1% absolute, reducing error by 15% relatively.The comparison uses the development set spanning 33 languages.
- LANGUAGE IDENTIFICATION: The English-only wav2vec 2.0 baseline is outperformed by XLS-R by 1.5% error rate on average.This shows that cross-lingual pretraining remains advantageous on this multilingual classification task.
- SPEAKER IDENTIFICATION: XLS-R performs very well for speaker identification on VoxCeleb1, despite the utterances being mostly in English.All XLS-R parameters are fine-tuned, whereas some comparison results are not strictly comparable because their pretrained models were not fine-tuned.
5.4 DISCUSSION
XLS-R’s cross-lingual pretraining provides a single multilingual model that performs strongly across resource levels and tasks. Larger models can match or surpass English-only pretraining, including for English-to-other-language speech translation.
- Single model: A single cross-lingual model can serve multiple languages instead of requiring separate monolingual models.Although training requires more effort than training one monolingual model, later architectural and training advances need to be deployed only once.
- Performance trends: XLS-R performs best for low-resource and mid-resource languages, with especially strong gains in low- and mid-resource X →English speech translation.Many directions that previously scored 1-5 BLEU improve to over 10-20 BLEU.
- Performance trends: Large enough cross-lingual models can surpass English-only pretrained models on English →X speech translation, despite that setting favoring monolingual pretraining.This extends the competitiveness of cross-lingual pretraining beyond settings where multilingual transfer is most obviously beneficial.
- Performance trends: Models trained on more data from more languages can perform as well as or better than same-sized comparable models across speech recognition benchmarks.The discussion reports this trend across BABEL, CommonVoice, MLS, and VoxPopuli, with larger capacity often improving performance further.
6 CONCLUSION
XLS-R scales cross-lingual speech representation learning in model size, training data, and language coverage, achieving strong results across translation, recognition, and language identification. Its main exception is MLS, where prior in-domain pretraining explains the different performance pattern.
- CONCLUSION: XLS-R scales to 436K hours of speech audio in 128 languages, with a training corpus an order of magnitude larger than prior work.The largest model contains 2B parameters.
- CONCLUSION: XLS-R achieves state-of-the-art X →English speech translation on CoVoST-2, with the largest improvements on mid- and low-resource directions.It also performs competitively on English →X without self-training or language model decoding.
- CONCLUSION: XLS-R sets new speech-recognition state of the art on CommonVoice, VoxPopuli, and several BABEL languages, while remaining competitive on MLS with much less labeled data.These datasets span languages, data regimes, and domains, supporting broad generalization claims within the evaluated scope.
- CONCLUSION: MLS is a notable exception because prior work pretrained on large amounts of in-domain data.The authors attribute MLS’s different performance pattern to this prior-work advantage.
- CONCLUSION: XLS-R also sets a new state of the art on VoxLingua107 language identification.The 2B-parameter model can outperform a strong English-only pretrained model on English →X speech translation.
A DETAILED SPEECH TRANSLATION RESULTS
The detailed speech-translation results report full CoVoST-2 comparisons for both X →English and English →X directions. Baselines include prior XLSR, VoxPopuli, XMEF, and wav2vec 2.0 systems.
- X →English: Table 14 reports full CoVoST-2 X →English results against XLSR-53, VP-100K, XMEF-En, and XMEF-X baselines.The listed baselines cover multilingual, VoxPopuli-based, English-pretrained, and prior literature systems.