Source-linked AI summary
Larger-Scale Transformers for Multilingual Masked Language Modeling
Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, Alexis Conneau
TL;DR
The paper asks whether increasing multilingual masked language model capacity can improve cross-lingual understanding while retaining strong high-resource language performance. It trains and evaluates 3.5B- and 10.7B-parameter XLM-R models, finding stronger cross-lingual results and a modest advantage over RoBERTa-Large on English GLUE. The comparison with mT5 is limited by major differences in data, training, context length, and model size.
Problem
Prior multilingual models improved low-resource transfer but could underperform on high-resource languages, motivating study of whether greater capacity can improve both.
Method
The paper scales XLM-R into 3.5B- and 10.7B-parameter multilingual masked language models trained on monolingual data with the XLM-R procedure.
Results
XLM-RXXL improves cross-lingual benchmark performance and reaches 93.2 average accuracy on GLUE versus 92.9 for RoBERTa-Large while handling 99 more languages.
Takeaways & Limitations
Larger multilingual models can achieve strong high-resource performance while maintaining cross-lingual transfer and low-resource language understanding.
Takeaways & Limitations
Comparison with mT5 is not apple-to-apple because the models differ in dataset size, token exposure, context length, batch size, and parameter count.
Abstract
from arXiv · showhide
Recent work has demonstrated the effectiveness of cross-lingual language model pretraining for cross-lingual understanding. In this study, we present the results of two larger multilingual masked language models, with 3.5B and 10.7B parameters. Our two new models dubbed XLM-R XL and XLM-R XXL outperform XLM-R by 1.8% and 2.4% average accuracy on XNLI. Our model also outperforms the RoBERTa-Large model on several English tasks of the GLUE benchmark by 0.3% on average while handling 99 more languages. This suggests pretrained models with larger capacity may obtain both strong performance on high-resource languages while greatly improving low-resource languages. We make our code and models publicly available.
1 Introduction
The paper studies whether substantially increasing multilingual masked language model capacity improves cross-lingual understanding without sacrificing performance on high-resource languages. It introduces XLM-RXL and XLM-RXXL, which show stronger cross-lingual and English benchmark performance than earlier models.
- The study examines how larger-capacity models affect cross-lingual language understanding.
- XLM-RXL and XLM-RXXL scale XLM-R to 3.5B and 10.7B parameters using the same CC100 dataset.
- Multilingual pretraining improves cross-lingual transfer, especially for low-resource languages, by using data from multiple languages.
- Larger multilingual models are motivated by the trade-off between low-resource gains and lower high-resource performance.
- XLM-RXXL reaches new state-of-the-art results on some cross-lingual benchmarks and outperforms RoBERTa-Large on English GLUE tasks.
2 Pretraining and evaluation
The models use multilingual masked language modeling on monolingual streams and are evaluated across cross-lingual inference, question answering, and English classification tasks. Training uses the XLM-R procedure with scaled model architectures and tensor parallelism.
- Pretraining: The models train a Transformer with a multilingual masked language modeling objective on monolingual data from multiple languages.
- Pretraining: XLM-RXL has 3.5B parameters, while XLM-RXXL uses a larger architecture with 10.7B parameters.
- Evaluation: Evaluation covers cross-lingual natural language inference, cross-lingual question answering, and the English GLUE benchmark.
- Evaluation: XNLI measures English-to-other-language transfer across 15 languages, with both zero-shot and machine-translated training comparisons.
- Evaluation: MLQA and XQuad report F1 and exact-match scores for cross-lingual transfer from English.
- Training setup: Scaling uses tensor model parallelism, batch size 2048, and 500,000 training updates for both models.
3 Analysis and Results
Scaling XLM-R improves cross-lingual understanding across XNLI, MLQA, and XQuad while restoring competitive English performance against RoBERTa-Large. Comparisons with mT5 are informative but limited by substantial differences in data, training, and model configurations.
- Cross-lingual understanding results: XNLI accuracy improves as XLM-R capacity increases from 550M to 3.5B and 10.7B parameters, with gains under zero-shot and multilingual transfer.Scaling from XLM-RLarge to XLM-RXL yields 1.4 average accuracy points on zero-shot transfer and 1.8 on multilingual fine-tuning; further scaling yields 2.2 zero-shot and 2.4 translate-train-all points.
- Cross-lingual understanding results: 4.1 F1 and 3.9 EM points are gained on MLQA when scaling from XLM-RLarge to XLM-RXXL for cross-lingual zero-shot transfer.XQuad shows corresponding gains of 4.4 F1 and 5.5 scores, with new state-of-the-art results on several languages.
- Comparison to monolingual English model: 0.3 average GLUE accuracy points separate XLM-RXXL from RoBERTaLarge, despite XLM-RXXL handling 99 more languages.The comparison rises from 92.9 to 93.2 average accuracy, reversing the earlier disadvantage observed for smaller XLM-R models.
- Discussion and comparison to mT5: mT5 and XLM-R achieve strong cross-lingual and English performance, but direct comparison is difficult because their pretraining data, token counts, batch sizes, and context lengths differ.mT5 uses the 6.4T-token mC4 dataset, compared with CC100's 167B tokens, while the models also differ in updates, batch size, context length, and parameter count.
- Discussion and comparison to mT5: XLM-RXL exceeds mT5XL on XNLI, XQuad, and MLQA averages, whereas mT5XXL leads XLM-RXXL on Swahili XNLI zero-shot.The reported margins are 0.6% on XNLI, 1.3% on XQuad, and 0.9% on MLQA average EM for XLM-RXL over mT5XL; mT5XXL leads by 8.4 points on Swahili.
4 Conclusion
The study scales XLM-R to 10.7B parameters and reports stronger cross-lingual understanding than previous XLM-R models. The enlarged multilingual model also outperforms RoBERTa-Large on English benchmarks while maintaining strong high-resource performance.
- Conclusion: Scaling XLM-R to 10.7B parameters produces stronger cross-lingual understanding than previous XLM-R models.The study reports new state-of-the-art performance on some languages and publicly releases code and models.
- Conclusion: Additional capacity enables XLM-RXXL to outperform the RoBERTaLarge baseline on English benchmarks.The conclusion presents larger multilingual models as able to combine state-of-the-art cross-lingual results with strong high-resource-language performance.