Unsupervised Cross-lingual Representation Learning at Scale

Alexis Conneau; Kartikay Khandelwal; Naman Goyal; Vishrav Chaudhary; Guillaume Wenzek; Francisco Guzmán; Edouard Grave; Myle Ott; Luke Zettlemoyer; Veselin Stoyanov

Source-linked AI summary

Unsupervised Cross-lingual Representation Learning at Scale

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov

arXiv:1911.02116v2Nov 5, 2019cs.CL

TL;DR

Cross-lingual language understanding requires stronger multilingual representations and careful study of large-scale unsupervised pretraining. The paper introduces XLM-R, a multilingual masked language model trained on 100 languages, which substantially outperforms prior multilingual models across cross-lingual benchmarks.

Problem
The paper studies how large-scale unsupervised multilingual pretraining can improve cross-lingual language understanding across diverse tasks.
Method
XLM-R is a Transformer-based multilingual masked language model trained with multilingual masked-language modeling on monolingual text from 100 languages.
Results
XLM-R sets state-of-the-art results across cross-lingual classification, sequence labeling, and question answering, outperforming mBERT by 14.6% average accuracy on XNLI.
Takeaways & Limitations
A single multilingual model can remain competitive with monolingual models while delivering strong gains, particularly for low-resource languages.
Takeaways & Limitations
Monolingual Wikipedia corpora are too small for unsupervised representation learning, with a few hundred MiB usually needed to train a BERT model.

Abstract

from arXiv · show

This paper shows that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks. We train a Transformer-based masked language model on one hundred languages, using more than two terabytes of filtered CommonCrawl data. Our model, dubbed XLM-R, significantly outperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks, including +14.6% average accuracy on XNLI, +13% average F1 score on MLQA, and +2.4% F1 score on NER. XLM-R performs particularly well on low-resource languages, improving 15.7% in XNLI accuracy for Swahili and 11.4% for Urdu over previous XLM models. We also present a detailed empirical analysis of the key factors that are required to achieve these gains, including the trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource languages at scale. Finally, we show, for the first time, the possibility of multilingual modeling without sacrificing per-language performance; XLM-R is very competitive with strong monolingual models on the GLUE and XNLI benchmarks. We will make our code, data and models publicly available.

1 Introduction

The paper introduces XLM-R, a Transformer-based multilingual masked language model pretrained on 100 languages to improve cross-lingual understanding. It analyzes multilingual scaling trade-offs and reports strong cross-lingual and monolingual benchmark performance without sacrificing per-language performance.

100-language XLM-R is a Transformer-based multilingual masked language model that achieves state-of-the-art results on cross-lingual classification, sequence labeling, and question answering.
Scaling the number of languages improves low-resource cross-lingual performance until a point, after which monolingual and cross-lingual benchmark performance degrades under fixed model capacity.The paper names this trade-off the curse of multilinguality and studies language sampling and vocabulary size.
23% accuracy is the maximum improvement XLM-R achieves over mBERT on cross-lingual classification for low-resource languages.
5.1% average accuracy on XNLI, 2.42% average F1-score on Named Entity Recognition, and 9.1% average F1-score on cross-lingual Question Answering exceed the previous state of the art.
XLM-R is competitive with state-of-the-art monolingual models on GLUE and XNLI, demonstrating a single model can serve all languages without sacrificing per-language performance.RoBERTa is given as an example of a strong monolingual model.

2 Related Work

Prior work advanced unsupervised and cross-lingual representation learning through pretrained contextual and multilingual language models, including mBERT and XLM. Related research also established the value of scaling and identified multilingual high–low-resource trade-offs and capacity dilution.

Unsupervised representation learning progressed from pretrained word embeddings and contextualized representations to Transformer-based language models, improving natural language understanding.
mBERT and XLM introduced multilingual masked language models trained without cross-lingual supervision, while TLM used parallel data to advance XNLI performance.
These multilingual language-modeling efforts operated at a relatively modest training-data scale compared with the approach described here.
Scaling model size and training data has been extensively studied, with large-scale LSTM, GPT, and RoBERTa work highlighting its importance for stronger language-modeling performance.
Massively multilingual machine-translation studies exposed high- and low-resource trade-offs and capacity dilution, including a 103-language model trained on over 25 billion parallel sentences.

3 Model and Data

XLM-R follows the multilingual masked-language-modeling approach while scaling it to 100 languages and substantially larger, filtered CommonCrawl data. Its design uses monolingual text, subword tokenization, language-balanced sampling, and a large vocabulary, with data scale motivated by the insufficiency of Wikipedia corpora.

Masked Language Models: XLM-R trains a Transformer with a multilingual masked language-modeling objective on monolingual data, predicting masked input tokens from sampled language streams.Subword tokenization is applied directly to raw text using SentencePiece with a unigram language model.
Masked Language Models: The model samples batches across languages with α = 0.3, omits language embeddings to better handle code-switching, and uses a 250K vocabulary with a full softmax.XLM-R is trained in Base and another model configuration, though the supplied passage truncates the latter specifications.
Scaling to a hundred languages: XLM-R is trained on 100 languages, including a benchmark set spanning English, French, German, Russian, Chinese, Swahili, and Urdu.The benchmark set covers multiple language families and includes low-resource languages; high-resource and low-resource results average English/French and Swahili/Urdu, respectively.
Scaling the Amount of Training Data: The training corpus is a clean CommonCrawl collection in 100 languages, filtered with language-identification models and language-specific language models.One dump is used for English and twelve dumps for other languages, increasing dataset sizes especially for Burmese and Swahili.
Scaling the Amount of Training Data: A few hundred MiB of text is usually the minimum needed to learn a BERT model, while monolingual Wikipedia corpora are too small for unsupervised representation learning.CommonCrawl increases data by several orders of magnitude, particularly for low-resource languages.

4 Evaluation

The evaluation covers cross-lingual natural language inference, named entity recognition, question answering, and English GLUE performance. It assesses cross-lingual transfer, per-language and multilingual learning, and comparisons with translation, task-specific, and state-of-the-art baselines.

Cross-lingual Natural Language Inference: XNLI evaluates cross-lingual transfer from English across 15 languages, using an English training set and machine-translated training data for the other 14.The evaluation includes machine-translation baselines for translate-test, translate-train per language, and translate-train-all across languages.
Named Entity Recognition: NER uses CoNLL-2002 and CoNLL-2003 in English, Dutch, Spanish, and German, reporting F1 for cross-lingual transfer, per-language performance, and multilingual learning.Models are fine-tuned on English, each language set, or all sets, with comparisons to Lample et al. (2016) and Akbik et al. (2018).
Cross-lingual Question Answering: MLQA extends English SQuAD to Spanish, German, Arabic, Hindi, Vietnamese, and Chinese, measuring cross-lingual transfer with F1 and exact match scores.The benchmark evaluates transfer from English.
GLUE Benchmark: GLUE evaluates English performance across classification tasks including MNLI, SST-2, and QNLI, using BERTLarge and RoBERTa as baselines.GLUE is used to compare XLM-R with other state-of-the-art models.

5 Analysis and Results

The analysis identifies scaling trade-offs in multilingual pretraining: adding languages initially enables positive transfer but can cause capacity dilution, while increased model and vocabulary capacity and larger CommonCrawl data mitigate these effects. XLM-R achieves strong cross-lingual transfer and downstream performance, including state-of-the-art XNLI and MLQA results, while maintaining competitive per-language performance.

Transfer-dilution Trade-off and Curse of Multilinguality: 71.8% to 67.7%: overall XNLI accuracy decreases when scaling from XLM-7 to XLM-100, after positive transfer gives way to multilingual interference.The initial increase from 7 to 15 languages particularly improves low-resource languages, but further scaling degrades performance across languages.
Transfer-dilution Trade-off and Curse of Multilinguality: Added capacity brings XLM-30 on par with XLM-7, but remains insufficient for XLM-100, which still lags because of higher vocabulary dilution.The models use hidden sizes of 768, 960, and 1152 for 7, 30, and 100 languages, respectively.
Importance of Capacity and Vocabulary: 2.8%: XNLI average accuracy increases as vocabulary size grows from 32K to 256K at fixed model capacity.The experiment adjusts Transformer width to keep the total number of parameters constant.
Larger-scale Datasets and Training: CommonCrawl-trained models significantly outperform models with the same BERTBase architecture trained on Wikipedia.The collected CommonCrawl corpus contains substantially more monolingual data than the previously used Wikipedia corpora.
Cross-lingual Understanding: 80.9% accuracy: XLM-R outperforms XLM-100 and mBERT on XNLI by 10.2% and 14.6% average accuracy, respectively.On Swahili and Urdu, XLM-R exceeds XLM-100 by 15.7% and 11.4%, and mBERT by 23.5% and 15.8%.
Question Answering: 70.7% F1 and 52.7% accuracy: XLM-R surpasses the previous MLQA state of the art, 61.6% and 43.5%, and exceeds mBERT by 13.0% F1-score and 11.1% accuracy.MLQA evaluation trains on English data and evaluates across the benchmark’s seven languages.

6 Conclusion

The paper introduces XLM-R, a state-of-the-art multilingual masked language model trained on 2.5 TB of clean CommonCrawl data across 100 languages. It reports gains over mBERT and XLM, identifies key limitations and trade-offs in multilingual MLMs, and demonstrates strong low-resource and multilingual-model performance.

6 Conclusion: XLM-R is a state-of-the-art multilingual masked language model trained on 2.5 TB of clean CommonCrawl data in 100 languages.The model is the paper’s central contribution.
6 Conclusion: XLM-R provides strong gains over mBERT and XLM on classification, sequence labeling, and question answering.These gains span multiple multilingual language-understanding task types.
6 Conclusion: The analysis exposes the high-resource versus low-resource trade-off, the curse of multilinguality, and the importance of key hyperparameters.These findings characterize important limitations of multilingual masked language models.
6 Conclusion: Multilingual models are surprisingly effective compared with monolingual models and show strong improvements on low-resource languages.The conclusion highlights multilingual modeling as effective without limiting the claim to a specific benchmark.

Appendix · A Languages and statistics for CC-100 used by XLM-R

This appendix documents the 100-language CC-100 corpus created for XLM-R, reporting per-language token counts and data sizes. It also notes the inclusion of romanized variants for selected non-Latin languages.

A Languages and statistics for CC-100 used by XLM-R: The appendix presents the languages included in the CC-100 corpus created for training XLM-R.
A Languages and statistics for CC-100 used by XLM-R: The appendix reports the number of tokens for each language’s monolingual corpus.
A Languages and statistics for CC-100 used by XLM-R: The appendix reports the size of each language’s data in GiB.
A Languages and statistics for CC-100 used by XLM-R: CC-100 contains a list of 100 languages.
A Languages and statistics for CC-100 used by XLM-R: Token counts are reported in millions.
A Languages and statistics for CC-100 used by XLM-R: CC-100 includes romanized variants of Bengali, Hindi, Tamil, Telugu, and Urdu.

B Model Architectures and Sizes

This section compares the architectures and parameter sizes of monolingual and multilingual Transformer models. XLM-R is among the largest listed models, but remains smaller than newer multilingual machine-translation and transfer-learning Transformers.

Model architectures and sizes: XLM-R is among the largest models partly because of its large embedding layer.Its parameter count is similar to XLM-100.
Model architectures and sizes: XLM-R remains significantly smaller than recently introduced Transformer models for multilingual machine translation and transfer learning.