Source-linked AI summary
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, Melvin Johnson
TL;DR
Broad multilingual NLP benchmarks were missing despite the need to evaluate models across diverse languages and tasks. The paper introduces XTREME, a 40-language, 9-task benchmark for zero-shot cross-lingual transfer, and finds substantial performance gaps outside English, especially on syntactic and sentence retrieval tasks.
Problem
Broad-coverage NLP benchmarks were mostly limited to English, leaving comprehensive evaluation of multilingual models across diverse languages and tasks missing.
Method
XTREME evaluates zero-shot cross-lingual transfer using English training data across 40 languages and 9 tasks spanning syntax and semantics.
Results
Models approach human performance in English on many tasks, but cross-lingually transferred performance drops significantly, with the largest differences in syntactic and sentence retrieval tasks.
Takeaways & Limitations
XTREME provides broader coverage and more fine-grained analysis tools to encourage research on cross-lingual generalization.
Takeaways & Limitations
XTREME’s language coverage does not capture all aspects of each language necessary for transfer, so it offers only a glimpse of true cross-lingual generalization.
Abstract
from arXiv · showhide
Much recent progress in applications of machine learning models to NLP has been driven by benchmarks that evaluate models across a wide variety of tasks. However, these broad-coverage benchmarks have been mostly limited to English, and despite an increasing interest in multilingual models, a benchmark that enables the comprehensive evaluation of such methods on a diverse range of languages and tasks is still missing. To this end, we introduce the Cross-lingual TRansfer Evaluation of Multilingual Encoders XTREME benchmark, a multi-task benchmark for evaluating the cross-lingual generalization capabilities of multilingual representations across 40 languages and 9 tasks. We demonstrate that while models tested on English reach human performance on many tasks, there is still a sizable gap in the performance of cross-lingually transferred models, particularly on syntactic and sentence retrieval tasks. There is also a wide spread of results across languages. We release the benchmark to encourage research on cross-lingual learning methods that transfer linguistic knowledge across a diverse and representative set of languages and tasks.
1. Introduction
XTREME addresses the lack of broad multilingual benchmarks by evaluating cross-lingual transfer across diverse languages and tasks. Its results show that English performance does not transfer uniformly, with the largest gaps in syntactic and sentence retrieval tasks and across several language families.
- XTREME evaluates multilingual representations across 40 typologically diverse languages, spanning 12 language families and 9 tasks involving syntax and semantics.
- Pseudo test sets extend diagnostics for natural language inference and question answering to all 40 languages by automatically translating English test sets.
- The benchmark focuses on zero-shot transfer, training with annotated English data and transferring to languages without annotated training data.
- State-of-the-art models approach human performance in English on many tasks, but transferred performance drops significantly, especially for syntactic and sentence retrieval tasks.
- Performance is generally stronger for Indo-European languages and lower particularly for Sino-Tibetan, Japonic, Koreanic, and Niger-Congo languages.
- The benchmark releases 9 tasks, an evaluation platform and leaderboard, strong baselines, code, and analyses of cross-lingual model limitations.
2. Related Work
Prior cross-lingual evaluation has examined representations through translation and many individual NLP tasks, but coverage has often been narrow or affected by dataset limitations. These issues motivate broader, multi-task evaluation across diverse languages.
- Cross-lingual representations have been learned using parallel corpora, bilingual dictionaries, self-training, and unsupervised strategies.
- Cross-lingual evaluation has commonly used word-level bilingual lexicon induction or sentence-level machine translation, often on typologically related languages and similar domains.
- Prior work evaluates cross-lingual approaches on tasks including parsing, named entity recognition, sentiment analysis, inference, document classification, and question answering.
- Single-task evaluation can be problematic because standard datasets may permit shortcuts, such as keyword matching in MLDoc.
3. XTREME
XTREME selects accessible, challenging, and diverse tasks and languages for zero-shot cross-lingual evaluation. Its coverage spans multiple task types and languages, while diagnostics and author analyses clarify both the benchmark’s breadth and its limits.
- 3.1. Design principles: XTREME selects tasks for difficulty, task diversity, training efficiency, multilinguality, sufficient monolingual data, and accessibility.The design targets tasks that remain below human cross-language performance and can be trained on a single GPU in less than a day.
- 3.2. Tasks: The benchmark contains nine tasks spanning four categories that require reasoning over different levels of meaning.
- 3.2. Tasks: Tasks include inference, paraphrase identification, POS tagging, named entity recognition, question answering, and sentence retrieval.
- 3.2. Tasks: TyDiQA-GoldP covers nine languages and is more challenging than XQuAD and MLQA because questions were written without seeing the answers, producing lower lexical overlap.Its lexical overlap is 3× lower than XQuAD and 2× lower than MLQA.
- 3.3. Languages: XTREME covers 40 languages belonging to 12 language families and two isolates, selected using typological diversity and monolingual-data availability.
- 3.3. Languages: Although each language has gold-standard data in at least one task, XTREME does not cover every language aspect relevant to transfer and offers only a glimpse of generalization.
- 3.3. Languages: Coverage across most languages is possible only for POS, NER, and Tatoeba, while pseudo test sets broaden comparisons for classification and question answering.
- 3.4. Pseudo test data for analyses: Automatically translated XQuAD test sets underestimate mBERT’s performance by 3.0 F1 and 0.2 EM points, while translated XNLI sets overestimate accuracy by 2.4 points.
4. Experiments
XTREME evaluates multilingual models in English-source zero-shot transfer across diverse tasks and languages, comparing representation-based, translation-based, and in-language baselines. The results show strong progress from XLM-R and machine translation, but persistent cross-lingual gaps and task- and language-dependent limitations.
- Training and evaluation setup: XTREME evaluates multilingual representations by fine-tuning on English labelled data and testing on target-language data without target-language annotations.The benchmark focuses on zero-shot cross-lingual transfer with English as the source language.
- Baselines: The baselines include multilingual pretrained models, translation-based transfer, in-language training, few-shot training, and multilingual multi-task training.Translation-based settings translate training or test data, while in-language settings use target-language labelled examples when available.
- Results: XLM-R is the best-performing zero-shot transfer model and generally improves significantly upon mBERT, although gains are smaller on structured prediction tasks.MMTE is competitive with mBERT on most tasks and is stronger on XNLI, POS, and BUCC.
- Results: A strong machine-translation system improves translated-training performance over zero-shot transfer, while translate-test offers similar benefits and is especially effective for complex QA at higher inference cost.The benchmark notes that translation-based methods can provide strong baselines but may be further improved for data augmentation.
- Results: Multilingual models trained on in-language data outperform zero-shot models where such data exists, but zero-shot transfer can outperform models trained on only 1,000 target-language examples for complex QA.For structured prediction, 1,000 in-language examples achieve performance similar to full labelled-data training; multi-task learning generally improves single-language training.
- Results: XLM-R reduces transfer gaps on challenging QA tasks more than mBERT, but has less impact on syntactic structured prediction; machine translation reduces gaps across tasks, yet substantial gaps remain.English performance is used as a proxy for human performance, although performance may differ across languages.
5. Analyses
The analyses show that cross-lingual performance varies substantially across tasks and languages, with particularly persistent weaknesses in syntactic transfer and several language families. Performance also correlates with pretraining-data size for most tasks, but less so for structured prediction.
- Best zero-shot model analysis: Cross-lingual transfer performance varies substantially across both tasks and languages, motivating evaluation beyond a single benchmark task or language.The analysis examines model performance across tasks, languages, language families, and scripts.
- Best zero-shot model analysis: Scores are relatively tightly clustered for XNLI and PAWS-X, whereas the remaining tasks show substantially wider spreads, especially with pseudo test sets.Figure 1 compares English, non-English averages, human estimates, and language-family performance.
- Correlation with pretraining data size: ρ ≈0.8 for most mBERT tasks, compared with ρ ≈0.35 for structured prediction, in correlations between performance and Wikipedia article counts.Similar trends occur for XLM and XLM-R, although XLM-R has lower correlations associated with a different pretraining domain.
- Analysis of language characteristics: Indo-European branches generally show the best transfer, while Niger-Congo, Kra-Dai, Sino-Tibetan, Japonic, and Koreanic languages show lower performance.The analysis also reports differences among writing scripts, including syntactic-task differences between Latin and ideogram scripts.
- Errors across languages: mBERT predicts 71.8% of XNLI examples correctly in another language when they were correct in English, while misclassified English examples perform about randomly.Across languages, XNLI predictions are consistent for 68.3% of examples; on XQuAD, roughly 60% of English-correct examples and 20% of English-incorrect examples are correct.
- Generalization to unseen tag combinations and entities: mBERT struggles most on unseen POS-tag trigrams and 4-grams, indicating difficulty transferring target-language syntactic characteristics.The analysis compares target-language tag combinations that did and did not appear in English training data.
6. Conclusions
XTREME concludes that cross-lingual transfer varies significantly between tasks and languages, while offering broader coverage and analysis tools for estimating multilingual generalization. The benchmark covers diverse typologies and releases resources to support further research.
- 6. Conclusions: XTREME’s cross-lingual transfer performance varies significantly between tasks and languages.The conclusion identifies this variation as central to estimating cross-lingual generalization ability.
- 6. Conclusions: XTREME remains limited by the data coverage of its constituent tasks for many low-resource languages.The authors nevertheless describe its coverage and analysis tools as significantly broader and more fine-grained.
- 6. Conclusions: The authors release code and fine-tuning scripts to encourage research on cross-lingual generalization.The released resources support evaluation and model training on XTREME tasks.
- 6. Conclusions: The benchmark includes language-family members spanning Afro-Asiatic, Austro-Asiatic, Austronesian, Dravidian, Indo-European, Japonic, Kartvelian, Kra-Dai, Niger-Congo, Sino-Tibetan, Turkic, and Uralic families, plus Basque and Korean.The listed languages provide typological coverage for the benchmark.
C. Translations for QA datasets
The QA translation pipeline protects answer spans during translation and uses fuzzy matching as a fallback. This combination recovers more than 97% of answer spans in the training and test data.
- C. Translations for QA datasets: Answer spans are enclosed in HTML tags because quotes were often dropped during translation at language-dependent rates.The authors found HTML tags more reliable than enclosing answer spans in quotation marks.
- C. Translations for QA datasets: If tagged spans fail, the pipeline fuzzy-matches translated answers to the context and drops examples exceeding a specified edit-distance threshold.The threshold is min(10, answer len/2).
- C. Translations for QA datasets: More than 97% of answer spans are recovered across the training and test data using the combined procedure.Recovery uses HTML tagging first and fuzzy matching as a back-off.
D. Performance on translated test sets
Automatically translated QA test sets provide a useful proxy for cross-lingual performance, but their bias differs between mBERT and translate-train baselines.
- D. Performance on translated test sets: Automatically translated test sets underestimate mBERT by 2.9 F1 / 0.2 EM points but overestimate translate-train by 4.0 F1 / 6.7 EM points.The largest part of the margin is explained by differences on the Thai test set.
- D. Performance on translated test sets: Automatically translated test sets are useful as a proxy for cross-lingual performance, despite these baseline-specific score differences.
E. mBERT performance across tasks and languages
This section reports mBERT’s performance across all XTREME tasks and languages, alongside benchmark language statistics and baseline hyperparameters.
- mBERT performance is reported across all XTREME tasks and languages in Table 4.
- XTREME’s languages span 12 language families and two isolates, with Indo-European containing the most members.
- Baseline and state-of-the-art model hyperparameters are summarized, excluding XLM-15 and XLM-R-Base from the experiments.
F. Correlation with pretraining data size
The paper examines how zero-shot transfer relates to pretraining data size and evaluates mBERT on seen and unseen POS tag sequences.
- Pearson correlations between zero-shot transfer performance and Wikipedia size are reported across datasets and models.
- mBERT is evaluated on POS tag trigrams and 4-grams that were seen or unseen in English training data.
H. Generalization to unseen entities
The section analyzes generalization using unseen target-language entities and compares representation choices and evaluation conditions across tasks.
- Generalization to unseen entities: mBERT performance is compared on target-language NER entities seen and unseen during English NER training.
- Sentence representations: Sentence retrieval experiments compare [CLS], middle-layer averages, and concatenated bottom, middle, and top-layer representations in BUCC.
- Evaluation analyses: The section includes comparisons involving translated and automatically translated test sets, language families, scripts, and Wikipedia-size correlations.
J. Results for each task and language
Detailed results are provided for each XTREME task and language, with tables specifying the evaluation metric for each task.
- Detailed results cover XNLI, PAWS-X, POS, NER, XQuAD, MLQA, TyDiQA-GoldP, BUCC, and Tatoeba.
- The tables report XNLI and PAWS-X accuracy, POS accuracy, NER F1, and BUCC F1 scores for each language.
- Tatoeba results are reported using accuracy for each language.
- Question-answering results are reported as F1 and EM for XQuAD, TyDiQA-GoldP, and MLQA.
- Additional comparisons include translate-train and monolingual baselines.