PaperlayerView exact revision on arXiv

Source-linked AI summary

XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation

Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, Melvin Johnson

arXiv:2104.07412v2cs.CLcs.AI

TL;DR

Multilingual evaluation still lacks sufficiently nuanced evidence across languages and tasks, despite rapid but uneven progress. The paper analyzes cross-lingual model performance and introduces XTREME-R, an expanded benchmark with diagnostics and fine-grained evaluation. It reports concentrated gains in retrieval, while other tasks and low-resource or non-Latin-script languages remain challenging.

  • Problem

    Multilingual evaluation must cover many typologically distinct languages despite limited, heterogeneous data, and aggregate metrics provide insufficiently nuanced comparisons.

  • Method

    The paper analyzes state-of-the-art multilingual models and extends XTREME to 50 languages, 10 tasks, language-agnostic retrieval, MultiCheckList, fine-grained evaluation, and a metadata-rich leaderboard.

  • Results

    Progress is uneven: gains concentrate on cross-lingual retrieval, while other tasks and low-resource or non-Latin-script languages remain difficult.

  • Takeaways & Limitations

    The analyses identify strong alignment, syntactic transfer, fine-grained understanding, and abstract-question answering as directions where scale alone is insufficient.

  • Takeaways & Limitations

    mT5 language-agnostic retrieval scores are not directly comparable with mBERT and XLM-R because compute limits required frozen embeddings rather than dual-encoder fine-tuning.

Abstract

from arXiv · show

Machine learning has brought striking advances in multilingual natural language processing capabilities over the past year. For example, the latest techniques have improved the state-of-the-art performance on the XTREME multilingual benchmark by more than 13 points. While a sizeable gap to human-level performance remains, improvements have been easier to achieve in some tasks than in others. This paper analyzes the current state of cross-lingual transfer learning and summarizes some lessons learned. In order to catalyze meaningful progress, we extend XTREME to XTREME-R, which consists of an improved set of ten natural language understanding tasks, including challenging language-agnostic retrieval tasks, and covers 50 typologically diverse languages. In addition, we provide a massively multilingual diagnostic suite (MultiCheckList) and fine-grained multi-dataset evaluation capabilities through an interactive public leaderboard to gain a better understanding of such models. The leaderboard and code for XTREME-R will be made available at https://sites.research.google/xtreme and https://github.com/google-research/xtreme respectively.

1 Introduction

Multilingual NLP has advanced beyond English-focused evaluation, but progress remains uneven across languages and tasks. XTREME-R responds with broader coverage, harder tasks, diagnostics, and more detailed evaluation tools.

  • Multilingual NLP can benefit more than three billion low-resource-language speakers while reducing the field’s overfitting to English.
  • The gap between XLM-R Large and human-level performance shrank from roughly 25 to less than 12 points, but remains substantial.
  • Progress has concentrated on cross-lingual retrieval, while other task categories show more modest improvements and weaker performance on low-resource or non-Latin-script languages.Fine-tuning on additional translated data generally gives the best performance.
  • XTREME-R covers 50 typologically diverse languages and 10 challenging, diverse tasks, including language-agnostic retrieval from large multilingual candidate pools.
  • MULTICHECKLIST probes question-answering capabilities in 50 languages, while ExplainBoard enables language- and task-specific performance diagnosis.
  • The benchmark adds a newly created retrieval task, a massively multilingual diagnostic suite, fine-grained evaluation, experiments, and an interactive metadata-rich leaderboard.

2 Examining the State of Multilingual Benchmarking

Recent multilingual models substantially improve aggregate XTREME performance, but gains are concentrated in retrieval rather than question answering or structured prediction. Language family and script differences remain important sources of variation.

  • Multilingual models improved average XTREME performance from 55.8 to 81.4, with retrieval rising from 47.7 for mBERT to 92.7 for VECO.
  • Question answering and structured prediction improved only slightly compared with retrieval-based tasks.
  • Recent models still struggle with some low-resource Tatoeba languages, while MLQA remains below English and POS tagging is lower for some non-Latin-script and low-resource languages.
  • Token-level MLM pre-training is of limited use for cross-lingual sentence retrieval, whereas sentence-level fine-tuning can mitigate this weakness.
  • Parallel-data pre-training improves retrieval by making pre-training more similar to the downstream setting, but does not significantly improve other tasks.

3 XTREME-R

XTREME-R extends XTREME into a more challenging and nuanced benchmark for cross-lingual transfer, retaining accessibility while broadening task and language coverage. It also adds diagnostic and fine-grained evaluation capabilities for examining multilingual models.

  • XTREME-R retains XTREME’s accessible benchmark design while targeting challenging research directions in cross-lingual model generalization.
  • XTREME-R makes structured prediction more realistic by providing full input sentences and removing gold tokenization from NER.These changes support models operating directly from raw input strings and require predicted or tokenization-free approaches.
  • Its two new retrieval tasks evaluate language-agnostic ranking over multilingual candidate pools for answers and entity descriptions.LAReQA contains 13,090 questions with 11 target answers among 13,014 candidates, while Mewsli-X retrieves entities from pools reaching 1M candidates across 50 languages.
  • The diagnostic and evaluation suite adds multilingual capability tests and granular breakdowns by task-specific and task-independent attributes.MULTICHECKLIST probes question answering capabilities in 50 languages, while EXPLAINABOARD partitions test sets into attribute-based buckets.
  • An interactive leaderboard exposes submission metadata and task- and language-specific sub-leaderboards to support more informed model comparison.Requested metadata includes parameter counts and pre-training data amounts.

4 Experiments

XTREME-R evaluates multilingual models with established multilingual and translation-based baselines across challenging tasks. Results show strong task-dependent differences, benefits from translated training data, and limited comparability for mT5 retrieval scores.

  • Baselines: The experiments compare mBERT, XLM-R Large, mT5-XXL, and translate-train baselines under zero-shot and translated-data settings.mT5-XXL has 13B parameters, while translate-train fine-tunes models on translated English training data.
  • Evaluation: Human performance estimates are unavailable for the new retrieval tasks because identifying translations among many candidates is too time-consuming for humans.For XCOPA, human performance is estimated by averaging disagreement with the majority label across languages.
  • Results: New retrieval tasks are challenging for current models, while mT5 improves classification and question answering but performs less well on structured prediction and retrieval.The retrieval tasks test cross-lingual alignment, and XCOPA requires cross-lingual common-sense reasoning.
  • Results: Fine-tuning on task-specific translations is beneficial in all cases and generally performs best, although improvements on question answering tasks are diminishing.The paper uses translated training data for multilingual baselines and creates new translations for XCOPA and SIQa.
  • Caveats: mT5 language-agnostic retrieval results are not directly comparable with mBERT and XLM-R because they use frozen embeddings rather than a fine-tuned dual encoder.This choice was made because of compute limitations.

5 Analyses

The analyses use MULTICHECKLIST and system-diagnosis attributes to expose multilingual weaknesses that aggregate scores obscure. They identify language, script, input-length, lexical-overlap, and question-type patterns across systems.

  • MULTICHECKLIST: XLM-R is substantially more robust cross-lingually than mBERT, whose average error rate exceeds 85% in four of six MULTICHECKLIST categories.XLM-R was fine-tuned on English SQuAD v1.1 and evaluated across six tests.
  • MULTICHECKLIST: XLM-R performs worst in low-resource or non-Latin-script languages, including Gujarati, Haitian Creole, Japanese, Thai, and Chinese.The analysis lists languages with limited or no pre-training data and languages using non-Latin scripts.
  • Diagnostic attributes: The diagnosis represents question-answering examples with context, question, and answer lengths, lexical-overlap measures, and frequent question types.BLEU-AQ measures answer-question overlap, while BLEU-QC measures question-context overlap.
  • Single-system analysis: ERNIE-M performs best on shorter answers and worst on longer answers for almost all languages, with a difference exceeding 40 absolute points in Greek.Question and context length effects vary by language.
  • System comparison: ERNIE-M struggles with frequent “what” and “how” questions, while XLM-R surpasses it in selected language-and-attribute buckets.XLM-R is better on longer answers and questions in English, shorter answers and contexts in Turkish, and answer-similar questions in Chinese.

6 Conclusions

The analyses identify directions where scaling alone is insufficient, including strong alignment, syntactic transfer, fine-grained understanding, and abstract-question answering. The paper encourages better inductive biases, pre-training objectives, and evaluation resources.

  • 6 Conclusions: Scale alone is insufficient for strong alignment, syntactic transfer, fine-grained natural language understanding, and abstract-question answering.The conclusion frames these as important directions revealed by the analyses and experiments.
  • 6 Conclusions: The paper encourages developing better inductive biases, pre-training objectives, and evaluation resources.It also makes data, translations, evaluation resources, and an interactive leaderboard available for comparative analyses.

7 Ethical Considerations

XTREME-R broadens multilingual evaluation while acknowledging under-representation, dataset scarcity, transfer bias, and the cost of large-scale models.

  • Language representation: XTREME-R has little representation of languages from the Americas and Africa because labeled datasets for these languages are scarce.
  • Language representation: Some low-data languages in XTREME-R appear in only a small number of datasets.The authors release translated training data and MULTICHECKLIST to ameliorate this limitation.
  • Evaluation: XTREME-R and its diagnostic tools aim to discourage overreliance on aggregate metrics and misleading “superhuman” interpretations.
  • Bias: Multilingual and zero-shot transfer can reflect biases toward languages with more pretraining data and toward the source language.
  • Environmental concerns: Large-scale Transformer models perform best on XTREME-R but require substantial computational resources.The paper encourages efficient adaptation methods rather than training multilingual models entirely from scratch.

B Task scores on XTREME

The XTREME leaderboard reports model performance broken down by language family for the remaining XTREME tasks.

  • Figure 2 presents model performance on the XTREME leaderboard by language family.The passage introduces the leaderboard breakdown without reporting numerical scores.
  • The language-family breakdown covers the remaining XTREME tasks.
  • The reported analysis is presented in Figure 2.

C XTREME tasks retained in XTREME-R

XTREME-R retains multilingual understanding tasks spanning inference, tagging, named entity recognition, question answering, and retrieval, while covering 50 typologically diverse languages. Its languages represent 14 families and two isolates, with diversity comparable to earlier multilingual benchmarks despite broader coverage.

  • Task descriptions: XNLI evaluates whether a hypothesis is entailed, contradicted, or neutral given a premise, using English training data and professionally translated evaluation data in 14 other languages.
  • Task descriptions: UD-POS uses Universal Dependencies v2.7 treebanks covering 104 languages, with English training data and target-language test evaluation.
  • Task descriptions: WikiANN-NER identifies location, person, and organization spans in Wikipedia text, while XQuAD requires extracting an answer span from a corresponding paragraph.
  • Evaluation coverage: XTREME-R includes language-agnostic retrieval tasks and reports performance broken down by language family on the remaining tasks in Figure 2.
  • Language coverage: XTREME-R covers languages from 14 families and two isolates, including Basque and Korean.
  • Language coverage: XTREME-R is similarly diverse to XTREME and XGLUE while covering a larger number of languages, according to typology and family indices.

E Mewsli-X Dataset

Mewsli-X is a more carefully sampled multilingual entity-linking dataset spanning 11 languages and candidates drawn from a 50-language entity base. Its evaluation uses doubly zero-shot conditions and controlled sampling, while MultiCheckList adds multilingual diagnostic tests with documented translation challenges.

  • Dataset construction: Mewsli-X is a more carefully sampled variant of Mewsli-9, expanding to 11 languages by adding Polish, Romanian, and Ukrainian while dropping Serbian.
  • Dataset construction: Entity descriptions in Mewsli-X range over all 50 XTREME-R languages, and the entity base contains approximately 12 million entities with a Wikipedia page in at least one covered language.
  • Dataset statistics: The dataset contains 15,000 test mentions covering 9,647 distinct gold entities, with candidates sampled to produce a 1,000,000-entity candidate set.The candidate set includes the gold entities plus additional entities sampled from the broader entity base.
  • Training and evaluation: Mewsli-X uses 115K English-only fine-tuning mention–entity pairs sampled from English Wikipedia hyperlinks that map outside the candidate set.The evaluation setting is doubly zero-shot because neither candidate or test entities nor non-English text are observed during fine-tuning.
  • MultiCheckList: MultiCheckList was created by translating roughly 550 words into 49 languages, generating 200 test cases per category and language, at a total translation cost of $4,360.
  • MultiCheckList: Translation design addresses gender agreement, declension, normalization, names, profession gender, question syntax, and adjective syntax through language-specific templates and controls.Gender-related controls include restricting affected tests to male names and sampling genders equally for other tests.

I Language-agnostic Retrieval Results

The paper evaluates language-agnostic retrieval by comparing query-answer language pairs and reports detailed results across tasks and languages. XLM-R Large substantially improves cross-lingual retrieval over mBERT, although cross-lingual performance remains challenging.

  • Mewsli-X and LAReQA support performance analysis by input and target language pairs.Mewsli-X has a single correct target per mention, while LAReQA uses “Limit to One Target” evaluation with other correct answers removed from the candidate pool.
  • +38% on Mewsli-X and +137% on LAReQA are XLM-R Large’s cross-lingual improvements over mBERT in pairwise mAP@20.These gains come with a slight drop on same-language cases.
  • 29–36 mAP@20 remains the cross-lingual performance range, leaving retrieval a challenging area for future work.
  • Detailed results are reported for XNLI, XCOPA, UD-POS, WikiANN-NER, XQuAD, MLQA, TyDiQA-GoldP, Tatoeba, Mewsli-X, and LAReQA.

K Nuanced Multilingual Evaluation

The paper introduces fine-grained multilingual evaluation by analyzing model performance across language pairs, task attributes, and diagnostic examples. Its interface supports single-system and pairwise analyses over filtered multilingual benchmark results.

  • Fine-grained evaluation categorizes testing examples into attribute buckets and measures performance within each bucket.The evaluation is designed to reveal performance changes across attribute values.
  • Attribute Definition: QA attributes include context, question, and answer lengths, lexical overlap measures, and question type.The five most frequent question types cover 85% of training-set questions.
  • Language-Pair Analysis: Pairwise retrieval results compare query and answer languages, with Table 14 reporting mAP@20 for same- and different-language cases.
  • Attribute Definition: Structured-prediction attributes include span label, sentence token length, entity-span token length, entity-span character length, and relative entity position.
  • Attribute Bucketing: Continuous attributes use equal-size sample intervals, while discrete attributes group examples sharing the same type.
  • Interactive Evaluation: The interface filters results by publication year, task, metric, and languages, then supports dataset-bias, single-system, and pairwise analyses.
  • Diagnostic Resources: Tables 10–13 document multilingual QA diagnostic errors, failure cases, and metadata for XTREME submissions.
Loading 2104.07412v2…