PaperlayerView exact revision on arXiv

Source-linked AI summary

The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants

Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, Madian Khabsa

arXiv:2308.16884v2cs.CLcs.AIcs.LG

TL;DR

Multilingual reading-comprehension evaluation lacks broad, parallel coverage, especially for lower-resource languages. The paper introduces BELEBELE, a 122-language multiple-choice benchmark, and finds that English-centric LLMs transfer across languages but smaller MLMs trained on balanced multilingual data understand more languages. The study also reports that larger, deliberately constructed vocabularies correlate with better performance on low-resource languages.

  • Problem

    High-quality parallel benchmarks and labeled data for evaluating multilingual text comprehension remain limited, especially across low-resource languages.

  • Method

    The paper constructs BELEBELE from FLORES passages as a parallel multiple-choice reading-comprehension benchmark and evaluates multilingual MLMs and LLMs in several settings.

  • Results

    English BELEBELE is difficult for models: ROBERTA-base reaches 71.7 accuracy with full fine-tuning, while LLAMA 2 70B reaches 90.9 in five-shot evaluation.

  • Takeaways & Limitations

    BELEBELE enables direct multilingual comparisons and shows that balanced multilingual pretraining can support broader language understanding than larger English-centric models.

  • Takeaways & Limitations

    Comparisons involving GPT3.5-TURBO may be unfair because its pretraining and fine-tuning data cannot be verified.

Abstract

from arXiv · show

We present Belebele, a multiple-choice machine reading comprehension (MRC) dataset spanning 122 language variants. Significantly expanding the language coverage of natural language understanding (NLU) benchmarks, this dataset enables the evaluation of text models in high-, medium-, and low-resource languages. Each question is based on a short passage from the Flores-200 dataset and has four multiple-choice answers. The questions were carefully curated to discriminate between models with different levels of general language comprehension. The English dataset on its own proves difficult enough to challenge state-of-the-art language models. Being fully parallel, this dataset enables direct comparison of model performance across all languages. We use this dataset to evaluate the capabilities of multilingual masked language models (MLMs) and large language models (LLMs). We present extensive results and find that despite significant cross-lingual transfer in English-centric LLMs, much smaller MLMs pretrained on balanced multilingual data still understand far more languages. We also observe that larger vocabulary size and conscious vocabulary construction correlate with better performance on low-resource languages. Overall, Belebele opens up new avenues for evaluating and analyzing the multilingual capabilities of NLP systems.

1 Introduction

BELEBELE addresses limited multilingual reading-comprehension evaluation by providing a parallel benchmark across 122 language variants. Its carefully designed questions challenge models while enabling direct cross-language comparisons.

  • Existing multilingual benchmarks provide limited coverage, while many languages lack labeled data needed to build functional NLP systems.FLORES-200 primarily focuses on machine translation, and common multilingual benchmarks together cover around 30 languages.
  • The paper examines how multilingual masked language models and large language models perform across the benchmark.
  • BELEBELE evaluates language models across 122 language variants using multiple-choice reading-comprehension questions.The dataset contains 900 unique passages and questions designed to distinguish models with different language-comprehension competence.
  • English BELEBELE questions challenge numerous models even though humans answer them with near-perfect accuracy.
  • Because BELEBELE is parallel across languages, it supports direct performance comparisons and cross-lingual evaluation settings.The benchmark evaluates masked language models through English fine-tuning or translation and evaluates LLMs through in-context learning and zero-shot settings.

2 Background

Prior multilingual NLU resources support cross-lingual evaluation but remain limited in language coverage or task design. BELEBELE extends this evaluation landscape to 122 languages, many without existing NLU benchmarks.

  • 2.1 Cross-Lingual Evaluation Benchmarks: Existing parallel NLU datasets include XNLI, XQUAD, MLQA, and XL-SUM, but together cover under 30 languages, mostly high- or medium-resource.
  • 2.1 Cross-Lingual Evaluation Benchmarks: MASSIVE covers 51 languages in spoken conversational-agent tasks, while NER and TYDIQA provide extensive coverage but are not parallel.
  • 2.1 Cross-Lingual Evaluation Benchmarks: BELEBELE expands cross-lingual evaluation to 122 languages, many of which previously lacked any NLU benchmark.
  • 2.2 Non-English Machine Reading Comprehension: Machine reading comprehension uses a provided passage to answer each question, and most existing MRC datasets are in English.
  • 2.2 Non-English Machine Reading Comprehension: BELEBELE aims to cover languages represented by monolingual MRC resources in one consistent dataset.
  • 2.3 Multiple Choice QA: Multiple-choice MRC is less common than extractive QA, while multilingual EXAMS covers 28 languages without providing passages.

3 The BELEBELE Dataset

BELEBELE was constructed as a parallel, multilingual multiple-choice reading-comprehension benchmark from FLORES passages, with iterative quality control and expert translation. Its 900 questions span 122 languages and are designed to resist shortcut strategies while supporting comparable evaluation.

  • 3.1 Creation of Multiple Choice Questions & Answers: The dataset begins with English multiple-choice questions and answers, which are translated to create comparable samples across languages.The authors chose multiple-choice questions because span extraction is more sensitive to morphological differences and can be harder to scale multilingualy.
  • 3.1 Creation of Multiple Choice Questions & Answers: Strong negatives, iterative Language Service Provider feedback, and manual and automatic checks were used to improve question quality.The process included five iterations and filtered about 20% of questions in the final iteration using heuristic checks.
  • 3.1 Creation of Multiple Choice Questions & Answers: 0.81 was the final collection’s t-test p-value, compared with p-value < 0.01 for MCTEST, indicating similar feature distributions for correct and wrong answers.
  • 3.1 Creation of Multiple Choice Questions & Answers: 0.28 was the best accuracy of a bag-of-words logistic-regression model, only slightly above random accuracy of 0.25 and below MCTEST’s 0.44.
  • 3.3 Translating the Corpus: BELEBELE uses FLORES-200 passages, with expert bilingual annotation and transliteration for five Indo-Aryan languages in Latin script.The corpus includes native and romanized forms for Hindi, Bengali, Urdu, Nepali, and Sinhala.
  • 3.5 The BELEBELE Dataset in Summary: BELEBELE is intended as a test set rather than a training set, with English multiple-choice QA samples supplied for models requiring task fine-tuning.
  • 3.5 The BELEBELE Dataset in Summary: 900 questions with four answers each produce 109,800 rows across 122 languages, 29 scripts, and 27 language families.The corpus contains 488 distinct passages, and none belongs to the hidden FLORES test set.
  • 3.5 The BELEBELE Dataset in Summary: Semantic alignment across passages, questions, and answers supports equivalent challenge levels and cross-lingual swapping of benchmark components.

4 Experiments

The experiments establish baseline accuracy across 122 language variants, comparing multilingual masked language models and large language models under fine-tuning, in-context, and zero-shot settings. Results contrast models trained on intentionally multilingual data with models trained predominantly on English.

  • Evaluation Setup: Accuracy is the central metric, with four candidate answers yielding 0.25 expected accuracy for random sequence-classification guesses.This lower bound does not apply to sequence-to-sequence models evaluated with exact match.
  • Masked Language Models: The study evaluates XLM-V, INFOXLM, and XLM-R, whose intentionally multilingual pretraining data covers about 100 languages.High-resource languages are typically down-sampled and low-resource languages up-sampled to favor multilingual performance.
  • Large Language Models: The evaluated large language models include GPT3.5-TURBO, FALCON 40B, and LLAMA 1 and 2, including LLAMA-2-CHAT.LLAMA 1 and 2 use predominantly English pretraining data, while FALCON 40B was trained on filtered web-crawled samples.
  • Evaluation Settings: MLMs are evaluated after full-model fine-tuning on English or machine-translated training data across target languages.The two settings measure zero-shot cross-lingual transfer and Translate-Train-All performance, respectively.
  • Evaluation Settings: LLAMA 1, LLAMA 2, and FALCON 40B are evaluated with five-shot in-context learning using examples from the English training set.Predictions use the highest-probability answer, with average cumulative scores reported over three runs.
  • Evaluation Settings: GPT3.5 and LLAMA-2-CHAT are evaluated zero-shot with English instructions, while translated prompts and Translate-Test assess alternative language settings.Translate-Test machine-translates passages, questions, and answers into English before prompting the model.

5 Results

BELEBELE reveals substantial variation in multilingual reading-comprehension performance across model sizes, pretraining distributions, scripts, vocabulary sizes, and translation settings. English-centric LLMs can generalize broadly, but balanced multilingual pretraining and larger vocabularies remain important for lower-resource languages.

  • 5.1 How difficult is BELEBELE?: 71.7 accuracy was the maximum achieved by ROBERTA-base in English with full fine-tuning, versus 90.9 for LLAMA 2 70B in five-shot evaluation.LLAMA 1 7B reached 37.3, demonstrating wide performance differences among model types and sizes.
  • 5.1 How difficult is BELEBELE?: 97.6 mean accuracy was achieved by four authors on sampled English questions, exceeding every evaluated model.The result indicates that the English task remains challenging for models.
  • 5.1 How difficult is BELEBELE?: r = 0.85 was the score correlation across 15 languages and three models when comparing BELEBELE with XNLI.In Translate-Train-All, XLM-V, INFOXLM, and XLM-R scored about 10 accuracy points lower on BELEBELE than on XNLI Translate-Train.
  • 5.2 Multilingual Generalization of MLMs and LLMs on BELEBELE: LLAMA-2-CHAT exceeded 35 accuracy in 59 languages, while XLM-R reached accuracy 50 in about twice as many languages as LLAMA 2.LLAMA 2 largely outperformed XLM-R on high-resource languages, whereas XLM-R covered more languages at the stated threshold.
  • 5.2 Multilingual Generalization of MLMs and LLMs on BELEBELE: Translate-Test was better than in-language LLAMA-2-CHAT in 68 of 91 evaluated languages, while only German and Italian were non-trivially better in-language.Zero-shot accuracy above 50% increased from 33 languages in the traditional setting to 71 with Translate-Test.
  • 5.2 Multilingual Generalization of MLMs and LLMs on BELEBELE: XLM-V outperformed XLM-R and INFOXLM on low-resource languages, while smaller-vocabulary models fell off abruptly for medium- and low-resource languages.The compared MLMs share the same architecture and CC-100 training dataset; XLM-V uses a 900k-token vocabulary versus 250k for XLM-R and INFOXLM.
  • 5.2 Multilingual Generalization of MLMs and LLMs on BELEBELE: Only LLAMA 1 30B and 65B performed non-trivially in Japanese and Greek, language families not reported in the pretraining corpus.Performance increased significantly across the six language families and English as parameter size grew from 7B to 65B.
  • 5.2 Multilingual Generalization of MLMs and LLMs on BELEBELE: All evaluated models except FALCON performed more strongly in native scripts than in Latin-script variants for the listed languages.The comparison covered Romanized and original scripts for Hindi, Urdu, Bengali, Sinhala, Nepali, and Modern Standard Arabic.

6 Conclusion

BELEBELE addresses the limited availability of annotated multilingual reading-comprehension benchmarks by evaluating models in 122 languages. The results link stronger medium- and low-resource performance with larger vocabularies and balanced pretraining, while showing that English-centric LLMs can generalize beyond English.

  • 6 Conclusion: Annotated benchmarks are scarce for low- and moderate-resource languages, limiting sound evaluation of language-model capabilities.BELEBELE is introduced to address this limitation with multilingual reading-comprehension data.
  • 6 Conclusion: BELEBELE contains passages and multiple-choice questions for reading-comprehension evaluation in 122 languages.The benchmark is described as the first of its kind in many medium- and low-resource languages.
  • 6 Conclusion: Larger vocabulary size and balanced pretraining data correlate with the highest performance on medium- and low-resource languages.The conclusion also reports that English-centric LLMs can generalize to over 30 languages.
  • 6 Conclusion: Future evaluations with BELEBELE may connect multilinguality with capabilities such as reasoning and support broader understanding beyond high-resource languages.The authors propose using the benchmark to complement investigations into specific model capabilities.

Limitations

The paper identifies limitations involving opaque model-training data, quality issues in FLORES translations, and possible cross-language differences caused by translationese.

  • Pretraining Documentation: Inconsistent pretraining-corpus documentation limits the model analyses, and GPT3.5-TURBO data cannot be verified.Comparisons involving GPT3.5-TURBO may therefore be unfair because its training data are not transparent.
  • Errors in FLORES: Annotators found some FLORES translation quality issues, particularly in lower-resource languages, despite extensive quality assurance.The authors reviewed cases with the language service provider to improve appropriateness and cross-language consistency.
  • Translationese: Translationese may change the task across languages because perfect translations often do not exist.The authors warn that non-English accuracy may not be directly comparable with English accuracy.

Ethics Statement

Open-sourcing BELEBELE may weaken future benchmark fairness, while the benchmark’s design does not capture language- and culture-specific phenomena.

  • Open-Source: Open-sourcing BELEBELE may allow samples into future pretraining corpora, undermining fair comparison, especially for zero- or few-shot evaluation.The authors nevertheless judge the value of releasing the full dataset to outweigh this concern.
  • English-centrism: BELEBELE prioritizes cross-language comparability and therefore does not capture phenomena such as formality, values, and aboutness.The authors state that future benchmarks should capture these culture- and language-specific phenomena.

A.1 Languages and Variants

BELEBELE represents 122 language variants across diverse scripts and language families, including selected languages in both native and Latin scripts.

  • Script Variants: Bengali, Hindi, Sinhala, Nepali, Urdu, and Modern Standard Arabic appear in both their native scripts and Latin script.Chinese is included in Simplified and Traditional characters.
  • Language Families: The dataset includes 50 Indo-European languages, with Indo-Aryan the largest listed family at 17 languages.Atlantic-Congo and Afro-Asiatic each contain 16 languages in the displayed grouping.
  • Coding: The language code used is from FLORES-200 and differs from the older FLORES-101 code.The paper directs readers to the FLORES website for details.

A.2 Training Set

The authors construct English training and development data from compatible multiple-choice reading-comprehension datasets because similarly formatted options were limited outside English.

  • Dataset Selection: English datasets were selected because few similarly formatted multiple-choice datasets were available outside English.The training and development sets were created to support task fine-tuning for evaluation.
  • Dataset Selection: 33 candidate MRC datasets were considered, and six were selected for compatibility: RACE, SCIQ, MULTIRC, MCTEST, MCSCRIPT2.0, and RECLOR.The source passages and questions were unpacked and restructured, including reformatting MCSCRIPT2.0’s two-option questions.
  • Dataset Size: The resulting training set contains 67.5k samples and the development set contains 3.7k samples.More than half of these samples come from RACE.
  • Licensing: The training set is a joint sample of other datasets and is governed by a different license than BELEBELE.The authors do not claim ownership of those source works or datasets.

A.4 Experiment Details

The experiments specify training, prompting, translation, and evaluation procedures for multilingual models, alongside comparisons across scripts, resource levels, and model families.

  • Evaluation settings: MLMs were fine-tuned on English data or translated data, while LLMs were evaluated with in-context learning and zero-shot prompting.Translate-Train-All machine-translated passages, questions, and answers separately before training.
  • MLM training: 3 or 4 epochs was optimal for MLM fine-tuning on the English development set.
  • Scoring: Candidate answers were selected by highest relative probability, and reported results averaged scores over 3 runs.
  • LLM evaluation: Zero-shot LLM evaluation presented each passage, question, and four answers, instructing models to output an answer letter.Instructions were provided in English for all languages, with post-processing accepting variants such as “(A)”.
  • Cross-lingual MLMs: INFOXLM and XLM-V generally outperformed XLM-R, with INFOXLM stronger in higher- and medium-resource languages and XLM-V stronger in the lowest-resource languages.XLM-V consequently had the most scores above 50, while INFOXLM had more scores above 70.
  • Translate-Test: Translate-Test outperformed in-language zero-shot performance for LLAMA-2-CHAT on most of 91 evaluated languages, with nearly all low-resource languages improving by over 20 accuracy points.German and Italian were among the few languages with higher traditional in-language scores.
Loading 2308.16884v2…