PaperlayerView exact revision on arXiv

Source-linked AI summary

ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic

Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Boda Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, Timothy Baldwin

arXiv:2402.12840v2cs.CL

TL;DR

Arabic reasoning and knowledge evaluation is constrained by limited Arabic-specific datasets, with assessments often relying on English-to-Arabic translations. ArabicMMLU introduces a native-context benchmark from school exams across Arabic-speaking regions. Evaluations find Arabic-centric models outperform multilingual models but remain below GPT-4.

  • Problem

    Arabic LLM reasoning and knowledge are often evaluated with English-to-Arabic translated datasets, limiting evaluation of Arabic-specific content and potentially introducing errors and biases.

    Source: §1 · p. 1

  • Method

    ArabicMMLU is an MSA multiple-choice benchmark of 40 tasks sourced from school exams in eight countries, with questions selected and designed with native Arabic speakers.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Arabic-centric LLMs outperform multilingual LLMs on ArabicMMLU, but achieve lower accuracy than GPT-4.

    Source: §5 · p. 9

  • Takeaways & Limitations

    ArabicMMLU is intended as a resource for tracking future Arabic LLMs’ real-world knowledge and reasoning capabilities.

    Source: §5 · p. 9

    ArabicMMLU represents countries unevenly: it includes over 6K questions from Jordan, while some countries have only 100 or none.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The focus of language model evaluation has transitioned towards reasoning and knowledge-intensive tasks, driven by advancements in pretraining large models. While state-of-the-art models are partially trained on large Arabic texts, evaluating their performance in Arabic remains challenging due to the limited availability of relevant datasets. To bridge this gap, we present \datasetname{}, the first multi-task language understanding benchmark for the Arabic language, sourced from school exams across diverse educational levels in different countries spanning North Africa, the Levant, and the Gulf regions. Our data comprises 40 tasks and 14,575 multiple-choice questions in Modern Standard Arabic (MSA) and is carefully constructed by collaborating with native speakers in the region. Our comprehensive evaluations of 35 models reveal substantial room for improvement, particularly among the best open-source models. Notably, BLOOMZ, mT0, LLaMA2, and Falcon struggle to achieve a score of 50%, while even the top-performing Arabic-centric model only achieves a score of 62.3%.

1 Introduction

ArabicMMLU addresses the limited evaluation of Arabic-specific reasoning and knowledge by introducing an MMLU-style benchmark built from regionally sourced school exams. The paper also evaluates multilingual, Arabic-centric, and closed-source models and analyzes top open-source models across multiple dimensions.

1 Introduction

  • English-to-Arabic translated datasets are commonly used to assess Arabic LLM knowledge, limiting evaluation of content specific to Arabic and potentially introducing errors and biases.The paper argues for datasets localized to Arabic-speaking regions and cultures.Source: §1 · p. 1
  • Reasoning and knowledge evaluation has gained prominence, and MMLU assesses knowledge through multiple-choice questions across subjects based on the US education system.Recent Arabic-centric models Jais and AceGPT used MMLU translated from English to Arabic for knowledge evaluation.Source: §1 · pp. 1–2
  • ArabicMMLU draws on school exams from North Africa, the Levant, and the Gulf, with native-speaker collaboration from Jordan, Egypt, UAE, Lebanon, and Saudi Arabia.The highlighted subject areas include history, geography, law, civics education, and driving tests.Source: §1 · p. 2
  • ArabicMMLU is the first Arabic MMLU-style dataset in MSA, with 40 tasks across subject areas and educational levels in eight countries; over 50% of questions are tailored to Arabic-specific contexts.The authors evaluate 22 open-source multilingual models, 11 open-source Arabic-centric models, and 2 closed-source models, and analyze top open-source models by subjects, levels, countries, Arabic-specific topics, few-shot performance, confidence, and negation.Source: §1 · p. 2

2 Related Work

Prior Arabic language models range from early, mostly monolingual models to newer bilingual Arabic-centric and multilingual LLMs. Existing Arabic benchmarks include many language tasks and reading-comprehension datasets, while MMLU-style real-world reasoning and knowledge evaluation remains less represented.

2.1 Language Models in Arabic

  • Early Arabic pretrained language models typically had less than 2 billion parameters and were primarily monolingual, falling into three categories: encoder-only, decoder-only, and encoder–decoder models.Recent Arabic-centric decoder-only models Jais and AceGPT have parameter sizes of up to 30B and 13B, respectively.Source: §2.1 · p. 2
  • Only GPT-4 among the cited original multilingual-model papers was evaluated for Arabic reasoning and knowledge using translated MMLU, reporting 80% accuracy.The passage describes this as English–Arabic translated MMLU.Source: §2.1 · p. 2

2.2 Arabic Benchmarks for Evaluating Language Models

  • Arabic benchmarks include classification, language generation, and question-answering datasets, but TyDiQA, Arabic-SQuAD, and MLQA primarily target reading comprehension rather than MMLU-style real-world reasoning and knowledge.EXAMs is based on multilingual school exams and includes about 500 Arabic questions.Source: §2.2 · p. 3

3 ArabicMMLU

ArabicMMLU is an Arabic multiple-choice dataset spanning 40 tasks, eight countries, and multiple education levels. The section describes native-speaker collection, answer-key checks, and distributions across levels, subjects, and countries.

3 ArabicMMLU

  • ArabicMMLU contains 40 tasks across eight countries and multiple education levels, with English questions excluded.The questions are drawn from countries in North Africa, the Levant, and the Gulf.Source: §3 · p. 3

3.1 Data Construction

  • Ten native Arabic speakers from several countries collected questions and metadata, while workers excluded questions without answer keys or with multimodal content.Manual collection took place over two months; workers were instructed to include any context referenced by multiple questions with each corresponding question.Source: §3.1 · p. 4

3.2 Quality Control

  • 96% of 100 randomly sampled questions and answer keys matched on average, which the authors treat as ArabicMMLU’s maximum meaningfully achievable score.The dataset was filtered from over 15,000 questions to 14,575 unique questions.Source: §3.2 · p. 4

3.3 Data Statistics

  • About 4.9K questions are from primary school versus 3.2K from high school; high-school and university questions and options are generally longer.The “NA” category’s questions are four times longer than school-exam questions, reflecting lengthy contexts in subjects such as Arabic language.Source: §3.3 · p. 5
  • STEM, social science, and humanities each contain roughly 3.2K–3.5K questions, while language questions average 10 times the length of other categories.The subject areas differ only slightly in average question length outside the language category.Source: §3.3 · p. 5
  • Arabic-specific content represents 57.7% of the dataset, and Jordan, Egypt, and Palestine are its top source countries.The authors note that curriculum differences across regions may influence assessment-question design.Source: §3.3 · p. 5

4 Experiments

The experiments show that GPT-4 leads overall, while open-source models—especially multilingual models—lag behind. Results also vary with prompt configuration, education level, country, and analysis dimension.

4.1 Set-Up

  • Across 35 models, the four tested prompt/script configurations favored English prompts and English alphabetic output; Arabic-centric models were most robust with Arabic options.The English-prompt, English-output configuration was used for the remaining experiments.Source: §4.1 · pp. 5–6, §4.2 · p. 6, Figure 5 · p. 7

4.2 Results

  • GPT-4 scores 72.5% accuracy, 10 points above Jais-chat (30B), the top-performing open-source model at 62.3%.Jais-chat (30B) surpasses GPT-3.5 by 4.6 points; BLOOMZ (7B) and mT0 (13B) trail Jais by more than 14 points.Source: Table 4 · p. 6, §4.2 · p. 7
  • BLOOMZ (7B) scores 15.9 points higher than BLOOMZ (560M), and mT0 (13B) scores 13.8 points higher than mT0 (300M), with XGLM an exception to the size trend.The trend is also present in AceGPT and Jais, but less pronounced in LLaMA2 and Falcon.Source: §4.2 · p. 7
  • 6% of questions are university-level, a potentially skewing share; high-school items were harder, while Jais led country-grouped subjects except Morocco.On high-school questions, GPT-4 scores 61.7% and Jais-chat scores 51.2%; BLOOMZ performs less well on UAE- and Morocco-sourced questions.Source: §4.2 · pp. 7–8, Table 5 · p. 7

4.3 Analysis

  • Few-shot learning improves base-model results but worsens instruction-tuned results; AceGPT and Jais gain 2–10 points, while BLOOMZ and Jais-chat drop.The pattern is reported as consistent with prior research over IndoMMLU and CMMLU.Source: Figure 7 · p. 8, §4.3 · p. 8
  • BLOOMZ, AceGPT, and Jais are well calibrated across tasks (r > 0.9), while question length has no correlation with confidence for Jais or AceGPT.Calibration compares each task's correct-answer probability with its accuracy.Source: Figure 8 · p. 8, §4.3 · p. 8, Figure 9 · p. 8
  • Negated questions generally have slightly lower accuracy in Biology and Economics, but models score higher on negated Geography questions.The comparison covers Geography, Biology, and Economics.Source: Table 6 · p. 9, §4.3 · p. 9

4.4 Discussion

  • Open-source models perform poorly, particularly multilingual models; Arabic-centric models still struggle with cultural knowledge across education levels, while GPT-4 leads across tasks.The authors say it remains unclear whether GPT-4's performance reflects scaling or memorization, since the questions come from public sources.Source: §4.4 · p. 9

5 Conclusion and Future Work

The paper presents ArabicMMLU as a large-scale, multi-task dataset for evaluating real-world Arabic knowledge. Experiments find Arabic-centric models outperform multilingual models but remain below GPT-4; the authors propose expanding the benchmark's coverage.

5 Conclusion and Future Work

  • ArabicMMLU is a multi-task dataset designed to evaluate real-world knowledge in Arabic.It contains over 14,000 multiple-choice questions across subjects and education levels.Source: §5 · p. 9
  • Arabic-centric LLMs outperform multilingual LLMs on ArabicMMLU, but achieve lower accuracy than GPT-4.Source: §5 · p. 9
  • Future extensions could add short-answer and essay questions, more modalities, broader regional coverage, and professional-setting questions.The authors say these additions would make evaluation better reflect real-world scenarios.Source: §5 · p. 9

Limitations

ArabicMMLU has limits in country representation, language variety, and assessment modality. Its current scope is MSA text questions, and country coverage is uneven.

Limitations

  • Jordan contributes over 6K questions, while some countries have only 100 or none, limiting balanced country coverage.The authors attribute the imbalance largely to the availability of publicly accessible exams and say their internet search was not exhaustive.Source: Limitations · p. 9
  • The dataset primarily covers Modern Standard Arabic, although multilingual and Arabic LLMs are often exposed to dialectical Arabic.Source: Limitations · p. 9
  • ArabicMMLU assesses text only; multimodal questions are left for future work.Source: Limitations · p. 9

Ethical Considerations

The experiments do not conclusively establish LLM performance in Arabic. GPT-4 outperformed all models, but its training regimen is insufficiently documented to rule out pretraining-data contamination.

A Data Statistics

Table 7 presents ArabicMMLU’s data distribution by subject and educational level.

B Examples

Figure 10 shows an example of the study’s prompts in Arabic and English, using a Natural Science question.

C Detailed Experiment Results

The appendix reports zero-shot performance across subjects and education levels, plus results under alternative prompt and answer-output languages.

  • Table 8 reports zero-shot accuracy by subject and education level for BLOOMZ, AceGPT-chat, Jais-chat, GPT-3.5, and GPT-4.Source: Table 8 · p. 15
  • Table 9 reports zero-shot accuracy with Arabic prompts and Arabic alphabetic outputs, combined across subject groups.Source: Table 9 · p. 16
  • Tables 10 and 11 report combined-subject zero-shot accuracy for Arabic prompts with English outputs and English prompts with Arabic outputs, respectively.These prompt and alphabetic-output variants complement Table 8.Source: C Detailed Experiment Results · p. 15, Table 10 · p. 17, Table 11 · p. 18

D Model Artifacts

The study’s models were sourced from Huggingface, except GPT-3.5 and GPT-4.

D Model Artifacts

  • Except GPT-3.5 and GPT-4, all models used in the study were sourced from Huggingface (Wolf et al., 2020).Table 12 identifies the source of the models used in the study.Source: Table 12 · p. 19

Related papers

Loading 2402.12840v2…