PaperlayerView exact revision on arXiv

Source-linked AI summary

Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation

Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David I. Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Wei-Yin Ko, Sebastian Ruder, Madeline Smith, Antoine Bosselut, Alice Oh, Andre F. T. Martins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, Marzieh Fadaee, Beyza Ermis, Sara Hooker

arXiv:2412.03304v2cs.CL

TL;DR

Translated multilingual benchmarks can encode Western-centric cultural bias and translation artefacts, limiting fair global evaluation. This paper analyzes these effects and introduces Global MMLU, finding that model rankings change across culturally sensitive and culturally agnostic subsets.

  • Problem

    Translated English benchmarks such as MMLU lack comprehensive, culturally relevant coverage for evaluating multilingual models fairly across languages and cultures.

    Source: §1 · p. 2

  • Method

    The paper analyzes MMLU’s cultural biases and translation effects, then constructs Global MMLU across 42 languages with culturally sensitive and culturally agnostic subsets.

    Source: §3.1 · pp. 11–12, §6 · p. 24

  • Results

    Model rankings change across culturally sensitive and culturally agnostic subsets, while 28% of MMLU questions require culturally sensitive knowledge.

    Source: §6 · p. 24

  • Takeaways & Limitations

    Translated MMLU performance is insufficient as an indicator of multilingual model performance, motivating evaluation with culturally differentiated Global MMLU subsets.

    Source: §6 · p. 24

    Identifying cultural sensitivity does not fully resolve the lack of diverse, culturally grounded knowledge needed for inclusive and fair multilingual evaluation.

    Source: §7 · p. 25

Abstract

from arXiv · show

Cultural biases in multilingual datasets pose significant challenges for their effectiveness as global benchmarks. These biases stem not only from differences in language but also from the cultural knowledge required to interpret questions, reducing the practical utility of translated datasets like MMLU. Furthermore, translation often introduces artefacts that can distort the meaning or clarity of questions in the target language. A common practice in multilingual evaluation is to rely on machine-translated evaluation sets, but simply translating a dataset is insufficient to address these challenges. In this work, we trace the impact of both of these issues on multilingual evaluations and ensuing model performances. Our large-scale evaluation of state-of-the-art open and proprietary models illustrates that progress on MMLU depends heavily on learning Western-centric concepts, with 28% of all questions requiring culturally sensitive knowledge. Moreover, for questions requiring geographic knowledge, an astounding 84.9% focus on either North American or European regions. Rankings of model evaluations change depending on whether they are evaluated on the full portion or the subset of questions annotated as culturally sensitive, showing the distortion to model rankings when blindly relying on translated MMLU. We release Global MMLU, an improved MMLU with evaluation coverage across 42 languages -- with improved overall quality by engaging with compensated professional and community annotators to verify translation quality while also rigorously evaluating cultural biases present in the original dataset. This comprehensive Global MMLU set also includes designated subsets labeled as culturally sensitive and culturally agnostic to allow for more holistic, complete evaluation.

1 Introduction

The paper argues that translated English benchmarks can fail as global multilingual evaluations because they preserve Western-centric cultural bias and introduce translation artifacts. It introduces Global-MMLU, a professionally and community-verified multilingual benchmark with cultural-sensitivity annotations, and shows that cultural subsets and translation quality materially affect model evaluation.

1 Introduction

  • Blindly translating English MMLU risks Western-centric evaluation standards, overfitting to cultural biases, and degraded quality from translationese and automatic curation.MMLU includes US-specific subjects such as US History, US Accounting, and US Law.Source: §1 · p. 2
  • The study evaluates 14 open-weight and proprietary models from 9 families to measure how culturally sensitive and culturally agnostic questions affect multilingual model rankings.Questions requiring cultural, geographical, or dialect-specific knowledge are labeled Culturally-Sensitive, while others are Culturally-Agnostic.Source: §1 · p. 3
  • 28% of annotated MMLU questions require Western cultural knowledge, while 84.9% of geographically oriented questions concern North American or European regions.These findings show that MMLU progress depends heavily on Western-centric concepts.Source: §1 · p. 3

• Introducing Global-MMLU

  • Global-MMLU spans 42 languages and combines professional translations with post-edits, crowdsourced translations, machine translations, and extensive cultural metadata.The release also includes Global-MMLU Lite as a compact alternative for multilingual evaluation.Source: • Introducing Global-MMLU · pp. 3–4
  • CA evaluations averaged 3.4 rank changes and 3.7 position shifts, whereas CS evaluations averaged 5.7 rank changes and 7.3 position shifts across languages.These results indicate greater ranking variability for culturally sensitive subsets than for culturally agnostic subsets.Source: • Introducing Global-MMLU · p. 3
  • Human-translated datasets produce more reliable assessments than machine-translated datasets, particularly for low-resource languages where machine translation may obscure model capabilities.The paper recommends prioritizing Global-MMLU over translated MMLU and reporting culturally sensitive and culturally agnostic performance separately.Source: • Introducing Global-MMLU · pp. 3–4

2 Evaluating cultural bias in MMLU

Annotators reviewed a representative 2,850-question sample of MMLU to identify cultural, geographic, and dialect sensitivities, finding substantial and uneven cultural bias. The resulting CS and CA subsets reveal strong Western and subject-level skews in MMLU’s composition.

2.1 Data Annotation Process

  • Annotators labeled 2,850 randomly sampled questions from all 57 MMLU subjects, with each item reviewed by at least three annotators and 96.4% reviewed by more than three.Labels were assigned by majority vote, and dialect assessment targeted the original English sentences rather than translation-specific variation.Source: §2.1 · pp. 4–6

2.2 Analysis of MMLU Cultural Biases

  • 28% of MMLU questions require culturally sensitive knowledge, with geographic knowledge comprising 54.7% of CS questions, cultural knowledge 32.7%, and dialect knowledge 0.5%.Additionally, 10.6% require both cultural and geographic knowledge, while 1.5% involve all three types.Source: §2.2 · p. 6
  • 86.5% of culturally sensitive samples involve Western cultural knowledge, while 73.9% of Western-culture questions concern the United States.South Asian culture accounts for 4% of cultural tags, and Asian-culture questions predominantly concern India at 59%.Source: §2.2 · pp. 6–7, Figure 4 · p. 7
  • Cultural sensitivity varies sharply by subject: 68% of Humanities questions are CS, whereas only 30 of 950 STEM samples are CS.More than 80% of Philosophy, Moral Scenarios, High School US History, and High School Government and Politics samples are CS, while some medical and technical subjects are entirely CA.Source: §2.2 · p. 8
  • The annotation resulted in two aggregated annotated subsets, including CS and CA; Social Sciences rose from 21.1% in MMLU Annotated to 26.3% in CS, while STEM fell from 33.3% to 2.9%.Source: §2.2 · pp. 8–9

3 Introducing Global-MMLU

Global-MMLU improves multilingual MMLU by expanding coverage to 42 languages, using human-verified translation workflows, and enabling analysis of culturally sensitive and culturally agnostic subsets. It comprises 589,764 samples integrating human translations, machine translations, and the original English MMLU.

3.1 Translation Process

  • Google Translate achieved higher ChrF++ scores across all subjects and lower cross-language performance deviation than GPT-3.5-Turbo, after which native speakers reviewed translations.The workflow combined professional and community annotators, with a minimum of 50 human-translated samples required for each included community-translated language.Source: Figure 7 · p. 10, §3.1 · p. 11
  • The dataset incorporates professionally human-translated MMMLU content from 10 non-overlapping languages to maximize human-translated coverage.The incorporated languages are Bengali, Chinese, German, Indonesian, Italian, Japanese, Korean, Portuguese, Swahili, and Yoruba.Source: §3.1 · p. 12
  • Professional and community annotators made 7,565 edits, covering 36.9% of reviewed samples, while edit rates were not interpreted as translation-quality differences across languages.Professional annotators edited an average of 789 samples per language in the Gold Set, compared with 362 samples per language for community contributors.Source: §3.1 · p. 12

3.2 Data Composition of Global-MMLU

  • Global-MMLU spans 14K MMLU samples across 42 languages, totaling 589,764 samples from human-translated datasets, machine translations, and original English data.The benchmark supports reporting on subsets of Global-MMLU for model evaluation.Source: §3.2 · p. 12
  • Global-MMLU propagates English annotations for cultural, geographic, dialectal, and temporal knowledge requirements across 41 other languages, creating a 119,700-sample annotated subset.Its culturally sensitive subset contains 33,264 entries, while the subset without cultural, regional, or dialectal references contains 86,436 entries.Source: §3.2 · pp. 12–13

4 Model Evaluations

This section examines how cultural sensitivity, language-resource availability, and translation quality affect multilingual model performance and rankings across models and language sets. It highlights substantial variability and ranking differences associated with dataset and language characteristics, particularly for culturally sensitive and low-resource evaluations.

4 Model Evaluations

  • Evaluations compare Global-MMLU Annotated, Culturally-Sensitive, and Culturally-Agnostic subsets to assess performance with and without culturally specific contextual knowledge.The analysis focuses on human-translated data across 14 languages and also examines model behavior across language-resource levels.Source: §4 · p. 13, §4.2 · pp. 14–15

4.1 Experimental Setup

  • The study evaluates 14 state-of-the-art models from 9 families using 5-shot prompting, with separate handling for closed models because API log probabilities are unavailable.Results are reported for human-translated languages and broader language-resource groups spanning high-, mid-, and low-resource languages.Source: §4.1 · pp. 13–14

4.2 Results

  • Closed-source models such as GPT-4o and Claude 3.5 Sonnet consistently outperform smaller open-source models, while their performance gap narrows on culturally sensitive data.Direct proprietary-versus-open-weight comparisons remain limited by differences in model size and evaluation methods.Source: §4.2 · p. 14
  • Models generally perform better on culturally agnostic data, whereas culturally sensitive data shows higher cross-language variance because it requires contextual understanding and is more affected by translation and cultural biases.Culturally sensitive tasks are concentrated in Social Sciences and Humanities, while culturally agnostic data includes more challenging Medical and STEM categories, which contributes to the observed accuracy pattern.Source: §4.2 · p. 15
  • High-resource languages achieve the highest average accuracy, while performance declines and variability increases substantially for low-resource languages, especially on culturally sensitive data.Limited high-quality training data hinders generalization, and culturally sensitive tasks amplify sensitivity to regional and dialectal references.Source: §4.2 · pp. 15–17
  • Culturally sensitive evaluations alter model rankings more than culturally agnostic evaluations, with high-resource languages averaging 3.3 versus 6.8 rank changes and reaching 9.1 position shifts on culturally sensitive data.Chinese and Hindi are especially sensitive to culture-specific knowledge, while Italian, Japanese, and Portuguese show no rank changes on culturally agnostic data.Source: §4.2 · p. 17
  • Low-resource languages amplify ranking instability, including rank shifts of up to 5 positions for Malagasy and 13 ranking changes for models on Ukrainian.Resource level therefore affects ranking differences even within culturally agnostic evaluations.Source: §4.2 · p. 17
  • Large models are highly consistent across datasets and resource levels, with average rank changes of 0.21 on culturally agnostic data, while mid-size models show greater variability, especially on culturally sensitive data.The maximum position shift is 3 for large models and 5 for small models.Source: §4.2 · pp. 18–19
  • Small models show the smallest ranking changes, but this apparent stability reflects weaker performance across both datasets rather than robustness.Small models average 51.3% accuracy on CA, compared with 59.1% and 61.7% for mid-size models and 61.6% and 66.8% for large models.Source: §4.2 · p. 19
  • Dataset characteristics significantly affect model performance across all model sizes, with models remaining sensitive to cultural and linguistic nuances in culturally sensitive datasets.The magnitude of performance variability differs by model size, but the pattern is consistent across groups.Source: §4.2 · p. 19
  • Models generally perform better on human-translated data for high-resource languages, while the HT–MT gap narrows for some mid-resource-language models.Claude Sonnet and Qwen2.5 32B show narrower gaps, whereas CommandR+ and Aya Expanse 32B continue to favor human-translated data, consistent with stronger in-language training support.Source: §4.2 · p. 19
  • For low-resource languages, several models perform better on machine-translated data, whereas Aya Expanse 32B is the only model with consistent performance across HT and MT datasets.This pattern likely reflects training-data alignment with machine-translated test distributions; Aya Expanse 32B’s consistency is attributed to broad low-resource-language coverage.Source: §4.2 · p. 20

5 Related Work

Prior work has expanded multilingual and culturally aware evaluation through translated, language-specific, and cross-cultural benchmarks, while participatory projects have broadened multilingual data collection. However, existing resources often have limited language coverage, lack standardized cross-language comparison, or focus on modalities and tasks distinct from text evaluation.

5.1 Multilingual Knowledge Evaluation

  • Multilingual MMLU extensions and language-specific exam benchmarks broaden evaluation, but many remain English-only, single-language, limited in language coverage, or lack standardized cross-language comparison.INCLUDE is a notable exception, covering 44 languages, while translated MMLU variants span 14–26 languages and differ in translation reliability.Source: §5.1 · pp. 21–22

5.2 Culturally-aware Evaluation

  • Culturally aware evaluation studies examine cross-cultural values, everyday knowledge, multimodal benchmarks, and translation or training effects, including persistent Western-aligned model values and stronger performance from native-instruction tuning.SEA-HELM emphasizes Southeast Asian languages with manually crafted, translated, and validated tasks; related visual benchmarks cover 31–100 languages, while PangeaBench covers 47 languages.Source: §5.2 · p. 22

5.3 Participatory Open Science Projects

  • Participatory open-science efforts engage native speakers, linguists, students, and global communities to build culturally grounded resources, but prior projects often target image or speech data or remain small and region-specific.MaRVL collected culturally reflective images in five languages but contained fewer than 8,000 data points, whereas the Aya Initiative involved 3,000 contributors across 114 languages.Source: §5.3 · pp. 23–24

6 Conclusion

The paper finds that cultural bias and translation artifacts limit MMLU’s effectiveness as a global multilingual benchmark. Global-MMLU addresses these issues by separating culturally sensitive and agnostic knowledge for more holistic model evaluation.

Conclusion

  • 28% of MMLU questions require culturally sensitive knowledge, with progress depending heavily on Western-centric concepts and geographic questions focusing largely on North America and Europe.These biases persist in translated MMLU variants, risking over-indexed evaluations of Western-centric idioms and knowledge.Source: §6 · p. 24
  • Global-MMLU introduces multilingual, multidomain datasets that distinguish culturally sensitive from culturally agnostic knowledge using professional and crowd-sourced annotations.These subsets enable more rigorous multilingual model evaluation.Source: §6 · p. 24
  • Model rankings change between culturally sensitive and culturally agnostic subsets, showing that translated MMLU progress is insufficient as a multilingual performance indicator.The authors recommend reporting Global-MMLU subset results as part of holistic multilingual LLM evaluations.Source: §6 · p. 24
  • The authors release Global-MMLU under a fully permissive license for use in multilingual evaluations.The release includes Global-MMLU and Global-MMLU-Lite resources.Source: §6 · p. 24

7 Limitations … B Global-MMLU

The paper identifies limitations in Global-MMLU’s language, dialect, annotation, safety, geographic, and cultural coverage, while documenting its language-resource taxonomy and six subject categories. These limitations show that broader language coverage and culturally grounded knowledge remain necessary for inclusive multilingual evaluation.

7 Limitations

  • Community-annotator participation is uneven across languages, with many contributors providing only one or two annotations and some languages dominated by frequent contributors.This may produce uneven dataset distributions and limited annotator diversity across languages.Source: §7 · p. 24
  • Global-MMLU covers 42 languages, but this represents only a tiny fraction of the world’s linguistic diversity and requires expansion beyond the current set.Of approximately 7,000 world languages, only about half have written forms and only a few hundred appear in machine-readable internet corpora.Source: §7 · p. 24
  • Many dialects and creoles associated with the represented parent languages are absent, and the evaluation does not address how technology serves different dialects.Dialect variation can support cultural identity, making its omission a meaningful coverage limitation.Source: §7 · p. 25
  • Global-MMLU lacks dedicated annotation flags for toxic, harmful, or offensive speech, so potentially harmful data may remain unidentified.The authors consider the risk relatively low because the source emphasizes examination material, but they did not explicitly monitor or track this content.Source: §7 · p. 25
  • The geographic-sensitivity scheme uses six regions, but the authors recommend the World Bank’s more granular taxonomy, including separate Central America and Sub-Saharan Africa designations.The recommendation follows discussions about improving region assignments.Source: §7 · p. 25
  • Identifying cultural sensitivity or cultural agnosticism does not itself guarantee cultural inclusion, because translation can create an illusion of inclusivity without integrating diverse culturally grounded knowledge.Future work must prioritize culturally grounded knowledge for fairer multilingual evaluation.Source: §7 · p. 25

A Global-MMLU Languages

  • Global-MMLU groups languages into lower-, mid-, and higher-resourced categories using written and catalogued NLP resources, while acknowledging that this single-dimensional taxonomy is imperfect.The grouping supports aggregation in the dataset-distribution analysis rather than universal classification of languages or varieties.Source: A Global-MMLU Languages · p. 37, Table 5 · p. 38

B Global-MMLU

  • Global-MMLU organizes subjects into STEM, Humanities, Social Sciences, Medical, Business, and Other, refining MMLU’s original Other category.Medical includes clinical and medicine-related subjects; Business includes ethics, management, marketing, and accounting; Other retains global facts and miscellaneous.Source: Subject Categories · pp. 38–39

C Global-MMLU Lite

Global-MMLU Lite is a lighter multilingual evaluation set designed for efficient evaluation across languages while balancing culturally sensitive (CS) and culturally agnostic (CA) coverage across subjects.

Dataset distribution

  • Figure 13 shows the resulting distribution of samples across subject categories in Global-MMLU Lite.Social Science and Humanities subjects are more prevalent after excluding subjects lacking both CS and CA representation.Source: Figure 13 · p. 39, C Global-MMLU Lite · p. 39

Sampling strategy

  • The dataset uses 50 samples per subject and excludes subjects tagged exclusively as CS so both CS and CA categories are represented.This selection makes Social Sciences and Humanities more prevalent in the resulting subject distribution.Source: C Global-MMLU Lite · p. 39
  • Sampling targets five CS samples per subject where available, with smaller CS/CA selections for subjects having limited CS coverage.Business and Medical subjects were slightly upsampled to maintain adequate representation.Source: C Global-MMLU Lite · p. 39
  • The General Knowledge category is upsampled to 22 Miscellaneous and 8 Global Facts samples per category to ensure sufficient general-knowledge coverage.The category contains only Miscellaneous and Global Facts.Source: C Global-MMLU Lite · p. 39

Dataset design

  • Global-MMLU Lite provides a balanced dataset for efficient multilingual evaluation across multiple languages.It samples across 15 human-translated or post-edited languages, including English.Source: C Global-MMLU Lite · p. 39

D Temporal Knowledge · E Models Covered

The annotation identifies a small temporal-knowledge subset, concentrated outside STEM. The evaluation covers multilingual open and closed models spanning different parameter sizes and language supports.

D Temporal Knowledge

  • 2.4% of MMLU Annotated samples are time-sensitive, concentrated in Social Sciences, Humanities, Medical, and Other categories, while STEM has none.Time-sensitive answers can change with current political leaders or economic statistics.Source: D Temporal Knowledge · p. 40

E Models Covered

  • Aya Expanse evaluates 8B and 32B models supporting 23 languages, building on the Aya multilingual model and dataset initiatives.The supported languages include Arabic, Chinese, Czech, Dutch, English, French, German, Greek, Hebrew, Hindi, Indonesian, Italian, Japanese, Korean, Persian, Polish, Portuguese, Romanian, Russian, Spanish, Turkish, Ukrainian, and Vietnamese.Source: E Models Covered · p. 40
  • Command R and R+ are 34B and 104B open-weight models supporting 10 languages, evaluated using their 08-2024 versions.The evaluated models are Command-R 08-2024 and Command-R+ 08-2024.Source: E Models Covered · p. 40
  • Gemma2 contributes instruct-tuned 9B and 27B variants, while SEA-LION contributes a Gemma2-9B instruct model trained for Southeast Asian languages.Gemma2’s targeted languages are not explicitly reported; SEA-LION includes Burmese, Chinese, English, Filipino, Indonesian, Javanese, Khmer, Lao, Malay, Sundanese, Tamil, Thai, and Vietnamese.Source: E Models Covered · p. 41
  • Llama 3.1 offers 8B, 70B, and 405B sizes with eight supported languages, and evaluation uses its 8B- and 70B-Instruct variants.Supported languages include English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai.Source: E Models Covered · p. 41
  • Mistral Nemo is a 12B model supporting 11 languages, while Qwen 2.5 supports up to 29 languages and is evaluated in 7B- and 32B-Instruct variants.Mistral Nemo includes English, French, German, Spanish, Italian, Portuguese, Chinese, Japanese, Korean, Arabic, and Hindi; Qwen’s listed languages include Chinese, English, French, Spanish, and Portuguese.Source: E Models Covered · p. 41
  • GPT-4o is a multilingual, multimodal closed model whose targeted languages are not explicitly reported.It is part of the GPT-4 family and is described by Hurst et al. (2024).Source: E Models Covered · p. 41
  • Claude Sonnet 3.5 is a multilingual, multimodal closed model from the Claude 3.5 family, but its supported languages are unknown.The model’s language coverage is not specified in the supplied description.Source: E Models Covered · p. 41

F Additional Results … G.1 Culture–Region Relations

Additional results show that evaluation rankings shift across language-resource groups, subject performance varies around Aya Expanse 32B’s average, and cultural tags strongly align with geographic regions. These findings provide supplementary evidence on ranking sensitivity, subject-level variation, and culture–region associations.

F.1 Model Rank Changes

  • Tables 6 and 7 report model rank and position changes relative to MA for high-, mid-, and low-resource languages.Their rightmost columns summarize the total number of models changing rank and the total number of ranking position shifts.Source: §F.1 · p. 41, Table 6 · p. 42, Table 7 · p. 43
  • The tables distinguish changes in CA and CS ranks, using color-coded arrows to indicate increases and decreases relative to MA.Source: §F.1 · pp. 42–43

F.2 Subject-level Performance

  • Aya Expanse 32B achieves 66.4% average accuracy across subjects.Source: §F.2 · p. 44, Figure 15 · p. 44
  • Most STEM subjects fall below Aya Expanse 32B’s average, while most Social Sciences and Humanities subjects exceed it.Source: §F.2 · p. 44

G.1 Culture–Region Relations

  • Among samples tagged with Western culture, 73.3% are associated with North America and 25.5% with Europe.Source: §G.1 · p. 44
  • Among samples tagged with Asian cultures, 97.2% are associated with the Asia region.Source: §G.1 · p. 44

G.2 Culture Country Relations · G.3 Region Country Relations

The culture–country relations show concentrated associations for Indigenous and Other cultures but a more balanced distribution for Latin American culture. Region–country relations are highly concentrated in North America, more balanced in Europe, and evenly split between Egypt and South Africa in Africa.

G.2 Culture Country Relations

  • Indigenous culture tags are concentrated in the USA at 66.7%, with Micronesia accounting for the remaining 33.3%.The tags are shared between two countries.Source: §G.2 · p. 44
  • The Other culture category captures cultures outside the pre-existing categories, and every Other sample falls under Russia.Figure 17 focuses on culture–country tag relationships, including Latin American, Indigenous, and Other culture categories.Source: §G.2 · p. 44, Figure 17 · p. 45

G.3 Region Country Relations

  • 89.6% of North American regional tags are associated with the United States, while Canada and the United Kingdom each account for 0.8%.This pattern is presented in Figures 18 and 19.Source: §G.3 · p. 45
  • In Africa, Egypt and South Africa share the top position at 33.3% of regional tags each.The Africa distribution is described as even more balanced than the North America distribution.Source: §G.3 · p. 45
  • Europe has a more balanced country distribution, led by the United Kingdom at 20.1% and France at 10.1% of tags.The comparison covers country-specific information for Europe.Source: §G.3 · p. 45

H Annotation Process · H.1 Compensated Annotator Pool for Gold Standard Languages · H.2 Agreement between Annotators

Global MMLU used compensated, language-proficient annotators and structured annotation workflows to assess cultural sensitivity and translation quality. Agreement was generally reasonable, although moral scenarios showed disagreement, while 12 subjects achieved unanimous time-sensitivity agreement.

H Annotation Process

  • Authors briefed annotators, supported questions through Discord, and used shared error patterns, comments, and discussion of ambiguous cases to calibrate ratings across languages.Annotators were encouraged to record difficult decisions and their rationales.Source: H Annotation Process · pp. 45–46
  • Annotation ran in flexible 2–3 week community-contributor sprints and used an Argilla interface deployed on Hugging Face Spaces with single sign-on.Contributors worked varying hours according to availability and speed.Source: H Annotation Process · pp. 46–47
  • 100% of selected cultural-sensitivity samples were labeled, while 37% of translation samples were fully reviewed and 12.3% were edited.These completion rates describe the two annotation tasks’ review coverage.Source: H Annotation Process · p. 46
  • Cultural-sensitivity annotators assessed questions from all 57 MMLU subjects for cultural, geographic, dialect, or regional knowledge.The task presented questions individually for labeling.Source: H Annotation Process · p. 47, Figure 20 · p. 47
  • Translation annotators compared translated questions and options with English originals, accepting adequate translations or editing inadequate ones.The interface displayed both the target-language content and the English reference.Source: H Annotation Process · p. 47, Figure 21 · p. 48

H.1 Compensated Annotator Pool for Gold Standard Languages

  • The compensated pool recruited people with self-reported native or professional proficiency in required languages and included diverse socioeconomic, career, educational, gender, and sexual-identity backgrounds.The project did not formally collect these demographic statistics.Source: §H.1 · p. 48

H.2 Agreement between Annotators

  • Each cultural-sensitivity sample received at least three annotations, which were aggregated by subject and evaluated using Krippendorff’s Alpha.Agreement scores were reported for cultural or regional knowledge and time-sensitive content.Source: §H.2 · p. 48, Figure 23 · p. 49, Figure 24 · p. 50
  • 12 subjects achieved complete unanimous agreement on time-sensitivity annotations, while moral scenarios showed reasonable disagreement across cultural-sensitivity and time-sensitivity tasks.Krippendorff’s Alpha ranges from -1 for opposite ratings to 1 for unanimous agreement.Source: §H.2 · p. 49, Figure 24 · p. 50

I Translation Analysis … L MMLU Subject Name Mapping

The translation analysis finds that Google Translate outperforms GPT-3.5-turbo for multilingual MMLU translation, while edit patterns vary substantially across subject categories. The appendices further document representative MMLU questions, annotation guidance for cultural and geographic knowledge, and standardized subject-name mappings.

I.1 Translation Quality

  • Google Translate is significantly better than GPT-3.5-turbo across MMLU subject categories in the translation-quality comparison.The analysis uses MMMLU samples as human references and considers languages overlapping across the machine- and human-translated sets.Source: §I.1 · p. 50

I.2 Translation Edits

  • Humanities questions exhibit the largest raw edit distances, with greater edits for questions than answers.Edit distance is computed by comparing machine translations with versions edited by professional and community annotators using Levenshtein distance.Source: §I.2 · p. 50, Figure 25 · p. 51
  • Normalized edit distance shows that STEM answers have the highest edit values, while Humanities questions are longest, separating edit complexity from text length.The analysis divides edit distance by text length and reports category-level question and answer lengths alongside normalized scores.Source: §I.2 · p. 50, Figure 26 · p. 51

J MMLU Annotated Examples

  • The annotated examples span questions in US history, accounting, jurisprudence, prehistory, econometrics, statistics, formal logic, and US foreign policy.Examples include culturally and geographically situated questions as well as technical subject-matter items with multiple-choice answers.Source: J MMLU Annotated Examples · pp. 52–53

K Examples of Cultural, Geographical and Dialect Knowledge

  • Annotation guidance distinguishes region-specific knowledge, such as landmarks, historical events, measurement systems, laws, and group behaviors, from globally applicable knowledge.Examples of culturally agnostic knowledge include global climate patterns, celestial bodies, and standardized geography terms.Source: K Examples of Cultural, Geographical… · pp. 54–55

L MMLU Subject Name Mapping

  • The subject-name mapping assigns concise labels to MMLU subjects, including abbreviated forms for biology, chemistry, computer science, history, mathematics, and government.Table 10 identifies these short names as the labels used for MMLU subjects in the paper’s figures.Source: L MMLU Subject Name Mapping · p. 56, Table 10 · p. 57

Related papers

Loading 2412.03304v2…