The State and Fate of Multilingual, Contextual Evaluation in the NLP World

1 Introduction

• Multilingual evaluation matters because English-centric system design can leave Global South languages and local contexts inadequately assessed.The passage links English-centric data, task design, and success criteria with risks of exclusion, stereotyping, or high-stakes failures.Source: §1 · p. 1
• Multilingual evaluation should capture the local contexts and cultural nuances that shape quality, usefulness, and safety, not merely measure English-defined tasks in another language.The passage also notes that the New Delhi Frontier AI Commitments call for effectiveness across languages, cultures, and real-world use cases.Source: §1 · pp. 1–2
• Existing multilingual efforts often extend English benchmarks rather than grounding evaluation in diverse communities’ linguistic, cultural, and practical realities.The paper frames its audit as a reckoning of evaluation growth and gaps, organized around coverage, representativeness, and rigor.Source: §1 · p. 2

Multilingual Benchmarks and Language Coverage

• Cross-lingual benchmark expansion scaled English-dataset translation, but coverage remains skewed toward high-resource languages and structural English-centrism persists.The cited work reports that low-resource languages lag and that multilingual models may use English as an internal pivot.Source: §2 · p. 2

Representativeness: Translation Artifacts and Cultural Grounding

• Translation can introduce translationese and English-centric assumptions, while language-specific and community-driven benchmarks offer alternatives but remain limited in community representation and scope.TyDiQA elicits questions directly in target languages; Pariksha and Samiksha incorporate stakeholder input, and TUMLU prioritizes sociolinguistic authenticity.Source: §2 · pp. 2–3

Evaluation Rigor and Data Contamination

• Translated benchmark variants and parallel corpora create cross-lingual leakage risks that surface-form overlap methods may miss.The passage also notes that contamination detection trained on one language may not generalize to linguistically dissimilar languages.Source: §2 · p. 3

3.1 Coverage

• Across 242 datasets covering 219 languages, 48 languages appear in only one dataset, leaving no independent replication of evaluation results or opportunity for cross-dataset comparison.Hindi, Chinese, Spanish, Swahili, and Arabic each appear in 19 or more datasets.Source: §3.1 · p. 4
• Indo-European languages make up 83 of 219 languages (38%), while Oceania and South America have very poor or no representation.Ge’ez, Khmer, and Myanmar scripts appear in only 1–3 benchmarks; script-specific failure modes may remain invisible when evaluation concentrates on Latin and Devanagari.Source: §3.1 · p. 4, Figure 2 · p. 5
• 160 low-resource languages are typically tested on only 1–3 tasks, versus a median of 14 categories for 10 Winner languages; safety evaluation is effectively absent for low-resource languages.Left-Behinds are concentrated on Translation (29%) and Sentiment/Classification (25%), while Coding, Safety/Toxicity, and Instruction Following each account for 1%.Source: §3.1 · p. 5

3.2 Representativeness

• 56% of dataset–language instances are translated from English, compared with 36% natively authored and 8% partially adapted.Translation scales efficiently but carries translationese artifacts, English-centric framing, and inherited contamination risk.Source: §3.2 · p. 5
• Community-driven datasets in Sub-Saharan Africa and South Asia have the highest cultural grounding rates, while broad-coverage suites score lower.The benchmarks strongest on representativeness tend to have the narrowest language coverage; only 8 of 51 benchmarks report annotator demographics, and 17 disclose compensation.Source: §3.2 · pp. 6–7, Figure 4 · p. 7

3.3.1 Metrics

• English-centric reference metrics can misjudge valid responses in morphologically rich or non-standard-spelling languages, while LLM judges often align poorly with human judgments in multilingual settings.The cited prior work particularly flags low-resource and culturally grounded, context-sensitive evaluations.Source: §3.3.1 · p. 7

3.3.2 Contamination

• STAMP flags contamination signals in older datasets, including 67.9% of PAWS-X and 45.5% of MGSM model–language pairs, while newer MILU, INCLUDE, and Global MMLU show 0.0%, 1.0%, and 1.7%.STAMP and surface-form methods diverge sharply for Flores (1.2% flagged by STAMP versus 7 of 7 models flagged by prior methods); the authors conclude no single detection approach suffices. Both STAMP and DCQ have language-dependent measurement limitations, and DCQ dataset-name cues can confound recognition.Source: §3.3.2 · pp. 8–9

3.3.3 Reporting Practices in Model Releases

• 87% of 23 recent model releases provide no contamination analysis, and 35% report zero multilingual datasets.Only 3 models report any contamination testing, and even these leave cross-lingual pathways unexamined; multilingual safety evaluation is reported by only 4 of 23 models.Source: §3.3.3 · p. 9

4 Discussion

• 36% of languages appear in only a single dataset, while entire world regions are absent and multilingual safety datasets offer little low-resource coverage.The broadest-coverage datasets are also the most translated and least culturally grounded; the most valid datasets cover the fewest languages.Source: §4 · p. 10
• Translation, English-centric metrics, and limited reporting in model releases undermine rigor, alongside rarely reported annotator demographics and researcher-led data creation.These patterns raise concerns about whose values and priorities multilingual datasets reflect.Source: §4 · p. 10
• Meaningful evaluation requires evidence of usefulness in local linguistic and cultural contexts, with communities helping define tasks, rubrics, and failure modes.The paper also calls for reporting what was evaluated, how data was created or adapted, which contexts are covered, and what claims results support.Source: §4 · p. 10
• Scaling across many languages often relies on standardization, while representativeness requires engagement with local variation, context, and heterogeneity.The authors argue that some variation must be expected when taking linguistic, cultural, and contextual diversity seriously.Source: §4 · p. 10

A.1 Data Extraction Methodology

• The pipeline surveys 51 benchmarks and extracts constituent datasets, languages, translation status, native-speaker annotation involvement, and cultural grounding.Extraction follows a benchmark → dataset → language hierarchy.Source: §A.1 · p. 21
• Language metadata is enriched with family, macroarea, primary script, and resource classification, while duplicate dataset-language entries are deduplicated and benchmark provenance retained.Resource classification uses Joshi et al. (2020)’s taxonomy.Source: §A.1 · p. 21
• Extracted property assessments include confidence ratings and evidence passages, then reviewers inspect, verify, or correct values through a web interface.Datasets with missing or uncertain property values, such as unknown native-annotator status, are flagged for mandatory review.Source: §A.1 · p. 21

A.2 Geographic Distribution

• 2 languages (1%) are covered in Oceania, a region with over 1,400 living languages.The authors describe Oceania as remaining near-zero in the evaluation perimeter.Source: §A.2 · p. 21
• 62 languages (28%) are covered in Sub-Saharan Africa, followed by 45 (21%) in Europe and 43 (20%) in South Asia.Sub-Saharan Africa's lead is attributed to dedicated datasets.Source: §A.2 · p. 21
• The Americas cover 14 languages and Central Asia 9, which the authors characterize as modest representation gains from recent datasets.Source: §A.2 · p. 21

A.3 Resource-Level Distribution

• Figure 7 presents the resource-level composition of all evaluated languages.Source: §A.3 · p. 21
• Figure 7 complements the task-equity analysis in section 3.2, shown in Figure 3.Source: §A.3 · p. 21
• The human-review interface displays extracted properties, evidence passages, and confidence ratings alongside dropdowns for reviewer adjudication.Reviewers adjudicate translation status, native-annotator involvement, and cultural grounding.Source: Figure 5 · p. 22

A.4 Dataset Source Type Breakdown

• 128 datasets (53%) are originally authored, while 85 (35%) are translated from English; 15 (6%) are adapted and 14 (6%) mix methods.Over a third of evaluation data originates in English, carrying risks of translationese artifacts and cultural misalignment.Source: §A.4 · pp. 22–23
• 65% of evaluated languages fall in the two lowest resource classes (levels 0–1), yet evaluation depth remains concentrated in high-resource languages.Source: Figure 7 · p. 23
• QA/Knowledge dominates task coverage across regions, while safety, reasoning, and coding are unevenly distributed and Central Asia and MENA are thinnest.Source: Figure 8 · p. 23

A.5 Translation Status by Resource Level

• Left-Behinds (n = 178) have ∼20% native data, versus ∼40% for Winners (n = 360), and both rely heavily on translated data.Across resource levels, roughly 35–40% of dataset–language instances involve native data.Source: §A.5 · p. 23

A.6 Native Annotators by Region

• Central and South Asia have the highest native-annotator share (∼50%); Sub-Saharan Africa reports ∼45% but has a large unknown-provenance fraction, while the Americas have the lowest share.The figure summary also identifies the Americas as lagging and annotator provenance as under-documented.Source: §A.6 · pp. 23–24, Figure 11 · p. 25
• Across 242 datasets, 35% are translated from English and 53% are originally authored in target languages.Source: Figure 9 · p. 24
• Left-Behinds have the lowest native-data share (∼20%) and are predominantly evaluated through translated datasets.Source: Figure 10 · p. 24

A.7 Model Release Transparency Audit

• Only 65% of releases report any multilingual evaluation, and just 13% report contamination results; closed models disclose less across all dimensions.Source: Figure 12 · p. 25
• The release audit scores 23 model releases from 2024–2026 on five binary disclosure dimensions, summing them into a 0–5 composite transparency score.The dimensions cover training-language composition, multilingual evaluation and its languages, and contamination detection and results.Source: §A.7 · p. 25

• For releases including GPT, Grok, Mistral 3, BharatGen, Command-R+, OLMo, and Kimi K2.5, publicly reported multilingual evaluation evidence is sparse.Sarvam's reported evaluation appears centered on a newly introduced dataset scored by an LLM judge, without results on other widely used multilingual datasets.Source: Detailed Findings · pp. 25–26
• Contamination testing is reported by only 3 models, all open-weight, and their methods target English benchmarks rather than cross-lingual leakage pathways.Translated dataset leakage, parallel-corpus overlap, and instruction-tuning propagation go unexamined.Source: Detailed Findings · p. 26
• INCLUDE, FLORES-200, and MGSM each appear in only 4 of 23 reports, while several natively authored, culturally grounded datasets are almost never cited.Source: Detailed Findings · p. 26
• Open-weight models average 2.47/5 on transparency versus 0.75/5 for closed models, yet even the most transparent open-weight models rarely report multilingual contamination testing.Source: Detailed Findings · p. 26
• Only 4 of 23 models report multilingual safety evaluation; other models test safety only in English or do not report safety testing.Source: Detailed Findings · p. 26

B Contamination Detection Results

• Table 2 records contamination presence across evaluated models using contaminated, not-contaminated, and not-evaluated symbols.Source: Table 2 · p. 27
• Table 3 compares original and paraphrased benchmark variants in popular open-source LLMs, with the difference row giving the generalizability gap.Bold values denote significantly lower generalizability, implying potential contamination.Source: Table 3 · p. 27
• Table 4 reports per-model, per-dataset contamination rates as the fraction of languages flagged, with counts in parentheses.Bold indicates whole-dataset contamination by Fisher’s combined p < 0.05.Source: Table 4 · p. 27

B.1 Data Contamination Quiz Results

• The DCQ was applied to 85 dataset–language partitions with 100 instances per partition, and its rates are recognition rates rather than contamination estimates.The five-option quiz has a 20% random-chance floor.Source: §B.1 · p. 27
• 9.7 percentage points mean absolute difference and a 23-point maximum across models on byte-identical GSM8K replicates make individual DCQ values approximate.The replicate used the AfriMGSM and MGSM dataset names.Source: §B.1 · p. 27
• Eight partitions with perturbation distance below 10% have nearly unsolvable quizzes, making their low scores uninformative.Bias-detection abstention is near zero for most models, indicating frequent guessing.Source: §B.1 · p. 27

C Released Resources and Interactive Dashboard

• The planned releases include structured metadata for 51 benchmarks and 242 datasets, the analysis codebase, and a living dashboard designed for updates as datasets are added.Metadata includes language script, family, resource level, task categories, translation status, annotator provenance, and cultural grounding.Source: C Released Resources and Interactive… · p. 27, Model AfriMGSM Global MMLU INCLUDE… · p. 29

B.1 Data Contamination Quiz Results

• Table 5 reports maximum recognition rates averaged over each benchmark’s language partitions; Table 6 compares English with non-English means across seven models, omitting three benchmarks without English partitions.Table 5’s maximum selects the best raw hit rate across tested answer positions, making it optimistic.Source: §B.1 · p. 27, Table 5 · p. 28, Table 6 · p. 28

Model AfriMGSM Global MMLU INCLUDE IndicParam MGSM MILU MMMLU PAWS-X

• The dashboard offers coverage and representativeness views, a benchmark explorer, and language lookup, with filters and per-language dataset details.Users can search datasets by language and inspect translation status and task types.Source: Model AfriMGSM Global MMLU INCLUDE… · p. 29, Figure 13 · p. 29
• Figure 14 drills from a language long-tail chart to matching benchmarks, while Figure 15 expands benchmark rows with references, descriptions, and language-family breakdowns.The explorer supports search, sorting, and regional filtering.Source: Figure 14 · p. 30, Figure 15 · p. 30