Source-linked AI summary
MEGA: Multilingual Evaluation of Generative AI
Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, Sunayana Sitaram
TL;DR
Generative LLMs have been evaluated predominantly in English, leaving their capabilities across diverse languages insufficiently understood. MEGA benchmarks these models across 16 datasets and 70 languages against multilingual baselines, finding persistent disadvantages for low-resource languages and only partially effective mitigation through translate-test prompting.
Problem
Large-scale evidence about generative LLM performance across diverse languages is limited because existing evaluations are largely English-centered.
Method
MEGA evaluates four generative LLMs across 16 datasets and 70 languages, compares them with multilingual baselines, and tests prompting strategies and language-related performance factors.
Results
Performance gaps consistently separate high-resource Latin-script languages from under-resourced languages; translate-test often helps low-resource languages, but GPT-4 can remain better with monolingual prompting on some cases.
Takeaways & Limitations
MEGA provides a framework and code for broader multilingual benchmarking while highlighting the need for automatic and human evaluation across more languages.
Takeaways & Limitations
The study does not evaluate all multilingual datasets or several closed models, and existing benchmarks underrepresent many typologically diverse and under-resourced languages.
Abstract
from arXiv · showhide
Generative AI models have shown impressive performance on many Natural Language Processing tasks such as language understanding, reasoning, and language generation. An important question being asked by the AI community today is about the capabilities and limits of these models, and it is clear that evaluating generative AI is very challenging. Most studies on generative LLMs have been restricted to English and it is unclear how capable these models are at understanding and generating text in other languages. We present the first comprehensive benchmarking of generative LLMs - MEGA, which evaluates models on standard NLP benchmarks, covering 16 NLP datasets across 70 typologically diverse languages. We compare the performance of generative LLMs including Chat-GPT and GPT-4 to State of the Art (SOTA) non-autoregressive models on these tasks to determine how well generative models perform compared to the previous generation of LLMs. We present a thorough analysis of the performance of models across languages and tasks and discuss challenges in improving the performance of generative LLMs on low-resource languages. We create a framework for evaluating generative LLMs in the multilingual setting and provide directions for future progress in the field.
1 Introduction
MEGA addresses the limited multilingual evaluation of generative LLMs by benchmarking models across diverse languages and tasks, comparing them with fine-tuned multilingual systems. The study examines performance disparities, prompting strategies, and challenges for low-resource languages.
- Motivation: Existing LLM evaluations largely use English data, leaving multilingual capabilities across diverse languages insufficiently measured.This gap matters because prior-generation models showed documented performance inequalities across languages.
- Contribution: The benchmark compares generative LLMs with fine-tuned state-of-the-art multilingual models to assess their relative performance.The comparison includes models such as TULRv6 and MuRIL.
- Research questions: MEGA investigates which languages models handle well, how performance trends vary across languages, and which prompting strategies are effective.These aims are framed as three research questions concerning multilingual benchmarks, cross-language trends, and prompting.
- Headline findings: The study reports a significant English–non-English performance disparity, especially for low-resource languages with non-Latin scripts.It also reports that fine-tuned models perform significantly better for these languages, while GPT-4 narrows but does not eliminate the discrepancy.
- Implications: The work provides strategies for building multilingual generative-AI systems and releases code to support broader evaluation.The paper positions these resources as a blueprint for multilingual users and community-scale evaluation.
2 MEGA
MEGA evaluates multilingual LLM capabilities using diverse NLP datasets, languages, models, baselines, and prompting configurations. Its methodology combines in-context learning and instruction following while systematically varying prompt design and evaluation settings.
- Datasets and tasks: The benchmark covers five NLP task families across 16 datasets, including classification, question answering, and sequence labeling.The datasets include multilingual, code-mixed, reasoning, sentiment, QA, and named-entity-recognition tasks.
- Datasets and languages: The 16 datasets span 70 languages and 21 language families, with evaluations using test subsets or validation data when public test sets are unavailable.For UDPOS, PAN-X, and XL-Sum, evaluations use the first 1000 test examples; several QA datasets use validation data.
- Models: The study compares GPT-3.5, GPT-4, and BLOOMZ with fine-tuned baselines including TULRv6, XLMR, multilingual BERT, mT5, and MuRIL.Most fine-tuned baselines use English training data and are evaluated zero-shot cross-lingually on target languages.
- Prompt construction: The prompting framework combines in-context learning with instruction following and represents each prompt through examples, instructions, templates, and answer verbalizers.The test example is combined with k few-shot exemplars and task-specific textual components before querying the LLM.
- Prediction: Predictions are obtained by providing the constructed prompt to the LLM and approximating the highest-probability answer through sampling.The answer space is the entirety of the language modeled by the LLM.
- Prompting strategies: The evaluation tests monolingual prompting, zero-shot cross-lingual prompting with English examples, and translate-test prompting.These strategies vary the relationship between the test language, prompt language, and few-shot example language.
- Evaluation settings: Prompt selection uses English validation performance from PromptSource, while most experiments use eight few-shot examples and longer-context tasks use four.The selected English template is reused across languages and prompt strategies, including for GPT-4 because of cost constraints.
3 Results and Analysis
Translate-Test generally improves multilingual LLM performance, with especially large gains for low-resource, non-Latin-script languages, but substantial English–non-English gaps remain. Across models, GPT-4 improves over GPT-3.5 models, while fine-tuned baselines generally remain stronger and tokenizer fertility and pre-training data explain only some task trends.
- Comparing different prompting strategies: Translate-Test often outperforms Monolingual prompting, especially for low-resource and non-Latin-script languages such as those in IndicXNLI and XStoryCloze.For high-resource languages, the two strategies perform similarly.
- Comparing different prompting strategies: Burmese, Tamil, and Telugu show relative Translate-Test improvements of > 30% over Monolingual prompting for GPT-3.5-Turbo.Figure 3 reports average relative improvement over tasks by language.
- Comparing different prompting strategies: Translate-Test raises GPT-4’s XStoryCloze accuracy on Burmese from 77.6% with Monolingual prompting to 93.2%.The comparison is based on the reported GPT-4 evaluation for XStoryCloze.
- Comparing different prompting strategies: After Translate-Test, non-English performance still trails English substantially: GPT-3.5-Turbo reaches 54% accuracy for Urdu on XNLI versus 76.2% in English.Urdu improves from 49.1% under Monolingual prompting to 54% with Translate-Test.
- Comparing different models: OpenAI models generally lag behind fine-tuned TULRv6 across tasks, while outperforming BLOOMZ except on PAWS-X, XQUAD, and TyDiQA-GoldP.The BLOOMZ advantage on those datasets is confounded because they were included in its multitask fine-tuning, with evaluation data potentially included for XQUAD and TyDiQA-GoldP.
- Factors Explaining Performance Trends: GPT-4 consistently improves over GPT-3.5 models and is Pareto Optimal across datasets except XL-Sum, while tokenizer fertility has significant negative correlations with performance on only some tasks.Pre-training data size also correlates positively with performance for PAWS-X, XNLI, XCOPA, and XQuAD; these correlations may not imply causation.
4 Challenges in Multilingual Evaluation
MEGA examines practical challenges in multilingual LLM evaluation, including prompt sensitivity, computational cost, and possible test-data contamination. These factors complicate interpreting model performance across languages.
- Prompting choices: Prompt-based evaluation varies with templates, instructions, examples, strategies, prompt language, external tools, and decoding parameters.Exploring all combinations across datasets and languages is computationally expensive.
- Prompting choices: English explanations have negligible impact on XStoryCloze performance, and models may translate problems into English before explaining them.
- Prompting choices: Larger few-shot sizes generally improve performance, but results are often stable beyond k = 8.Haitian Creole in XCOPA continues improving beyond this point.
- Prompting choices: Language-specific prompt tuning improves Haitian Creole performance in XCOPA but reduces Tamil accuracy, possibly because the validation set contains only 100 examples.
- Test-data contamination: GPT-4 contamination checks combine dataset-card knowledge, online test-set availability, and dataset release date.GPT-4 correctly fills most dataset cards, while newer or code-mixing datasets show weaker evidence of contamination.
- Test-data contamination: Test-data contamination may overestimate LLM capabilities, although models still substantially underperform on especially low-resource non-English languages.The observed English–non-English disparity could therefore be even larger than reported.
5 Related Work
Prior LLM evaluations have broad task coverage but remain predominantly English-focused, while multilingual benchmarks often emphasize cross-lingual transfer or limited task subsets. MEGA extends this landscape toward broader multilingual prompting evaluation.
- Evaluation of LLMs: BIG-bench covers diverse tasks, but its non-English tasks are mostly translation-oriented, limiting general task-based multilingual inferences.
- Evaluation of LLMs: HELM evaluates 30 language models across 42 scenarios and 7 metrics, but its scenarios use standard English or English dialect data.
- Multilingual benchmarks: XTREME, XTREME-R, and XGLUE measure cross-lingual transfer in pretrained language models, generally using pretrain-then-finetune setups.
- Multilingual benchmarks: Prior multilingual ChatGPT evaluation found failures on low-resource languages with non-Latin scripts, but covered only a few tasks and tested 50–100 examples.
- Multilingual prompting: Multilingual prompting research includes evidence that English prompts can outperform target-language prompts in some settings, while most prompting work remains English-focused.
6 Conclusion
MEGA evaluates multilingual LLM capabilities across prompting strategies, models, tasks, and languages while examining tokenizer quality and pretraining data. It finds persistent resource- and script-based performance gaps and argues that broader benchmarking is needed.
- MEGA evaluates prompting strategies, models, tasks, and languages to investigate multilingual LLM capabilities.
- Tokenizer quality and pretraining-data size are examined as underlying properties that may explain observed cross-language performance trends.
- Performance gaps persist between high-resource Latin-script languages and under-resourced languages, while translate-test prompting is effective but insufficient.
- The paper highlights the need to prioritize automatic benchmarking and human evaluation across as many languages as possible.
Limitations
The study’s multilingual evaluation is limited in model, dataset, language, and evaluation-dimension coverage. Available benchmarks also underrepresent many typologically diverse and under-resourced languages.
- The study does not evaluate closed models such as PaLM.
- The evaluation excludes some available multilingual datasets and plans to expand in future versions.
- Existing benchmarks provide little representation of African and Indigenous languages of the Americas.
A.1 Tasks and Datasets
MEGA evaluates 16 tasks spanning classification, sequence-to-sequence labeling, and generation. The section introduces the experimental setups and datasets used for benchmarking.
- MEGA covers 16 tasks across classification, sequence-to-sequence labeling, and generation.
- The section reviews experimental setups and datasets for the benchmarked task types.
- Table 3 lists the datasets and their covered languages.
A.1.1 Classification
Classification tasks assign sentences or sentence groups to discrete labels and are evaluated by accuracy, including exact-match scoring for unconstrained generated labels. The benchmark includes multilingual inference, paraphrase, commonsense, and story-cloze datasets.
- Classification assigns one sentence or a sentence group to a finite set of discrete labels.
- Prompt-based models are scored by exact match between generated outputs and verbalized labels.
- XNLI covers cross-lingual natural language inference in 14 languages, while IndicXNLI covers 11 Indic languages.
- PAWS-X provides professionally translated paraphrase data in six typologically diverse languages.
- XCOPA covers commonsense reasoning in 11 typologically diverse languages, including Eastern Apurímac Quechua and Haitian Creole.
- XStoryCloze translates the English StoryCloze dataset into 10 typologically diverse languages.
A.1.2 Question Answering
The QA experiments focus on span prediction across multilingual datasets, with retrieval used for context-length constraints in older GPT models but not newer long-context models. They evaluate answer extraction using Exact Match and F1.
- The experiments focus on span prediction, where the answer must be predicted from a supplied context.
- For older GPT models, answer-line few-shot contexts and embedding-based chunk retrieval address multilingual context-length constraints.
- GPT-3.5-Turbo and GPT-4 are prompted directly for QA because they support longer context lengths.
- QA performance is measured with Exact Match and F1, using TyDiQA and MLQA among four evaluated tasks.
- TyDiQA evaluates Gold-P passage selection and minimum answer span with monolingual and zero-shot cross-lingual prompting.
- MLQA includes same-language and cross-language variants, but the experiments use the same-language question, context, and answer variant.
A.2 Sequences Labeling
MEGA covers multilingual and code-switching evaluation settings, using prompting and translation-based procedures across diverse NLP tasks. The experiments also examine prompt language and explanation effects, while identifying retrieval quality as a limitation for some models.
- Sequence labeling: Sequence labeling uses linguistic-structure prompting, while translation-test prompting is excluded because token alignment would be required.GPT-3.5-Turbo is evaluated monolingually for these tasks to reduce compute costs across datasets with more than 30 languages.
- Dataset coverage: MEGA includes monolingual datasets, two code-switching datasets, and Responsible AI evaluations of fairness and toxicity.The code-switching datasets address Hindi-English inference and English-Spanish sentiment; the Responsible AI datasets are Jigsaw5 and WinoMT.
- Responsible AI evaluation: WinoMT evaluation translates English sentences into eight target languages, aligns entities with fast_align, and compares extracted target-side genders with English gold annotations.The dataset balances male and female genders and stereotypical and non-stereotypical role assignments.
- Multilingual evaluation challenges: DV003 performs significantly worse on IndicQA because imperfect retrieval can reduce retrieval accuracy to 5% in low-resource Telugu.GPT-3.5-Turbo avoids retrieval because its larger context size supports the task directly; better multilingual retrieval embeddings are suggested as a possible direction.
- Multilingual evaluation challenges: Native-language prompt templates perform much worse than English templates for DV003, while explanations leave most XStoryCloze results unchanged or slightly worse.For XCOPA, adding explanations does not improve performance and slightly hurts it; Telugu is an exception with a slight XStoryCloze gain.