M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models

1 Introduction

• Human exams demand a wider mix of abilities than task-specific NLP benchmarks, including language understanding and complex reasoning.The paper presents exams as testbeds for evaluating LLM general-intelligence capabilities.Source: §1 · pp. 1–2
• Existing human-exam benchmarks often underrepresent multilingual, image-based, and level-specific evaluation: many focus on English, text alone, or mixed education levels.Translated datasets may also miss culturally specific concepts in target languages.Source: §1 · p. 2
• M3Exam contains 12,317 official exam questions in nine languages across three educational periods, with about 23% requiring image information to solve.Questions also include answer options, ground truth, and metadata such as language, level, and subject.Source: §1 · pp. 2–3
• Current models struggle with multilingual text, especially low-resource and non-Latin-script languages, while multimodal models struggle with complex images and cross-image reasoning.GPT-4 is the only model reported above 60% accuracy in the introduction; BLIP-2 attains less than 50%.Source: §1 · p. 3

2.1 Design Principle

• The benchmark is designed around three assessment principles: authentic multilingual data, questions requiring multiple modalities, and exams from critical educational stages.The authors motivate authentic language data as important for retaining cultural and world knowledge, and staged exams as a way to assess different intelligence requirements.Source: §2.1 · p. 3

2.2 Language Selection and Data Collection

• The dataset spans nine languages selected for cultural and linguistic diversity, including languages with different resource levels, language families, and scripts.The set includes English, Chinese, Italian, Portuguese, Vietnamese, Thai, Swahili, Afrikaans, and Javanese.Source: §2.2 · p. 3
• Native speakers collected official exams and answers from the end of primary, middle, and high school, prioritizing exams with broad participation and diverse subject coverage.The collection totals 435 exam papers from nine countries.Source: §2.2 · pp. 3–4

2.4 Data Statistics

• Table 1 breaks down question counts by language and education level, distinguishing text-only from image-requiring questions with X/Y counts.Languages are ranked by their CommonCrawl corpus ratio (“CC Size”).Source: Table 1 · p. 4

2.3 Data Processing and Annotation

• OCR and annotator review standardize exam papers; the benchmark includes multiple-choice questions and adds contextual background needed to solve them.Annotators correct OCR errors, separate questions and options, enter correct answers, adapt formats, and conduct multiple quality checks; image locations are marked with placeholders and saved image files.Source: §2.3 · p. 4

2.4 Data Statistics

• M3Exam contains 12,317 questions with context, question text, options, answers, and metadata; language coverage ranges from high-resource to extremely low-resource languages.Image-question shares vary by country, exceeding 60% for Chinese; held-out development examples are selected by language, subject, and level.Source: §2.4 · pp. 4–5

3.1 Models

• The evaluation covers text-only multilingual models and multimodal models, including systems designed for single-image and multi-image inputs.The multimodal set includes BLIP-2 and InstructBLIP, which process one image per question, and Fromage and OpenFlamingo, which handle multiple images.Source: §3.1 · p. 5

3.2 Settings

• Experiments are primarily zero-shot; prompts specify the subject and request an answer option, with language-specific translations used to keep each prompt monolingual.The multimodal prompt omits the output-format constraint because constraint decoding is applied.Source: §3.2 · pp. 5–6
• Table 2 reports accuracy scores for questions in different languages.Source: Table 2 · p. 6
• Accuracy is computed by taking the first alphabetic letter of a model's output as its prediction and comparing it with the ground-truth answer.The questions are multiple-choice, enabling automatic correctness evaluation.Source: §3.2 · p. 6

4.1 Multilingual Evaluation

• 72.92% accuracy made GPT-4 the only model above 60%, and it consistently outperformed other models across languages, while models generally struggled with non-Latin and low-resource languages.ChatGPT and Claude scored similarly in English (75.98% vs. 74.25%), but ChatGPT performed better in other languages.Source: §4.1 · p. 6
• English instructions did not consistently improve ChatGPT’s performance, while English translations greatly improved performance for some languages, including Thai and Javanese.Translations could also lose essential information in questions tied to a specific language.Source: §4.1 · p. 7

4.2 Multimodal Evaluation

• Most multimodal models performed poorly overall; only BLIP-2 marginally surpassed text-only Flan-T5, and none of the models accurately identified the marked 65° angle in the example.The authors attribute difficulty to details such as axes in math questions and maps in geography questions.Source: Table 4 · p. 7, §4.2 · p. 8, Figure 3 · p. 8
• Multi-image questions require cross-image reasoning, yet Fromage and OpenFlamingo performed notably worse than single-image BLIP-2 and InstructBLIP.The authors report that multi-image pretraining does not necessarily guarantee better multimodal understanding.Source: §4.2 · p. 8

4.3 Multilevel Evaluation

• High-level questions generally had the lowest performance, but nearly all models showed no clear decrease in performance as educational level rose.The paper contrasts this pattern with its expectation that human performance would decline monotonically as exam levels increase.Source: §4.3 · pp. 8–9, Figure 4 · p. 8
• The authors argue that investigating LLM failures, including on primary-school questions, may matter more than creating more challenging datasets.They suggest LLM learning differs from human learning because models are trained first on massive datasets, biasing their knowledge toward more common data.Source: §4.3 · p. 9

4.4 Discussions

• Across English, Chinese, Vietnamese, and Thai, ChatGPT underperformed in math but performed relatively better in natural and social sciences.The authors suggest math questions’ reasoning demands present a challenge, while natural and social science questions involve more structured and factual information.Source: §4.4 · p. 9

• LLMs have advanced across NLP tasks and have been adopted in academia and industry, making careful evaluation important [9] [31] [3].Source: §5 · p. 9
• Traditional benchmarks target specific tasks, while multilingual and multimodal benchmarks evaluate language coverage and combined text-image-video abilities [39] [38] [34] [35].Source: §5 · p. 9
• Human-exam benchmarks include MMLU [17], AGIEval [43], C-Eval [19], GAOKAO [42], and IgakuQA [22].Source: §5 · p. 9
• Existing exam datasets have limited language diversity, lack multimodal and multilevel evaluation, and thus constrain comprehensive assessment [17] [43].Source: §5 · pp. 9–10

• Current models struggle with multilingual text, especially low-resource and non-Latin-script languages, and state-of-the-art multimodal models struggle to achieve reasonable accuracy.Source: §6 · p. 10
• M3Exam provides a multilingual, multimodal, and multilevel assessment of LLMs.Source: §6 · p. 10
• The benchmark offers a resource for tracking LLM progress and examining machine intelligence across education levels.Source: §6 · p. 10
• M3Exam currently includes only multiple-choice questions, so it does not evaluate questions requiring creative writing.Source: §6 · p. 10

• This section provides additional information about the M3Exam dataset.Source: §A.1 · p. 14

• M3Exam tests multilingual and multimodal abilities using questions from real and official human exams.Source: §A.1.1 · p. 14
• The benchmark aims to address limitations of multilingual and multimodal evaluation that focuses on traditional NLP tasks.Source: §A.1.1 · p. 14
• Questions from different education levels are intended to provide insights into the development of machine intelligence.Source: §A.1.1 · p. 14

A.1.2 Composition

• M3Exam contains 12,317 questions, including textual questions and questions that require images to solve.Source: §A.1.2 · p. 14
• Questions come from exam papers across multiple years for each language and are described as representative of expected knowledge.Source: §A.1.2 · p. 14
• The dataset is self-contained, identifies image-requiring questions, and releases corresponding images; it excludes specific-person information and offensive, insulting, or threatening content.Source: §A.1.2 · p. 14

A.1.3 Usage and Distribution

• The dataset is released as JSON under a CC BY-NC-SA license, with README usage information and code for reading and processing it.Source: §A.1.3 · p. 14

A.3 More examples on multimodal questions

• In one image question, the text names Gandhi; BLIP-2 and InstructBLIP give comparatively more accurate descriptions, while Fromage and Flamingo give irrelevant or hallucinated descriptions.Source: §A.3 · p. 14, Figure 7 · p. 16
• In an image-options example, models identify objects and sometimes cube counts, but struggle to describe every option accurately; questions may require understanding and reasoning across multiple images.Source: §A.3 · p. 14, Figure 7 · p. 16

A.2 Examples of different prompting strategies

• The prompting examples contrast target-language zero-shot prompts, few-shot examples, English instructions, and English-translated data; translating a Thai question may not remove its background-knowledge demands.Source: §A.2 · p. 14, Figure 6 · p. 15

A.4 Details on exam papers

• Table 5 lists the collected exam papers used to construct M3Exam, with exam names translated into English.Source: §A.4 · p. 15, Table 5 · p. 17

A.5 Examples questions of each language in M3Exam

• Figures 8–12 provide example questions in English, Chinese, Italian, Portuguese, and Vietnamese.Source: §A.5 · p. 15, Figure 8 · p. 16, Figure 9 · p. 18
• Figures 13–16 provide example questions in Thai, Swahili, Afrikaans, and Javanese.Source: §A.5 · p. 15, Figure 13 · p. 21, Figure 14 · p. 21