Source-linked AI summary
MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, Wenhu Chen
TL;DR
Existing LLM benchmarks, especially MMLU, are becoming saturated and contain shortcut-prone, knowledge-driven, and noisy questions. This paper introduces MMLU-Pro, which adds harder reasoning questions, ten-option choices, filtering, and expert review. MMLU-Pro challenges leading models and more clearly differentiates their capabilities, while remaining limited by its multiple-choice and language-only scope.
Problem
MMLU’s saturation, limited distractors, mostly knowledge-driven questions, and noisy items constrain reliable discrimination of LLM capabilities.
Method
MMLU-Pro constructs a 14-domain benchmark by filtering MMLU, adding questions from several sources, expanding choices to ten, and applying expert review.
Results
MMLU-Pro more clearly differentiates models and remains difficult for leading systems, with GPT-4o achieving 72.6% accuracy and GPT-4-Turbo 63.7%.
Takeaways & Limitations
MMLU-Pro is presented as a more challenging, discriminative, and robust benchmark for tracking progress in multi-task language understanding and reasoning.
Takeaways & Limitations
MMLU-Pro remains constrained by multiple-choice evaluation and excludes multimodal models and visual, auditory, and textual synthesis.
Abstract
from arXiv · showhide
In the age of large-scale language models, benchmarks like the Massive Multitask Language Understanding (MMLU) have been pivotal in pushing the boundaries of what AI can achieve in language comprehension and reasoning across diverse domains. However, as models continue to improve, their performance on these benchmarks has begun to plateau, making it increasingly difficult to discern differences in model capabilities. This paper introduces MMLU-Pro, an enhanced dataset designed to extend the mostly knowledge-driven MMLU benchmark by integrating more challenging, reasoning-focused questions and expanding the choice set from four to ten options. Additionally, MMLU-Pro eliminates the trivial and noisy questions in MMLU. Our experimental results show that MMLU-Pro not only raises the challenge, causing a significant drop in accuracy by 16% to 33% compared to MMLU but also demonstrates greater stability under varying prompts. With 24 different prompt styles tested, the sensitivity of model scores to prompt variations decreased from 4-5% in MMLU to just 2% in MMLU-Pro. Additionally, we found that models utilizing Chain of Thought (CoT) reasoning achieved better performance on MMLU-Pro compared to direct answering, which is in stark contrast to the findings on the original MMLU, indicating that MMLU-Pro includes more complex reasoning questions. Our assessments confirm that MMLU-Pro is a more discriminative benchmark to better track progress in the field.
1 Introduction
MMLU has become saturated and contains shortcut-prone, mostly knowledge-driven, and noisy questions. MMLU-Pro addresses these weaknesses with harder reasoning tasks, more answer options, and expert review, while separating leading models more clearly.
- MMLU performance has saturated as current language models rapidly improve, limiting its ability to distinguish model capabilities.
- Three weaknesses constrain MMLU: three-option distractors enable shortcuts, questions require limited reasoning, and some items are unanswerable or incorrectly annotated.
- MMLU-Pro spans 14 domains and over 12,000 questions to provide proficient-level multi-discipline language understanding and reasoning.
- MMLU-Pro uses ten options, harder college-level problems, and two rounds of expert review to reduce guessing, increase reasoning demands, and lower dataset noise.
- 72.6% is GPT-4o’s accuracy on MMLU-Pro, while GPT-4-Turbo reaches 63.7%, leaving substantial room for improvement.
- MMLU-Pro widens the GPT-4o–GPT-4-Turbo gap from 1% on MMLU to 9% on MMLU-Pro and increases the value of deliberate reasoning.
2 Related Work
LLM benchmarks have broadened evaluation beyond general language understanding, but rapidly improving models and prompt sensitivity have exposed weaknesses in existing comparisons. These trends motivate more challenging and robust benchmarks.
- Modern LLMs have improved complex reasoning across multiple domains, motivating rigorous evaluation of their expanding capabilities.
- Benchmarks such as GLUE, SuperGLUE, MMLU, HELM, BigBench, HellaSwag, and ARC broaden evaluation across language and reasoning tasks.
- Leaderboard scores have concentrated near the top as models approach near-perfect benchmark performance, highlighting the need for harder evaluations.
- Small prompt-style or phrasing changes can substantially shift scores, while four-option multiple-choice formats may contribute to instability and overestimation.
3 The MMLU-Pro Benchmark
MMLU-Pro combines filtered MMLU material with questions from additional STEM and theorem-focused sources, reorganizes subjects into 14 disciplines, and applies option augmentation and expert review. The resulting benchmark retains broad coverage while emphasizing difficult, reliable multiple-choice reasoning.
- Dataset overview: MMLU-Pro contains 14 discipline subsets and 12,032 questions assembled from original MMLU, STEM Website, TheoremQA, and SciBench.
- Dataset overview: The source integration adds selected STEM problems, theorem-application questions, and advanced college-exam science questions alongside filtered MMLU items.
- Dataset Construction Pipeline: Initial filtering merges 57 MMLU subjects into 14 broader categories and removes overly simple questions using evaluations from eight smaller language models.
- Dataset Construction Pipeline: New source questions are adapted by extracting short answers and generating three distractors before comparing solutions with the resulting options.
- Dataset Construction Pipeline: Four answer options are expanded to ten with six plausible distractors intended to require discerning reasoning and reduce correct guessing.
- Dataset Construction Pipeline: Expert review verifies correctness and appropriateness, then uses Gemini-1.5-Pro and human experts to identify and remove false-negative distractors.
- Dataset Construction Pipeline: 83% of questions have ten options, 17% have fewer, and the average question contains 9.47 options after review.
4 Experimental Setup
The evaluation uses five-shot Chain-of-Thought prompting tailored by discipline, while reporting model accuracies under a stated shot-count convention. Answer extraction uses regex-based parsing with a random-choice fallback when parsing fails.
- Experimental Setup: MMLU-Pro performance is measured with five-shot Chain-of-Thought prompting and five representative demonstrations selected for each discipline.
- Answer Extraction: Answers are first extracted with a regular expression matching the instructed format, then retried with a secondary pattern when formatting deviates.
- Answer Extraction: When both extraction attempts fail, a random answer option is selected as a fallback to provide an answer for evaluation.
- Experimental Setup: Table 2 reports MMLU-Pro model performance as accuracies in percentages; most models use five shots, while two Gemini variants use zero shots.
5 Results and Analysis
MMLU-Pro reveals substantial performance differences across models and subjects, while error analysis identifies reasoning, knowledge, calculation, and other failure categories in GPT-4o.
- Overall Performance: GPT-4o achieves 72.6% overall accuracy and outperforms other models across the evaluated subjects.
- Overall Performance: Llama-3-70B-Instruct leads the open-source models with 56.2% accuracy but significantly trails GPT-4o and Claude-3-Opus across subjects.
- Subject Performance: Math and Physics show disparities exceeding 70% accuracy for GPT-4o versus just over 20% for Mistral-7B-v0.1.
- Subject Performance: History and Psychology have higher performance floors, while DeepSeek-V2-Chat underperforms peers in these knowledge-intensive subjects.
- Subject Performance: Engineering and Law score consistently lower, with Engineering errors linked to complex formula derivations and multi-step calculations.
- Error Analysis: GPT-4o error analysis attributes 39% of errors to reasoning, 35% to specific knowledge, 12% to calculation, and the remainder to several smaller categories.
6 Comparison with MMLU
Compared with MMLU, MMLU-Pro better separates model performance, rewards chain-of-thought reasoning, and reduces score variability across prompt formulations.
- 6.1 Difficulty Level: MMLU scores cluster closely as model capabilities improve, whereas MMLU-Pro expands score differences and leaves more room for future improvement.
- 6.1 Difficulty Level: Four models cluster within a 4% range on MMLU, while MMLU-Pro expands their range to approximately 10%.
- 6.1 Difficulty Level: The score gap among GPT-4o, Claude-3-Opus, and GPT-4-Turbo widens from about 2% on MMLU to around 9% on MMLU-Pro.
- 6.2 Reasoning Level: GPT-4o’s CoT improvement rises from 1.5% on MMLU to 19.1% on MMLU-Pro, indicating stronger emphasis on complex reasoning.
- 6.3 Robustness Degree: Across 24 prompts, score variability is generally 4–5% on MMLU but around 2% on MMLU-Pro.
- 6.3 Robustness Degree: Prompt effects peak at 10.98% on MMLU and 3.74% on MMLU-Pro, supporting more consistent evaluations with the newer benchmark.
7 Limitations
MMLU-Pro remains limited by its multiple-choice format and its exclusive focus on language models rather than multimodal systems.
- The multiple-choice format may not capture comprehension and creative response generation as effectively as open-ended answers.
- Because MMLU-Pro evaluates only language models, it does not assess multimodal synthesis of visual, auditory, and textual data.
8 Conclusion
MMLU-Pro is introduced as a more challenging benchmark for assessing multi-task language understanding, using complex reasoning-intensive tasks to address performance saturation. Evaluations indicate that it differentiates model capabilities and reduces dependence on prompt styles.
- MMLU-Pro incorporates more complex, reasoning-intensive tasks to assess deeper language-model capabilities.
- MMLU-Pro addresses performance saturation observed in previous benchmarks by more effectively differentiating models’ capabilities.
- MMLU-Pro reduces benchmark sensitivity to prompt styles, improving robustness for evaluating language-model capabilities.
A.1 Dataset Construction Details
The dataset construction pipeline filters and sources questions across disciplines, uses language models for question transformation, and applies structured prompting for evaluation examples. The supplied passages also illustrate ten-option questions and chain-of-thought answer formats.
- Initial Filtering Details: Initial filtering removes MMLU items judged unsuitable across academic disciplines before retaining the remaining questions.Table 4 summarizes original counts, filtered counts, filtering percentages, and remaining counts by discipline.
- Question Sources: Questions come from MMLU, STEM websites, TheoremQA, and SciBench, with source dependence varying across disciplines.
- LLM-Assisted Construction: GPT-4-Turbo converts selected problems into multiple-choice questions, expands four-option questions to ten options, and recalls false-negative options with Gemini-1.5-Pro.
- Evaluation Prompts: Evaluation prompts contain an initial instruction, five demonstration examples, and a target question from the validation subset.
- Question and Answer Format: The examples use ten answer choices and instruct models to reason step by step before returning the selected answer letter.
A.3 List of Language Models Studied
The study covers multiple closed-source and open-source language-model families and reports an error analysis of GPT-4o on MMLU-Pro. The analysis separates reasoning, knowledge, calculation, and other error categories.
- Model Families: The study includes GPT, Claude, Gemini, Yi, LLaMA, Phi, DeepSeek, Qwen, and MAmmoTH2 model families.
- Model Coverage: Closed-source evaluations use recent API versions, while the study also examines open-source base and instruction-tuned models.
- Error Analysis: GPT-4o error analysis reviews 120 randomly selected erroneous predictions using expert annotators.
- Error Categories: 39% of reviewed GPT-4o errors are reasoning errors, involving difficulty logically processing steps toward the correct answer.
- Error Categories: 35% of reviewed errors reflect lack of specific knowledge, while 12% are calculation errors despite correct formulas.
- Other Errors: Other errors include no selection made (5%), question understanding errors (4%), generation issues (2%), annotation errors (2%), and answer extraction errors (1%).