Are We Done with MMLU?

1 Introduction

• 57% of the analysed Virology instances contain errors, including a suggestion to send the American army to West Africa to prevent Ebola outbreaks.The paper describes errors ranging from parsing and scraping mistakes to issues of context, interpretation, and dataset quality.Source: §1 · p. 1
• Fourteen human experts re-annotated 5,700 questions across all MMLU subjects, estimating that 6.49% of questions contain errors.The re-annotation used a novel error annotation protocol to construct MMLU-Redux.Source: §1 · p. 2
• Re-evaluation on MMLU-Redux notably altered leading LLMs’ performance metrics and changed their ranking.The study also quantitatively and qualitatively analysed how dataset errors affect LLM evaluation.Source: §1 · p. 2
• The study investigates LLM-assisted dataset error detection using in-context learning, chain-of-thought prompting, retrieval-augmented generation, and fine-tuning.MMLU-Redux is proposed as a benchmark for automatic error detection in NLP datasets.Source: §1 · p. 2

2 What Is Wrong with MMLU?

• MMLU’s under-documented errors range from misparsed answers to missing context, occur unpredictably, and are difficult to trace to their causes.The benchmark was assembled from freely available online sources with graduate and undergraduate student help, but its annotation procedure is under-documented.Source: §2 · p. 2

2.1 Error Detection Annotation Protocol

• MMLU-Redux applies the protocol to 100 randomly selected samples from each of MMLU’s 57 subsets.The resulting manual annotation was designed to cover different error types for further experiments.Source: §2 · p. 2, §2.1 · p. 2
• The protocol separates unclear questions or options from ground-truth errors, including no correct answer, multiple correct answers, and a wrong label.Question clarity covers presentation and information sufficiency, while option clarity includes unclear, similar, irrelevant, or incorrectly parsed choices.Source: §2.1 · p. 2

2.2 Heterogeneity of Errors in MMLU

• Virology has prevalent incorrect ground-truth labels, often on relatively simple questions, while College Chemistry includes parsing errors and unexplained label mismatches.In chemistry, multi-line questions were sometimes parsed so that part of the question became Option A and Option D was excluded.Source: §2.2 · pp. 2–3
• Professional Law omits jurisdictional distinctions, Formal Logic has many incorrect answers, and Global Facts questions often require external validation amid conflicting sources.The Formal Logic inaccuracies are mostly sourced from Oxford Learning Link; the Global Facts examples include conflicting evidence about political-party corruption in Ethiopia.Source: §2.2 · p. 3
• Machine Learning chiefly has unclear questions and options, Econometrics has unverifiable references, and some subjects omit needed context or reflect US- and Western-centric assumptions.Professional Law and Accounting assume US jurisdiction or practices without stating them, while examples of regional framing include Virology’s US-context reference to “the Latino population.”Source: §2.2 · pp. 3–4

3 MMLU-Redux: A Correct MMLU Subset

• MMLU-Redux manually labels a subset of MMLU questions by error type, checking original sources where available; annotator-suggested answers do not replace existing labels.The subset is designed to measure error prevalence and types and to explore whether annotated error types can enable automatic fixes, which the authors find challenging.Source: §3 · p. 4

3.1 Analysis of MMLU-Redux

• 6.49% of questions in the complete MMLU dataset are estimated to contain errors, and more than 57% of Virology examples contain errors.More than 20% of examples in Logical Fallacies and College Chemistry are wrong; more than 10% are wrong in Professional Law, Business Ethics, Formal Logic, Human Aging, Global Facts, Machine Learning, Miscellaneous, and Public Relations.Source: §3.1 · p. 4
• Cohen’s Kappa exceeded 0.6 for both specific error-type matching and binary error presence across most of the five evaluated subjects, indicating generally high annotator agreement.The small difference between the two agreement measures supports the effectiveness of the annotation protocol.Source: §3.1 · p. 4

3.2 Re-Evaluating the State-of-the-Art LLMs

• Virology rankings shifted sharply: Llama 3.1 Instruct Turbo (405B) ranked 16th on all instances but first on correct instances, showing errors can change model rankings.In Human Sexuality, GPT-4 (0613) scored 0.91 exact match across all instances but 0.43 on correct instances, dropping from fifth to last among the top 10 models.Source: §3.2 · p. 5

3.3 Correlation between Exact Match Score and Data Quality

• LLMs generally perform better in MMLU subjects with fewer erroneous instances, but outliers—including strong performance in Logical Fallacies—suggest other subject-specific factors may matter.The authors identify these deviations as requiring further investigation.Source: §3.3 · p. 5

4 Can We Fix the MMLU Dataset Automatically?

• The detection strategies include zero-shot and few-shot prompts, CoT prompting, retrieval-augmented prompting, and instruction-tuning Llama3 8B-Instruct on curated data.The RAG variants add five retrieved Wikipedia or MS-MARCO paragraphs to zero-shot or CoT prompts.Source: §4 · p. 7

4.1 Error Detection Experiments

• Experiments evaluate GPT-4 Turbo, GPT-4o, Claude-3 Opus, and Llama-3-70B using standard and CoT prompting.For Recall, F1, and F2 calculations, “not ok” is treated as the positive class and “ok” as the negative class.Source: §4.1 · p. 7
• Few-shot CoT outperforms other tested settings across models, but Claude 3 Opus reaches only an F2 Score of 40.29.The authors suggest that labelled examples and step-by-step reasoning can improve detection, while the best result still highlights the task’s difficulty.Source: §4.1 · p. 7, Table 3 · p. 7
• RAG retrieves five relevant paragraphs from enwiki-paragraphs and msmarco-v1passage using the question as the query, then adds them as classification context.Retrieval uses BM25 and Pyserini (Lin et al., 2021a).Source: §4.1 · p. 7
• Claude 3 Opus zero-shot RAG reaches 83.91 Recall on MS MARCO and 41.92 F2 on Wikipedia, yet overall error-detection reliability remains limited.RAG outperforms the few-shot CoT setting, but its best model–method combination still achieves relatively low scores.Source: §4.1 · pp. 7–8, Table 4 · p. 8
• GPT-4o scores lower than GPT-4 Turbo; Wikipedia generally beats MS MARCO for GPT-4o, while retrieval-index results are mixed for GPT-4 Turbo and Llama3-70B.Claude 3 Opus has the highest Recall, F1, and F2 among the compared models, particularly with RAG.Source: §4.1 · p. 8

Benchmark Issues

• Annotation artifacts in ImageNet likely overstated performance gains that did not necessarily transfer, and benchmark issues have also been reported in summarization and natural language inference.The cited studies are Beyer et al. (2020) and Russakovsky et al. (2015) for ImageNet; Tejaswin et al. (2021) and Bhandari et al. (2020) for summarization; and Gururangan et al. (2018), Poliak et al. (2018), Stacey et al. (2020), and Wu et al. (2022) for natural language inference.Source: §5 · p. 8
• Benchmark issues can arise from task framing, noisy annotations, web crawls, automated labeling, human error, weak supervision, and personal or group-level annotator biases.The cited work links these potential causes to Schwartz et al. (2017), Chen et al. (2016), Raffel et al. (2023), Lee et al. (2021), Kaddour (2023), Yuen et al. (2011), Peterson et al. (2019), Zhang et al. (2022), Goswami et al. (2021), Kaddour et al. (2023), Geva et al. (2019), and Liu et al. (2022).Source: §5 · p. 8

MMLU Issues and MMLU-Pro

• Errors from original MMLU persist in MMLU-Pro, while systematic categorization and quantification across MMLU remain limited (Wang et al., 2024).Global MMLU reevaluated cultural bias, and the authors observed similar Western bias in their qualitative analysis (Singh et al., 2024); a popular video also highlighted erroneous MMLU instances, but this study aims for a systematic assessment (L and Stapleton, 2023).Source: §5 · p. 8
• MMLU’s broad adoption makes identifying issues important, and related concerns include labeling errors and ambiguous questions in MedQA (Saab et al., 2024).MMLU is used for LLM evaluation by Anil et al. (2023a), OpenAI (2023), Touvron et al. (2023), Anthropic (2024), and Dubey et al. (2024).Source: §5 · p. 8

6 Conclusion

• 6.49% is the estimated MMLU error rate, based on MMLU-Redux, a manually reviewed subset of 5,700 questions across all 57 subjects.The analysis also found inaccuracies in 57% of Virology instances and 26% of Logical fallacies instances.Source: §6 · p. 9
• Re-evaluation with MMLU-Redux yields significant variation in LLM performance metrics and shifts model rankings for several subsets.These findings emphasize the impact of dataset quality on LLM evaluation.Source: §6 · p. 9
• Claude 3 Opus produces the most accurate automatic error-identification results, but reaches only a 41.9% F2 score with retrieval-augmented generation, insufficient for a high-quality dataset.Source: §6 · p. 9

Limitations

• Annotating the remaining 8,342 MMLU questions could improve the analysis, and the error-classification protocol may still be prone to annotator bias.The authors open MMLU-Redux for annotation of additional subjects and further review of already reviewed subsets.Source: Limitations · p. 9

A Responsible Research Checklist

• The authors say the paper’s claims reflect its contribution and scope and that it describes limitations, but it does not discuss potential negative societal impacts.The checklist points readers to Section 6 for the limitations and potential negative impacts discussion.Source: A Responsible Research Checklist · p. 13
• The checklist reports that code, data, and reproduction instructions are provided, with method details and prompting and fine-tuning information in Appendices I and J.It also says compute resources are detailed in Appendix J.Source: A Responsible Research Checklist · p. 13
• The authors did not report error bars because rerunning experiments multiple times exceeded their funding capabilities.They report that the types and amount of compute resources are detailed in Appendix J.Source: A Responsible Research Checklist · p. 13
• The checklist says the authors cited MMLU, its dataset license is CC-BY 4.0, and MMLU-Redux is provided as supplementary material.The authors report that they annotated MMLU-Redux and consented to the use of the data; MMLU is identified as MIT-licensed.Source: A Responsible Research Checklist · p. 13
• The checklist says MMLU and MMLU-Redux contain no personal data because they use publicly available test questions, and marks human-subjects and crowdsourcing items not applicable.The authors state that MMLU-Redux annotation was done by the paper’s authors, who consented to use of the data.Source: A Responsible Research Checklist · p. 13

B Data Collection and Organisation

• Annotators reviewed 100 randomly sampled questions per subject, checking presentation and ground-truth labels against credible sources or expertise, and marking unresolved cases “Expert.”The resulting MMLU-Redux subset follows the introduced error taxonomy.Source: B Data Collection and Organisation · p. 14
• Each data point has seven columns, including an error type, potential source, suggested correct answer for some errors, and a free-text potential reason.The error-type values include “ok,” five named error categories, and “expert.”Source: B Data Collection and Organisation · p. 14

C Usage Guidelines

• Users can filter samples by “error_type” and inspect “correct_answer” for no-correct-answer, multiple-correct-answers, or wrong-groundtruth cases; questions and options remain unmodified, including when unclear.The dataset is available on Hugging Face under a CC-BY-4.0 licence.Source: B Data Collection and Organisation · p. 14, C Usage Guidelines · p. 15

D.1 General Maintenance

• Dataset updates will be announced on Hugging Face, while older versions will be preserved and documented by repository git commit hash.Source: §D.1 · p. 15

D.2 Contributing Guidelines

• The authors invite contributors to annotate more questions beyond the current 5,700, using the taxonomy and submitting question, answer, error, source, and rationale details through Hugging Face discussions.Contributors are asked to title new annotation discussions with “[ADD]” and the MMLU subject.Source: §D.2 · p. 15
• Community members can propose fixes for existing annotations, including updated error types and answers, through “[FIX]” discussions; bulk contributions may be sent directly to the authors.The paper notes that annotations may become inaccurate, including when answers change over time.Source: §D.2 · p. 16

D.3 Contingency Plan

• If Hugging Face becomes unavailable, the authors plan to notify users, redirect them to mirror copies such as Zenodo, and reupload the dataset with updated documentation and links if needed.Source: §D.3 · p. 17

E Authors Statement

• The authors state that they bear all responsibility in case of rights violations.Source: E Authors Statement · p. 17

F MMLU-Redux Error Type Statistics

• 100 questions were annotated for each subject subset, although Table 5 also lists each subject’s total question count.The section presents these figures as an overview of error types found in MMLU-Redux.Source: F MMLU-Redux Error Type Statistics · p. 17, Table 5 · p. 18
• Table 5 groups errors as Bad Question Clarity (BQC), Bad Options Clarity (BOC), No Correct Answer (NCA), Multiple Correct Answers (MCA), and Wrong Ground Truth (WGT).The table labels these categories (1a), (1b), (2a), (2b), and (2c), respectively.Source: Table 5 · p. 18
• The percentages in Table 5’s Total row are estimated via stratified sampling over the subsets.Source: Table 5 · p. 18

G Prompting Methods

• The evaluation used the configured MMLU-Redux test split, temperature 0.0, top_p 1, zero frequency and presence penalties, max_tokens 600, and the default random seed.These settings were used for both standard and CoT prompting.Source: G Prompting Methods · p. 19
• The standard prompt asks whether a question, its choices, and ground-truth answer are ‘ok’ or ‘not ok’.It directs the model to classify without providing reasoning.Source: G Prompting Methods · p. 19
• ‘Ok’ means the question and choices are understandable and the ground-truth answer is correct; ‘not ok’ covers an incorrect answer or poor presentation.Source: G Prompting Methods · p. 19
• The CoT task frames the input as a triple of a multiple-choice question, its choices, and the corresponding ground-truth answer.Source: G Prompting Methods · p. 19
• The CoT prompt assesses question clarity and option presentation, then checks whether exactly one potentially correct answer exists before evaluating the ground-truth answer.Poor presentation, multiple or no potentially correct answers, or an incorrect ground truth lead to ‘not ok’; otherwise, the classification is ‘ok’.Source: G Prompting Methods · p. 19
• The CoT prompt requests a one-line JSON response with assessment fields for question and options presentation, answer evaluation, ground-truth evaluation, and classification.The supplied example continues across passages and specifies the output format.Source: G Prompting Methods · p. 19

• Table 6 compares Zero-Shot, Chain-of-Thought (CoT), and Few-Shot F1 scores on 10 MMLU-Redux subjects.Source: Table 6 · p. 21
• Table 7 compares Zero-Shot, Chain-of-Thought (CoT), and Few-Shot F2 scores on 10 MMLU-Redux subjects.Source: Table 7 · p. 22
• Table 8 compares Zero-Shot, Chain-of-Thought (CoT), and Few-Shot Recall scores on 10 MMLU-Redux subjects.Source: Table 8 · p. 23

I Heterogeneity of Errors in MMLU Subsets – Full List

• Virology has especially frequent wrong ground-truth labels, while other subsets show answer-extraction, parsing, or option-conversion errors.Reported examples include malformed numeric answers in accounting, incorrect formal-logic labels, omitted or misparsed choices in chemistry and mathematics, and extraction errors in electrical engineering and statistics.Source: I Heterogeneity of Errors in MMLU… · pp. 24–25
• Missing context and unclear wording affect subjects including law, logic, machine learning, psychology, and security studies.Examples include unspecified legal jurisdictions, omitted instructions or statements, vague or time-dependent questions, and abbreviations whose meanings relied on textbook context.Source: I Heterogeneity of Errors in MMLU… · pp. 24–26
• Some subsets pose verification or answerability problems, including conflicting external sources, unverifiable references, subjective prompts, and outdated questions.Global Facts questions often required external sources that could conflict; other examples include missing references, speculative Human Sexuality questions, and an Astronomy item lacking a correct answer after discoveries changed.Source: I Heterogeneity of Errors in MMLU… · pp. 24–27
• Many subsets are mostly correct or clean, although some still contain isolated errors, duplicates, or unclear items.The list describes Anatomy, High School Computer Science, Prehistory, Medical Genetics, and several high-school subjects as almost entirely correct or clean; College Physics has approximately 20% duplicated questions.Source: I Heterogeneity of Errors in MMLU… · pp. 24–27
• Other subject-specific issues include duplicated source content, missing reference material, and confusing or incomplete answer choices.College and Clinical Medicine share questions from one textbook; European History includes equivalent options or references without answers, while Microeconomics and Macroeconomics include underspecified or malformed items.Source: I Heterogeneity of Errors in MMLU… · pp. 25–27
• Several sampled subsets report small counts of problematic instances, with causes spanning unclear wording, missing content, wrong labels, and stereotyping.Professional Psychology and High School Psychology each report four problematic cases among 100; Security Studies reports six unclear cases among 100, while Moral Disputes reports four erroneous instances and Moral Scenarios three ambiguous instances.Source: I Heterogeneity of Errors in MMLU… · p. 26

J Error Detection via Fine-tuning

• LabelChaos mirrors MMLU’s error distribution and benchmarks models fine-tuned for error detection before evaluation on MMLU-Redux.The dataset contains approximately 264,000 samples, similar in size to MMLU.Source: J Error Detection via Fine-tuning · pp. 27–28
• LabelChaos combines six human-labelled datasets, standardizes them to MMLU’s format, and corrupts them to replicate MMLU’s quality and error distribution.The source datasets were chosen to avoid potential inaccuracies from automated labelling procedures.Source: J Error Detection via Fine-tuning · pp. 27–28
• LabelChaos defines five corruption types—wrong ground truth, poor question clarity, no correct answers, unclear options, and multiple correct answers—and includes a Correct category for the original, uncorrupted dataset.Source: J Error Detection via Fine-tuning · p. 28
• Fine-tuning uses a balanced label distribution of 0.1 for each error type and 0.5 for correct examples, with CoT prompts and “ok” or “not ok” outputs.This distribution addresses the predominance of correct labels in MMLU-Redux.Source: J Error Detection via Fine-tuning · p. 28
• Llama-3 (8B-Instruct) was fine-tuned for 2048 steps with batch size 64, LoRA rank 16, and learning rate 2 × 10−4 on one Nvidia A100 GPU.The setup used AdamW with no weight decay; LoRA was applied because of computational constraints (Dubey et al., 2024; Loshchilov and Hutter, 2018; Hu et al., 2021).Source: J Error Detection via Fine-tuning · p. 28