PaperlayerView exact revision on arXiv

Source-linked AI summary

Are We Done with MMLU?

Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile van Krieken, Pasquale Minervini

arXiv:2406.04127v3cs.CLcs.AI

TL;DR

MMLU is widely used, but errors in its questions and answers limit the reliability of its evaluations. The paper develops a hierarchical annotation protocol and manually constructs MMLU-Redux from 5,700 questions across all 57 subjects. Re-evaluation on the corrected subset produces substantial changes in performance metrics and model rankings, while the authors acknowledge remaining annotation limitations.

  • Problem

    MMLU contains errors that can mislead evaluation and model comparison, including parsing, context, interpretation, and dataset-quality problems.

  • Method

    The authors develop a hierarchical error-annotation protocol, manually re-annotate 5,700 questions, and study automated error detection with prompting, retrieval augmentation, and fine-tuning.

  • Results

    Re-evaluation on MMLU-Redux significantly varies performance metrics and shifts model rankings across several subsets.

  • Takeaways & Limitations

    The findings underscore the need to closely study and reassess benchmarks used to evaluate NLP models.

  • Takeaways & Limitations

    The analysis covers 5,700 questions, leaving 8,342 MMLU questions for further annotation, and the protocol may remain prone to annotator bias.

Abstract

from arXiv · show

Maybe not. We identify and analyse errors in the popular Massive Multitask Language Understanding (MMLU) benchmark. Even though MMLU is widely adopted, our analysis demonstrates numerous ground truth errors that obscure the true capabilities of LLMs. For example, we find that 57% of the analysed questions in the Virology subset contain errors. To address this issue, we introduce a comprehensive framework for identifying dataset errors using a novel error annotation protocol. Then, we create MMLU-Redux, which is a subset of 5,700 manually re-annotated questions across all 57 MMLU subjects. We estimate that 6.49% of MMLU questions contain errors. Using MMLU-Redux, we demonstrate significant discrepancies with the model performance metrics that were originally reported. Our results strongly advocate for revising MMLU's error-ridden questions to enhance its future utility and reliability as a benchmark. https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux-2.0.

1 Introduction

MMLU is a widely used, broad benchmark whose reliability is undermined by dataset errors. The study manually re-annotates 5,700 questions to quantify these errors and construct MMLU-Redux.

  • MMLU evaluates LLM understanding across diverse subjects including mathematics, history, computer science, logic, and law.
  • 57% of analysed Virology instances contain errors, including an incorrect recommendation to send the American army to West Africa to prevent Ebola outbreaks.
  • 14 human experts manually assessed and re-annotated 5,700 questions across all MMLU subsets using a novel error annotation protocol.
  • 6.49% of MMLU questions are estimated to be erroneous.
  • Re-evaluation on MMLU-Redux notably alters performance metrics and changes model rankings.

2 What Is Wrong with MMLU?

MMLU contains diverse, difficult-to-trace errors involving question clarity, answer options, ground-truth labels, missing context, and source verification. The paper addresses these issues with a hierarchical annotation protocol and MMLU-Redux, while documenting substantial variation across subjects.

  • MMLU errors range from parsing mistakes to missing context, and their random occurrence makes root causes difficult to trace.
  • MMLU-Redux manually annotates 100 randomly selected samples from each of the 57 MMLU subsets.
  • The hierarchical protocol separates question-clarity errors from ground-truth errors and classifies specific error types.
  • Virology commonly has incorrect ground-truth labels, while College Chemistry often has parsing errors that split questions or omit options.
  • Professional Law omits jurisdictional context, and Machine Learning includes vague or outdated descriptions that reduce question clarity.
  • Some Global Facts questions require external validation because sources provide conflicting answers.

3 MMLU-Redux: A Correct MMLU Subset

MMLU-Redux manually annotates MMLU errors to quantify their prevalence and assess their effects on LLM evaluation. The analysis finds substantial error rates, altered model rankings, and performance patterns linked to data reliability.

  • Dataset construction: MMLU-Redux manually labels a subset of MMLU questions by error type, while retaining the original MMLU labels for analysis.Annotations were confirmed against original sources where available.
  • Error prevalence: More than 9% of analysed MMLU examples are incorrect, with Virology exceeding 57% errors and 30% wrong ground-truth labels.Logical Fallacies and College Chemistry exceed 20% errors, while several other subjects exceed 10%.
  • Error prevalence: 6.49% of questions in the complete MMLU dataset are estimated to contain errors using stratified sampling.The authors report that these proportions could produce inaccurate model comparisons and invalid rankings.
  • Evaluation effects: Erroneous instances generally reduce model performance, although Professional Law and Formal Logic sometimes show similar or higher performance.The authors suggest these cases may indicate memorisation of erroneous instances during pretraining.
  • Annotation reliability: Manual annotation shows generally high inter-annotator agreement, with Cohen’s Kappa exceeding 0.6 for both detailed and binary error categories.The agreement analysis covers five erroneous subjects.
  • Evaluation effects: MMLU errors change model rankings: Llama 3.1 Instruct Turbo (405B) ranks 16th on all Virology instances but first on correct instances.In Human Sexuality, GPT-4 (0613) falls from fifth with an exact match score of 0.91 to last among the top 10 with 0.43 on correct instances.
  • Data quality and performance: LLMs generally perform better on subjects with fewer errors, but outliers indicate that subject-specific factors also influence performance.The correlation analysis compares ten high-performing LLMs with the ratio of non-erroneous instances across subjects.

4 Can We Fix the MMLU Dataset Automatically?

The paper tests whether prompting, retrieval, and fine-tuning can automatically detect errors in MMLU-Redux. Retrieval-augmented prompting improves detection in some settings, but automatic error detection remains unreliable.

  • Evaluation analysis: Figure 5 compares Original, Correct, and Erroneous Exact Match across all subjects and the seven subjects with the most MMLU-Redux errors.The three EM measures distinguish evaluation over all instances, correct instances, and erroneous instances.
  • Error detection strategies: The study evaluates zero-shot, few-shot, Chain-of-Thought, retrieval-augmented, and instruction-fine-tuned approaches for classifying MMLU-Redux questions as “ok” or “not ok”.Retrieval uses five relevant passages from Wikipedia or MS MARCO as additional context.
  • Prompting results: Few-shot CoT outperforms other settings across models, but Claude-3-opus reaches only an F2 Score of 40.29.The result indicates that even the strongest evaluated few-shot CoT configuration struggles with error detection.
  • Retrieval-augmented results: Claude 3 Opus with zero-shot prompting on MS MARCO achieves the highest Recall of 83.91.On the Wikipedia index, Claude 3 Opus reaches an F2 Score of 41.92 with zero-shot prompting.
  • Retrieval-augmented results: Retrieval-augmented prompting outperforms few-shot CoT, although the best RAG configuration still achieves relatively low scores overall.Wikipedia generally outperforms MS MARCO for GPT-4o, while results are mixed for GPT-4 Turbo and Llama3-70B.

5 Related Work

Prior work shows that benchmark artefacts, annotation noise, data collection, and task framing can distort evaluation. Related MMLU work therefore focuses on identifying labelling errors, ambiguity, and improved benchmark construction.

  • Benchmark issues: Benchmark problems have been documented in ImageNet, summarisation, and natural language inference, where annotation artefacts or errors can affect reported performance.These examples motivate scrutiny of benchmark reliability beyond MMLU.
  • Sources of benchmark error: Benchmark issues can arise from task framing, noisy annotations, web crawls, crowdsourcing, human errors, and programmatic weak supervision.The cited causes span both data collection and annotation processes.
  • MMLU-related work: The broad adoption of MMLU makes identifying its issues important for maintaining its applicability as an evaluation benchmark.Related studies report labelling errors and ambiguous questions in benchmarks such as MedQA.

6 Conclusion

The paper finds substantial inaccuracies in MMLU, constructs MMLU-Redux to study them, and shows that they can alter LLM evaluation and model rankings. Automatic detection remains insufficient for producing a high-quality dataset.

  • Dataset findings: MMLU-Redux contains 5,700 reviewed questions spanning all 57 MMLU subjects, and the authors estimate a 6.49% MMLU error rate.The analysis reports 57% inaccurate instances in Virology and 26% in Logical Fallacies.
  • Evaluation consequences: Re-evaluating LLMs with MMLU-Redux produces substantial performance variation and shifts model rankings on several subsets.The results emphasize the effect of dataset quality on LLM evaluation.
  • Automatic detection: Claude 3 Opus performs best for automatic error detection with retrieval-augmented generation, reaching a 41.9% F2 score, but remains insufficient for high-quality dataset creation.The conclusion treats automatic detection as promising but not yet dependable enough for this purpose.

Limitations

The study covers a substantial but incomplete subset of MMLU, and its error classifications may still reflect annotator bias. The experiments also omit repeated runs for error bars because of resource constraints.

  • Coverage: The analysis covers 5,700 questions, while annotation of the remaining 8,342 MMLU questions could further improve its quantitative and qualitative accuracy.The authors open MMLU-Redux to additional annotation of remaining subjects and reviewed subsets.
  • Annotation: The annotation protocol may still be prone to personal biases among annotators.This caveat concerns the classification of errors in the reviewed dataset.
  • Experimental reporting: The experiments do not report error bars because rerunning them multiple times exceeded the authors’ funding capabilities.This limitation affects repeated-run variability reporting.

B Data Collection and Organisation

MMLU-Redux is a publicly released, manually re-annotated resource built from sampled MMLU questions, with structured error labels, source information, and suggested corrections. The project also defines procedures for filtering errors, reproducing analyses, and submitting community fixes or additional annotations.

  • The dataset is publicly released in Croissant format, with code for data generation and analyses planned for release.
  • 100 questions were randomly sampled from each MMLU subject for annotation.
  • Each data point records the original question, four choices, ground-truth answer index, error type, and potential source.
  • Annotators can provide alternative correct answers and suspected causes for no-correct-answer or wrong-groundtruth cases.
  • Users should filter instances with the error_type column and can inspect suggested answers for no_correct_answer, multiple_correct_answers, and wrong_groundtruth cases.
  • MMLU-Redux preserves the original questions and options, including when clarity problems affect an instance.
  • Community members can propose fixes or additional annotations through Hugging Face discussions using the project taxonomy and specified templates.
  • 5,700 questions are covered by MMLU-Redux as a subset of the full MMLU dataset.

F MMLU-Redux Error Type Statistics

The error-statistics analysis reports error types by MMLU subset using annotations of 100 questions per subject, with total percentages estimated through stratified sampling.

  • 100 questions were annotated for each MMLU subject, while total error percentages were estimated via stratified sampling over subsets.
  • Table 5 categorizes subset errors as bad question clarity, bad options clarity, no correct answer, multiple correct answers, or wrong ground truth.

G Prompting Methods

The prompting methods evaluate whether MMLU-Redux question triples are valid using deterministic standard and Chain-of-Thought prompting. The protocols classify presentation, options, answer availability, and ground-truth correctness.

  • Prompting Methods: The evaluation uses standard prompting and Chain-of-Thought prompting methods.
  • Prompting Methods: Both prompting methods use temperature 0.0, top_p 1, zero frequency and presence penalties, and max_tokens 600.
  • Prompting Methods: The standard prompt asks the model to classify a question, its choices, and the ground-truth answer as ‘ok’ or ‘not ok’.
  • Prompting Methods: ‘Ok’ requires understandable question and choices plus a correct ground-truth answer; otherwise the triple is ‘not ok’.
  • Prompting Methods: The standard prompt requires outputting only the ‘ok’ or ‘not ok’ classification without reasoning.
  • Prompting Methods: The CoT prompt classifies the same question-choice-answer triple into ‘ok’ or ‘not ok’.
  • Prompting Methods: The detailed protocol sequentially checks question presentation, option presentation, number of potentially correct answers, and ground-truth correctness.
  • Prompting Methods: The CoT response must use JSON keys for the four assessments and final classification, following the exact one-line format without explanation.

H.1 Detailed Results on Error Detection Experiments for MMLU-Redux

The supplied passages identify comparison tables for zero-shot, Chain-of-Thought, and few-shot evaluation on MMLU-Redux subjects, reporting F1, F2, and recall scores.

  • Table 6 compares zero-shot, Chain-of-Thought, and few-shot F1 scores on 10 MMLU-Redux subjects.
  • Table 7 compares zero-shot, Chain-of-Thought, and few-shot F2 scores on 10 MMLU-Redux subjects.
  • Table 8 compares zero-shot, Chain-of-Thought, and few-shot recall scores on 10 MMLU-Redux subjects.

I Heterogeneity of Errors in MMLU Subsets – Full List

Manual validation reveals heterogeneous errors across MMLU subsets, including unclear questions, wrong ground truths, parsing mistakes, missing context, and unsupported answer references.

  • Professional Law questions face contextual limitations, including missing jurisdictional specificity and insufficient information needed to identify the relevant disease family.
  • Business Ethics questions can omit the full statement list, leaving respondents to identify multiple correct statements from only the final statement.
  • Clinical Knowledge contains mostly good question-answer pairs but also unclear questions, confusing powers of 10, and an outdated question lacking a correct answer.
  • Professional Psychology includes erroneous instances caused by incorrect parsing, absent correct answers, and answers dependent on geographical context.
  • High School European History includes equivalent answer options and reference passages that do not contain the implied answer, while Computer Security has three wrong ground truths.
  • College Biology has one wrong ground truth and one answer copied into the question, while High School Biology includes five erroneous instances, mostly involving bad parsing.

J Error Detection via Fine-tuning

The error-detection approach uses LabelChaos, a human-annotated and synthetically corrupted dataset designed to reproduce MMLU's error categories and distribution, then fine-tunes Llama-3 for detection.

  • LabelChaos is designed to mirror the original MMLU error distribution and benchmark models fine-tuned for error detection on MMLU-Redux.
  • LabelChaos merges six human-labelled datasets, standardizes them to MMLU format, and creates corrupted versions representing the annotation protocol's error types.
  • The resulting LabelChaos dataset contains approximately 264,000 samples and is intended to replicate MMLU's quality and distribution characteristics.
  • Its corruption categories include wrong ground truths, poor question clarity, no correct answers, unclear options, multiple correct answers, and uncorrupted examples.
  • Llama-3 (8B-Instruct) is fine-tuned with a label distribution of 0.1 for each error category and 0.5 for correct instances.
Loading 2406.04127v3…