Source-linked AI summary
Measuring Massive Multitask Language Understanding
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt
TL;DR
Existing benchmarks can reach human-level performance while models remain below human-level language understanding overall. This paper introduces a 57-subject multitask benchmark and finds that large models make progress but remain lopsided, uncalibrated, and especially weak on calculations, law, and morality.
Problem
Existing benchmarks have reached human-level performance even though models remain below human-level language understanding overall.
Method
The paper builds a 57-subject multiple-choice benchmark spanning disciplines and evaluates text models in zero-shot and few-shot settings.
Results
175-billion-parameter GPT-3 reaches 43.9% accuracy, but state-of-the-art models remain lopsided and do not reach expert-level performance on any subject.
Takeaways & Limitations
The benchmark helps researchers identify important shortcomings in models across a broad range of academic and professional subjects.
Takeaways & Limitations
The benchmark is text-only and therefore does not capture important concepts conveyed mainly through images, audio, or physical interaction.
Abstract
from arXiv · showhide
We propose a new test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. We find that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average. However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy. Models also have lopsided performance and frequently do not know when they are wrong. Worse, they still have near-random accuracy on some socially important subjects such as morality and law. By comprehensively evaluating the breadth and depth of a model's academic and professional understanding, our test can be used to analyze models across many tasks and to identify important shortcomings.
1 INTRODUCTION
The paper introduces a broad benchmark to test whether language models can learn and apply knowledge across 57 human subjects using zero-shot and few-shot evaluation. Results show that only the largest GPT-3 model exceeds random chance, while models remain uneven and near-random on important domains such as mathematics, law, and morality.
- Benchmark design: The benchmark covers 57 subjects across diverse human-learned domains, including specialized areas such as law and ethics, enabling identification of model blind spots.Its granularity and breadth are intended to assess knowledge across many domains rather than a narrow set of NLP capabilities.
- Benchmark design: Models are evaluated exclusively in zero-shot and few-shot settings to measure knowledge acquired during pretraining and make evaluation more similar to human assessment.This design bridges the gap between the broad information models encounter during pretraining and existing measures of success.
- Results: The largest GPT-3 model moves beyond random chance on the massive multitask test, unlike smaller models that perform above chance and improve more continuously on previous benchmarks.The contrast is reported against HellaSwag and SuperGLUE, where smaller models start above random chance.
- Results: 43.9% accuracy: the 175 billion parameter GPT-3 model substantially exceeds the 25% random-chance accuracy achieved by few-shot models up to 13 billion parameters.Meaningful progress on the benchmark had only become possible in recent months.
- Limitations: Near-random accuracy persists on calculation-heavy subjects such as physics and mathematics and on human-values subjects such as law and morality.The paper identifies the weakness on legal and ethical understanding as particularly concerning for future models.
2 RELATED WORK
Prior work shows that pretrained models acquire broad factual knowledge, but existing evaluations do not comprehensively measure knowledge across real-world domains. MMLU addresses this gap with a simple multiple-choice accuracy test spanning difficult specialized subjects.
- Pretraining: Pretraining exposes models to broad information, and recent models can serve as knowledge bases, but no prior work comprehensively measures knowledge across many real-world domains.This motivates evaluating breadth of knowledge directly rather than assuming it from pretraining.
- Evaluation paradigms: Few-shot learning with large pretrained models enables diverse evaluation tasks without fine-tuning on large datasets.The passage contrasts this newer evaluation setting with earlier reliance on fine-tuned downstream models.
- Benchmarks: Commonsense benchmarks increasingly approach human-level performance on several tasks, whereas this work targets harder specialized subjects that people must study.Examples include HellaSwag, Physical IQA, and CosmosQA.
- Evaluation methodology: Because natural-language generation is difficult to evaluate and lacks a standard metric, this work measures classification accuracy on multiple-choice questions.The approach favors a simple-to-evaluate test over NLG-based evaluation.
- Benchmarks: Existing question-answering benchmarks are limited by easy subject coverage or reading-comprehension focus, while this work spans difficult subjects beyond linguistic understanding.The cited benchmarks either test grade-school topics or emphasize reading comprehension.
3 A MULTITASK TEST
The multitask test uses 15,908 manually collected multiple-choice questions across 57 subjects spanning humanities, social sciences, STEM, and other knowledge areas. Its breadth and varied difficulty target world knowledge, reasoning, and expert-level problem solving rather than narrow linguistic understanding.
- Test scope: 57 tasks span humanities, social sciences, hard sciences, and other important areas of knowledge.The humanities include law, philosophy, and history; social sciences include economics, sociology, politics, geography, and psychology; STEM includes physics, computer science, and mathematics.
- Dataset construction: 15,908 questions are split into few-shot development, validation, and test sets, with 14,079 questions in the test set.The validation set contains 1,540 questions, the development set has five questions per subject, and each subject has at least 100 test examples.
- Human performance: 34.5% accuracy is achieved by unspecialized Amazon Mechanical Turk participants, while expert-level performance can be substantially higher.Real-world test-taker accuracy at the 95th percentile is around 87% for the US Medical Licensing Examinations, represented in Professional Medicine.
- Evaluation goal: Success requires well-rounded models with extensive world knowledge and expert-level problem-solving ability, making the test an informative long-term goalpost.Because models are pretrained on Internet data, the test also evaluates how well they extract useful knowledge from massive corpora.
- Skill coverage: The test covers legal rule application, economic world knowledge and reasoning, mathematical problem solving, and difficult professional medicine questions.Professional Medicine includes questions requiring humans many years of study to master, while college mathematics often requires chains of reasoning and abstract knowledge.
4 EXPERIMENTS
The experiments evaluate GPT-3 and UnifiedQA on the multitask test using classification accuracy, few-shot prompting, and fixed subject-level demonstrations. Results show strong dependence on model size, uneven and below-expert performance across tasks, weaknesses on procedural problems, unusual knowledge acquisition, and poor calibration.
- Assessment and Models: GPT-3 and UnifiedQA are evaluated using classification accuracy across all examples and tasks.GPT-3 includes Small, Medium, Large, and X-Large variants with 2.7B, 6.7B, 13B, and 175B parameters, respectively.
- Few-Shot Prompt: Few-shot GPT-3 evaluation adds up to 5 answered demonstrations before each question, with predictions based on the highest probability among A, B, C, and D.A dev set with 5 fixed few-shot examples is created for each subject.
- Model Size and Accuracy: 43.9% accuracy is achieved by the 175B-parameter GPT-3 model, while the three smaller models remain near random accuracy at around 25%.Zero-shot results are qualitatively similar, with smaller models also around 25% accuracy.
- Comparing Disciplines: GPT-3’s accuracy ranges from 69% on US Foreign Policy to 26% on College Chemistry, and both evaluated models remain below expert performance on all 57 tasks.UnifiedQA performs best on marketing, reaching 82.5% accuracy.
- Procedural Knowledge: Calculation-heavy STEM subjects are among GPT-3’s lowest-accuracy tasks, consistent with stronger acquisition of declarative than procedural knowledge.GPT-3 recognizes the PEMDAS acronym but can fail to apply order-of-operations knowledge; its accuracy is 29.9% in Elementary Mathematics versus 47.4% in College Medicine and 35.0% in College Mathematics.
- Calibration: GPT-3’s confidence is a poor accuracy estimator and can differ from actual accuracy by up to 24%.The calibration analysis compares average confidence with actual accuracy for each subject.
5 DISCUSSION
The discussion frames MMLU as a text-only benchmark grounded in knowledge acquired through Internet pretraining and evaluated without large task-specific training sets. It highlights major limitations in human-value reasoning, calculation, specialized-data adaptation, multimodal coverage, and the scalability of further model growth.
- Multimodal Understanding: MMLU uses a text-only format because many important concepts are conveyed mainly through images, audio, and physical interaction.The benchmark is designed for current models that do not incorporate multimodal information, while future multimodal systems may require broader evaluation.
- The Internet as a Training Set: The benchmark assumes models acquire requisite knowledge by reading vast quantities of diverse Internet text rather than from large task-specific training sets.This treats pretraining as learning in its own right, with evaluation testing knowledge a human might acquire from the same text.
- The Internet as a Training Set: The evaluation assesses pretrained models in zero-shot, few-shot, or transfer settings using separate dev, validation, and test sets.The dev set supports few-shot prompts, the validation set can support hyperparameter tuning, and the test set computes final accuracy.
- Model Limitations: Current large-scale Transformers remain poor at modeling human approval and disapproval, especially on Professional Law and Moral Scenarios, and also struggle with calculations.The discussion identifies higher accuracy on these tasks as crucial for future systems aligned with human values.
- Model Limitations: 32.8% test accuracy was achieved by a RoBERTa-base model fine-tuned on approximately 2,000 additional Professional Law training examples.This experiment illustrates the limited success of addressing shortcomings through specialized additional training data.
- Model Limitations: A 10× increase in model size requires an approximate 5× increase in data, while specialized knowledge may become a bottleneck because esoteric subjects have less written material.The discussion therefore questions whether scaling existing language models alone will solve the test.
6 CONCLUSION
The paper introduces a 57-subject test measuring how well text models learn and apply pretrained knowledge across varying difficulty levels. Although state-of-the-art models have made meaningful progress, their performance remains lopsided and rarely excels on individual tasks.
- The authors introduce a test of how well text models learn and apply knowledge encountered during pretraining.
- 57 subjects at varying difficulty levels assess language understanding with greater breadth and depth than previous benchmarks.
- State-of-the-art models make meaningful progress but show lopsided performance and rarely excel at any individual task.
A ADDITIONAL ANALYSIS
The appendix extends the evaluation with analyses of sorted results, few-shot behavior and calibration, fine-tuning, error patterns, and format sensitivity. Results across all 57 tasks show lopsided capabilities for both GPT-3 and UnifiedQA.
- Additional analyses: The appendix presents sorted results, few-shot examples versus accuracy, and few-shot calibration figures.It also includes sections on fine-tuning, error analysis, and format sensitivity.
- Additional analyses: GPT-3 few-shot accuracies are reported for all 57 tasks alongside UnifiedQA transfer accuracies.The results are presented together in Figure 9 for direct comparison across tasks.
A.1 ANALYSIS WITH MORE FINE-TUNED MODELS
Smaller fine-tuned models can exceed random accuracy on the multitask test, but performance varies substantially across model families. UnifiedQA outperforms similarly sized or larger-parameter baselines, suggesting an advantage from larger pretraining datasets.
- Model setup: The analysis evaluates RoBERTa-base, ALBERT-xxlarge, and GPT-2 after fine-tuning them on four-class UnifiedQA multiple-choice questions.The models use the dev+val set for fine-tuning and are tested on the multitask test set.
- Results: 27.9% overall accuracy is achieved by RoBERTa-base, compared with 27.1% for ALBERT-xxlarge and 32.4% for GPT-2.RoBERTa-base scores 27.9% in humanities, 28.8% in social sciences, 27.0% in STEM, and 27.7% in other; ALBERT-xxlarge scores 27.2%, 25.7%, 27.7%, and 27.9%, respectively.
- UnifiedQA comparison: 43.7% accuracy is attained by 3-billion-parameter UnifiedQA, versus 32.4% for the similarly sized 1.5-billion-parameter GPT-2 model.The passage again suggests that T5’s larger pretraining dataset explains the advantage.
- UnifiedQA comparison: 29.3% accuracy is attained by UnifiedQA’s 60-million-parameter variant, exceeding RoBERTa-base and ALBERT-xxlarge despite using fewer parameters.The passage attributes this result to UnifiedQA’s larger pretraining dataset.
A.2 ERROR ANALYSIS · A.3 FORMAT SENSITIVITY · B TEST DETAILS
GPT-3 often makes high-confidence errors that can resemble plausible human mistakes, while UnifiedQA is more sensitive to input formatting. For UnifiedQA, removing a closing </s> token reduces accuracy by several percentage points, whereas increasing few-shot examples monotonically improves accuracy.
- A.2 ERROR ANALYSIS: GPT-3 sometimes makes high-confidence mistakes that a human might also make.For example, it answered 23 chromosomes with 97.5% confidence when the correct answer was 46.
- A.2 ERROR ANALYSIS: 97.5% confidence accompanied GPT-3’s incorrect prediction of 23 chromosomes instead of the correct 46.The prediction would have been correct if the question had asked for chromosome pairs.
- B TEST DETAILS: The supplied passages describe error analysis and format sensitivity, but provide no substantive test-details finding for section B.No passage supplied for B TEST DETAILS supports an additional claim.
- A.3 FORMAT SENSITIVITY: UnifiedQA is more sensitive to question formatting than GPT-3, whose accuracies often remain similar across formatting choices.UnifiedQA uses a normalized, lowercase question-and-choice format.
- A.3 FORMAT SENSITIVITY: Removing the </s> token from UnifiedQA’s input decreases accuracy by several percentage points.The specified format places the question and answer choices before the closing token.
- A.3 FORMAT SENSITIVITY: Few-shot accuracy increases monotonically as the number of instruction examples grows.Zero-shot performance is only somewhat lower than 5-shot accuracy.
B.1 TASK DESCRIPTIONS AND EXAMPLES
This section catalogs the test’s tasks and topics, provides an example for each task, and analyzes how question length relates to difficulty and model confidence. It also reports calibration differences between few-shot and zero-shot settings.
- Task descriptions and examples: Few-shot models are more calibrated than zero-shot models, but their accuracy-confidence gap still reaches up to 14%.Confidence-accuracy correlation is r = 0.81 in few-shot settings versus r = 0.63 in zero-shot settings.
- Task descriptions and examples: The paper lists all tasks and tested topics in Table 2 and provides an example for each task beginning with Figure 14.It also analyzes question length and difficulty in Figure 12.
- Task descriptions and examples: For questions longer than 280 characters, question length has a slightly positive correlation with true-label confidence, indicating that longer questions are not necessarily harder.The 280-character threshold corresponds to a tweet.
B.2 EXACT QUESTION AND ANSWER CONTAMINATION
The authors find no evidence that exact question-and-answer memorization substantially inflates test accuracy. Accuracy is not positively associated with question likelihood or compression, although models encountered related material during pretraining.
- Exact question and answer contamination: Memorized questions should have low entropy and high accuracy, but the observed relationship does not support this contamination pattern.The authors argue that memorization would make questions unusually predictable and answers unusually accurate.
- Exact question and answer contamination: The authors conclude that their exact questions were probably not memorized, although models encountered related text while processing Wikipedia during pretraining.Most questions came from PDFs or websites where questions and answers are available.
- Exact question and answer contamination: The authors will provide a list of question sources to reduce the chance that future models encounter exact test questions during pretraining.They also reference prior work concluding that contamination had little effect on performance.
- Exact question and answer contamination: r = −0.43 for zero-shot and r = −0.56 for few-shot settings, indicating no strong positive correlation between question log probability and accuracy.Higher log probability indicates higher compression, which would suggest memorization if especially large.