Source-linked AI summary
Measuring Mathematical Problem Solving With the MATH Dataset
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, Jacob Steinhardt
TL;DR
Mathematical problem-solving ability lacks a challenging benchmark for machine learning models. This paper introduces MATH and AMPS, finding that models achieve low accuracy despite scaling and that stronger progress will require algorithmic advances.
Problem
Machine learning models need a challenging benchmark for mathematical problem-solving beyond plug-and-chug calculation.
Method
The paper introduces MATH, a 12,500-problem benchmark with step-by-step solutions, alongside the AMPS mathematics pretraining corpus.
Results
Models achieved only 3.0%–6.9% accuracy on MATH, while training on step-by-step solutions increased relative accuracy by 10%.
Takeaways & Limitations
MATH establishes a benchmark for mathematical problem solving and shows that strong performance will require conceptual and algorithmic advances.
Takeaways & Limitations
Accuracy remains relatively low and increases only modestly with model size, making continued scaling impractical for strong MATH performance.
Abstract
from arXiv · showhide
Many intellectual endeavors require mathematical problem solving, but this skill remains beyond the capabilities of computers. To measure this ability in machine learning models, we introduce MATH, a new dataset of 12,500 challenging competition mathematics problems. Each problem in MATH has a full step-by-step solution which can be used to teach models to generate answer derivations and explanations. To facilitate future research and increase accuracy on MATH, we also contribute a large auxiliary pretraining dataset which helps teach models the fundamentals of mathematics. Even though we are able to increase accuracy on MATH, our results show that accuracy remains relatively low, even with enormous Transformer models. Moreover, we find that simply increasing budgets and model parameter counts will be impractical for achieving strong mathematical reasoning if scaling trends continue. While scaling Transformers is automatically solving most other text-based tasks, scaling is not currently solving MATH. To have more traction on mathematical problem solving we will likely need new algorithmic advancements from the broader research community.
1 Introduction
The paper introduces MATH as a benchmark for mathematical problem-solving ability, comprising challenging competition problems with exact-match answers and step-by-step solutions. Results show that current models perform poorly, while AMPS pretraining improves efficiency without resolving MATH’s difficulty.
- MATH dataset: MATH contains 12,500 high-school competition mathematics problems designed to measure models’ ability to analyze problems, select heuristics, and chain them into answers.Answers are unique after normalization, enabling exact-match evaluation; problems are tagged by difficulty from 1 to 5.
- MATH dataset: Each MATH problem includes a full step-by-step solution, allowing models to learn to generate derivations and making their outputs more interpretable.The solutions are written in natural language and LATEX and can provide intermediate computations as scratch space.
- Results: Large language models achieved only 3.0% to 6.9% accuracy on MATH, although they reached up to 15% on the easiest difficulty level and generated coherent solutions even when incorrect.A computer science PhD student who did not especially like mathematics attained approximately 40% accuracy.
- Results: Generating step-by-step solutions before the final answer decreased accuracy compared with immediately producing the answer, indicating that models did not yet use the solutions effectively at test time.The dataset’s scratch-space format therefore did not translate into improved test-time accuracy.
- AMPS pretraining: AMPS is a large mathematics pretraining corpus containing hundreds of thousands of natural-language and LATEX solutions from Khan Academy and Mathematica data.A 0.1 billion-parameter model pretrained with AMPS performed comparably to a fine-tuned model 130× larger.
- Limitations: Despite progress from AMPS pretraining and step-by-step-solution training, MATH accuracy remained relatively low and increased only modestly with model size.The authors characterize MATH as distinctly harder than most existing text-based tasks for large Transformers.
2 Related Work
Prior mathematical-reasoning research has focused on formal theorem-proving benchmarks and neural calculators, while broader evidence suggests Transformer scaling often improves performance on other tasks. This work situates MATH as a more challenging natural-language mathematical problem-solving benchmark than existing proof and plug-and-chug tasks.
- Neural Theorem Provers: Existing neural theorem-proving benchmarks include Coq’s 1,602 theorems and lemmas and HOList’s formally evaluable language.Polu and Sutskever instead use the Metamath formal language, as described in the supplied passage.
- Neural Theorem Provers: Formal theorem proving may require open-ended self-improvement, but models must also understand mathematics written by humans because manual formalization is time-consuming.The passage attributes these views to McAllester (2020) and Szegedy (2020).
- Benchmark Difficulty: Compared to existing proof and plug-and-chug tasks, the paper’s mathematical problem-solving task is considerably more challenging.Figure 2 compares MATH with HOList, HOLStep, DeepMind Math, and Symbolic Integration.
- Neural Calculators: Transformers achieve greater than 95% accuracy on algorithmically generated symbolic integration problems, while other work studies plug-and-chug mathematics through sequence-to-program generation.The cited studies include Lample and Charton (2020), Amini et al. (2019), and Ling et al. (2017).
- Benchmarks for Enormous Transformers: Transformer performance predictably increases with model size and dataset size, and code-generation models scale reliably across several orders of magnitude.The passage frames these results as motivating whether natural-language processing can be solved by increasing compute and funding.
3 Datasets
The paper introduces MATH, a 12,500-problem benchmark for mathematical problem solving, and AMPS, a large pretraining corpus of mathematics problems and solutions. MATH includes structured solutions and formatting that support automatic evaluation, while its representations also enable assessment on geometry problems.
- MATH dataset: MATH problems come from competitions such as AMC 10, AMC 12, and AIME, and generally require more than straightforward application of standard K-12 mathematics tools.The competitions assess the problem-solving ability of highly talented young mathematicians in the United States.
- MATH dataset: MATH contains 12,500 competition mathematics problems, split into 7,500 training and 5,000 test examples, each with a step-by-step solution and final boxed answer.The problems span seven subjects and five human-rated difficulty levels.
- MATH dataset: Consistent LaTeX and Asymptote formatting encodes mathematical problems, solutions, figures, diagrams, and graphics, enabling models to process visual mathematical content and geometry problems.Asymptote represents mathematical figures as vector graphics rather than raster images.
- MATH dataset: MATH’s boxed final answers and normalization rules allow generated solutions to be automatically assessed despite a combinatorially large output space.The rules accommodate equivalent fractions, decimal forms, factor orderings, optional units, and ignored spaces.
4 Experiments
Experiments show that MATH accuracy remains low and improves slowly with model size, while AMPS pretraining and partial or training-time solutions provide measurable benefits. By contrast, models’ self-generated solution scratch space can reduce accuracy and often contains logical mistakes.
- AMPS Pretraining: 5.4% accuracy for GPT-2 (0.1B) pretrained on AMPS nearly matches 5.2% for GPT-3 (13B) without AMPS, equating AMPS with roughly 130× more parameters.AMPS pretraining substantially improves performance on MATH.
- AMPS Pretraining: 6.0% accuracy with AMPS plus Math StackExchange is below 6.2% with AMPS alone, making AMPS more useful than the tested diverse real-world mathematics data.The StackExchange pretraining corpus was approximately 3 GB.
- Model Size: 6.9% accuracy for GPT-2 with 15× more parameters is only a 28% relative improvement over 5.4% for GPT-2 (0.1B), leaving absolute accuracy low.The results indicate that scaling helps, but slowly compared with the challenge of MATH.
- Step-by-Step Solutions: 5.3% accuracy resulted when GPT-2 (1.5B) generated a full solution before its final answer, indicating that self-generated scratch space degraded performance.The authors hypothesize that mistakes in partial solutions can snowball and derail subsequent text.
- Step-by-Step Solutions: 0.6% lower accuracy, to 6.3%, occurred when GPT-2 (1.5B) was fine-tuned only on final answers rather than examples including step-by-step solutions.The experiment shows that step-by-step solutions can benefit training even though models’ generated solutions often contain logical mistakes.
5 Conclusion · A Appendix
The paper establishes MATH as a benchmark for measuring mathematical problem solving and provides explanations and AMPS to support model learning. The appendix extends the paper with dataset comparisons, methodological discussions, additional details, difficulty analyses, and BART results.
- 5 Conclusion: MATH provides a benchmark for measuring mathematical problem-solving ability.The benchmark is intended to enable community-wide evaluation of this capability.
- 5 Conclusion: Every MATH problem includes an answer explanation that models can learn to generate step-by-step solutions.The dataset therefore contains both answers and derivations for supervised learning or evaluation.
- 5 Conclusion: AMPS is a diverse pretraining corpus designed to help future models learn virtually all K-12 mathematics.The corpus is presented as complementary support for mathematical problem-solving research.
- 5 Conclusion: The conclusion frames MATH and AMPS as groundwork for future machine-learning research on mathematical problem solving.The stated contributions are intended to support continued investigation rather than close the problem.
- A Appendix: The appendix compares the work with previous datasets and discusses logic and intelligence tests.These materials broaden the paper’s contextual and evaluative analysis.
- A Appendix: The appendix provides further details about AMPS and MATH.These additional details supplement the main descriptions of the datasets and benchmark.
- A Appendix: The appendix analyzes model performance as difficulty level changes.This examines how results vary across problem difficulty.
- A Appendix: The appendix reports results obtained with the BART architecture.This adds an architectural comparison to the paper’s model-performance analysis.
A.1 Expanded Dataset Comparisons
The appendix expands comparisons with ten datasets by describing plug-and-chug benchmarks, including Dolphin18K and MathQA. Dolphin18K draws from Yahoo! Answers, while MathQA builds on AQuA-RAT and addresses issues with its rationales.
- Expanded Dataset Comparisons: The authors compare MATH with ten datasets and further describe plug-and-chug datasets.These comparisons supplement the main paper’s dataset analysis.
- Expanded Dataset Comparisons: Dolphin18K, an early modern dataset, is based on Yahoo! Answers and includes elementary arithmetic questions.An example asks for the sum of 4 2/5 and 17 3/7.
- Expanded Dataset Comparisons: MathQA builds on AQuA-RAT, whose rationales are described as noisy, incomplete, and sometimes incorrect.The passage states that MathQA then cleans these rationales, but the provided text ends before specifying how.
A.2 Logic and Intelligence Tests
Although enormous Transformers perform poorly on MATH, they perform well on other logic and intelligence tests. LogiQA performance is improving toward human level, while GPT-3 benefits on C-Test from converting letter sequences to numbers.
- Enormous Transformers perform poorly on MATH but do well on other logic and intelligence tests.
- Transformers are improving on LogiQA and may attain human-level performance relatively soon if current trends continue.LogiQA tests logical reasoning through questions involving relationships and inferred attributes.
- 77%: The C-Test correlates with human IQ, and GPT-3 (175B) performs better when letter sequences are converted to numbers.The C-Test is a pattern-completion test; GPT-3 was evaluated in a 5-shot setting, with poor performance on letter sequences before conversion.
A.3 Further Dataset Information
The dataset extends mathematical problem solving to graphical figures through Asymptote code, while its auxiliary AMPS collection combines Khan Academy and Mathematica modules. Compared with DeepMind Mathematics, AMPS spans substantially more exercise types and modules.
- Rendering Graphics: Asymptote code lets text-based models process graphical mathematical figures without requiring multimodal models.Figures can be concisely specified in code, sidestepping multimodal-model complexity.
- AMPS Examples: AMPS combines examples from Khan Academy with 100 Mathematica modules.Concrete AMPS examples are shown in Figure 8.
- Contrasting AMPS and DeepMind Mathematics: AMPS includes several hundred exercise types, compared with only a few dozen in DeepMind Mathematics.Khan Academy contributes 693 modules and Mathematica contributes 100; many DeepMind exercises vary coefficients and constants across problem instances.
A.4 Difficulty Analysis
MATH accuracy varies substantially across difficulty levels and subjects, with human and machine difficulty tracking each other. Longer problem statements or ground-truth solutions also tend to correspond to greater difficulty.
- Difficulty and length: Human and machine difficulty track each other across MATH difficulty levels.The paper breaks down accuracy by difficulty level and reports this relationship in Figure 9.
- Difficulty and length: Accuracy varies substantially by difficulty level and subject.This variation is reported in Figure 10.
- Difficulty and length: Problems with long questions or ground-truth solutions tend to be more difficult than those with short questions or solutions.Figures 11a and 11b analyze accuracy against problem and solution length.
A.5 Results with the BART Architecture
The section evaluates BART, a bidirectional-encoder and unidirectional-decoder architecture whose tokenizer preserves LaTeX symbols, on mathematical problem solving. BART-Large achieves 4.9% on MATH’s test set, slightly below the smallest GPT-2 model, indicating no marked accuracy improvement.
- Architecture: BART uses a bidirectional encoder and unidirectional decoder, and its tokenizer preserves LATEX symbols unlike T5’s tokenizer.The architecture was selected to test whether an existing alternative to unidirectional decoder models could improve performance.
- Results: 4.9% on MATH’s test set is achieved by BART-Large (0.4B) after pretraining on AMPS and fine-tuning on MATH.This result is slightly worse than that of the smallest GPT-2 model.
- Results: Bidirectional-encoder and unidirectional-decoder models do not yield marked changes in MATH accuracy.The conclusion follows from BART-Large’s 4.9% test accuracy, which is slightly below the smallest GPT-2 model.
A.6 Further Human Evaluation Details
The human evaluation used six university computer-science students because MATH requires substantial mathematical background and time to solve. Participants completed twenty problems in sixty minutes without calculators, providing a rough estimate of human-level performance.
- Participants: Six university computer-science students participated because MATH demands strong mathematical background and substantial solving time.Four participants were paper authors, while the other two were friends or acquaintances; the survey was intended only to provide a rough sense of human-level performance.
- Evaluation protocol: Participants had sixty minutes to complete twenty mathematics problems without using a calculator.Participation was voluntary, and participants could withdraw at any time.
- Analysis: Human-difficulty patterns were compared with GPT-2 accuracy across subjects, difficulty levels, problem lengths, and solution lengths.The plotted results used GPT-2 (1.5B) and excluded problems with asymptote figures.
B Checklist Information
This section explains the legal basis, intended use, and licensing of MATH and AMPS. The authors rely on Fair Use for research and release the data and code under an MIT license.
- Legal Compliance: AMPS reuses Mathematica-generated problems and Khan Academy code under attribution-compatible terms, with the exercise framework following an MIT License.Mathematica is treated as a calculator for numerical outputs, while Khan Academy reuse requires attribution.
- Legal Compliance: MATH and AMPS are covered by Fair Use §107 because they support scholarship or research and are noncommercial uses.The authors specifically cite the purpose and character of use and its effect on the potential market or value of the copyrighted work.
- Dataset Intended Uses: The dataset is intended to forecast reasoning capabilities, not to train models that help students cheat on mathematics exams.The paper documents the dataset and provides code for reproducing results at the listed GitHub repository.
- Author Statement and License: MATH data, AMPS data, and the open-source code are released under an MIT license, with the authors bearing responsibility for rights violations.The licensing statement assigns responsibility to the authors in case rights are violated.