Source-linked AI summary
Solving Quantitative Reasoning Problems with Language Models
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, Vedant Misra
TL;DR
Language models have struggled with quantitative reasoning in mathematics, science, and engineering, motivating methods that can produce complete solutions. Minerva further trains pretrained language models on technical content, including mathematical notation, and achieves strong benchmark performance without external tools while also evaluating more than 200 undergraduate-level problems.
Problem
Language models have struggled with quantitative reasoning tasks such as solving college-level mathematics, science, and engineering problems.
Method
Minerva further trains pretrained language models on a high-quality technical dataset pairing natural language with mathematical content and uses autoregressive sampling without external tools.
Results
Minerva achieves state-of-the-art results on technical benchmarks, with considerable improvement over previous results on most tasks.
Takeaways & Limitations
Minerva provides a baseline for quantitative reasoning by combining increased data quality and model size, while generating natural-language solutions with formal mathematical notation.
Takeaways & Limitations
The approach lacks automatic answer verification, has limited ability on tasks requiring complicated numerical calculations, and offers little direct control over acquired capabilities.
Abstract
from arXiv · showhide
Language models have achieved remarkable performance on a wide range of tasks that require natural language understanding. Nevertheless, state-of-the-art models have generally struggled with tasks that require quantitative reasoning, such as solving mathematics, science, and engineering problems at the college level. To help close this gap, we introduce Minerva, a large language model pretrained on general natural language data and further trained on technical content. The model achieves state-of-the-art performance on technical benchmarks without the use of external tools. We also evaluate our model on over two hundred undergraduate-level problems in physics, biology, chemistry, economics, and other sciences that require quantitative reasoning, and find that the model can correctly answer nearly a third of them.
1 Introduction
Minerva addresses language models’ difficulty with quantitative reasoning by training on technical content and evaluating self-contained solutions across mathematics, science, and engineering. The paper introduces a data-quality and model-scale approach, reports strong benchmark performance, and extends evaluation to undergraduate science problems.
- 1 Introduction: Language models have struggled with mathematics, science, and engineering problems requiring quantitative reasoning despite strong performance on many natural-language tasks.Quantitative problems require parsing language, recalling relevant knowledge, performing computations, and generating precise mathematical tokens.
- 1 Introduction: Minerva applies technical-content training to quantitative reasoning problems requiring complete, self-contained solutions without external tools.The tasks include mathematics word problems, competition mathematics, and science and engineering problems.
- 1.1 Our Contribution: Minerva achieves state-of-the-art performance on MATH, GSM8k, and STEM MMLU benchmarks, with considerable improvement over previous results on most tasks.The reported exception is GSM8k, where the improvement is not described as considerable.
- 1.1 Our Contribution: The paper’s main novelty is a large dataset pairing natural language with correctly formatted formal mathematics, including equations and diagrams.The data comes from processed arXiv papers and webpages, with mathematical content preserved during cleaning.
- 1.1 Our Contribution: Over 200 undergraduate-level science and mathematics problems from MIT OpenCourseWare extend evaluation beyond pure mathematics into multi-step quantitative reasoning.The collection provides a chain-of-thought-context measure of quantitative reasoning abilities.
- 1.2 Related Works: Majority voting over multiple sampled solutions significantly improves performance over greedy decoding and outperforms log-likelihood reranking.The approach groups predictions by final answer and selects the most common answer.
2 Training and Evaluation
Minerva is trained on mathematical and scientific content preserved in formal notation, then evaluated across quantitative reasoning benchmarks using few-shot generation and inference-time sampling. The evaluation includes established math and STEM datasets, a curated undergraduate problem set, and majority-voting comparisons with greedy decoding.
- 2.1 Mathematical Training Dataset: 38.5B tokens from mathematically filtered webpages and arXiv papers formed the technical training dataset, with mathematical notation preserved.The cleaning process retained LATEX symbols and formatting, including complete mathematical formulae.
- 2.2 Models and Training Procedure: Minerva continued training pretrained PaLM decoder-only transformers on the mathematical dataset using an autoregressive objective.The model family included 8B, 62B, and 540B parameter versions.
- 2.3 Evaluation Datasets: The evaluation covered MATH, GSM8k, MMLU-STEM, and 272 curated undergraduate STEM problems with automatically verifiable numeric or symbolic solutions.The OCWCourses set contained 191 numeric and 81 symbolic problems.
- 2.5 Inference-Time Techniques: Majority voting samples multiple solutions, groups predictions by final answer, and selects the most common answer instead of using a single greedy sample.The method is denoted maj1@k.
- 2.5 Inference-Time Techniques: 97% of large-k majority-voting accuracy is reached at k = 64 for MATH and k = 16 for GSM8k.Unlike majority voting, pass@k continues improving as k increases because it benefits from the tail of the sampling distribution.
- 2.5 Inference-Time Techniques: Majority voting performs significantly better than log-likelihood reranking.
3 Results
Minerva’s evaluation reports strong performance across the considered quantitative reasoning tasks, including state-of-the-art results on the reported technical benchmarks. Additional experiments examine model outputs, OpenAI’s davinci-002 under matched conditions, arithmetic accuracy, and fine-tuning effects.
- 3 Results: Minerva achieves state-of-the-art results on both MATH and MMLU-STEM.Figure 4 breaks these results down by subtopic.
- 3 Results: 57% was Minerva 62B’s score on the National Math Exam in Poland, matching the national average in 2021.
- 3 Results: The combination of training data, scale, and inference techniques yields state-of-the-art results on all considered technical tasks.The paper reports considerable improvement over previous results for all tasks except GSM8k.
- 3 Results: Fine-tuning Minerva on MATH produced no improvement, whereas fine-tuning PaLM on MATH produced a significant improvement.The authors suggest that standard fine-tuning has lower marginal utility as unsupervised training data quality and diversity improve.
- 3 Results: Minerva 540B achieves over 80% accuracy on 10-digit addition and over 20% accuracy on 18-digit addition.
4 Performance Analysis
The analysis compares Minerva model sizes, characterizes their failure modes, and examines false positives in answer-based evaluation. Larger models improve reasoning and calculation robustness, while false positives remain a concern, especially for harder problems and pass@k.
- Model Mistakes: 201 samples distinguished Minerva 8B and 62B outcomes, with 15 cases where 8B was correct and 201 where 62B was correct.The comparison used high-confidence majority decisions with at least 15% of votes for the top answer.
- Model Mistakes: Incorrect reasoning and calculations were the prevailing failure modes for Minerva 8B, while overly short solutions and hallucinated facts were relatively rare.Many calculation errors were benign arithmetic mistakes.
- False Positives: False positives were manually estimated from 100 random MATH questions, and the overall rate was low but increased with difficulty.A false positive is a correct final answer supported by incomplete or incorrect reasoning.
- False Positives: 84.5% pass@256 accuracy for Minerva 62B included false positives; after adjustment, estimated accuracy was bigger than 68%.The estimated false positive rate was 30% among samples selected because they passed pass@k but failed majority voting.
5 Memorization
The memorization analysis tests whether Minerva’s MATH performance reflects rote recall rather than genuine problem-solving. Searches for training overlap, robustness to modified problems, and solution similarity provide complementary evidence about this possibility.
- Motivation: The central interpretive question is whether Minerva’s performance reflects genuine analytic capability or rote memorization.The analysis considers both memorizing explicit evaluation problems and solutions and memorizing alternate answers to the same questions.
- Approach: Three MATH analyses test memorization through training-corpus searches, modified-problem robustness, and overlap between ground-truth and generated solutions.Together, these analyses examine direct recall, sensitivity to changes, and answer similarity.
- Training-Corpus Search: The corpus search examined the 100 correctly answered problems with the highest majority-vote scores and manually inspected the 250 documents with highest BLEU matches.The selection assumed that high majority-vote scores might identify problems more likely to have been memorized.
- Modified Problems: Modified problems changed wording or numerical values, enabling comparison of sampled-solution accuracy before and after modification.The experiment randomly selected twenty problems that Minerva 62B answered correctly under majority voting.
- BLEU Similarity: 160 of 5,000 MATH questions had a sample with BLEU score at least 80, while majority-vote performance remained robust after removing highly similar samples.This robustness indicates that performance cannot be attributed to outputs very similar to ground-truth answers.
6 Conclusions and Discussion
The paper concludes that technical training enables Minerva to solve quantitative problems through natural-language mathematical reasoning without external tools. It also identifies verification, computational, and capability-control limitations, while noting that societal impact remains limited by current performance and reliability.
- 6 Conclusions and Discussion: Minerva solves quantitative problems using mathematical reasoning expressed in natural language, without external tools and with autoregressive sampling at inference.The approach targets logical reasoning, numerical calculation, and symbolic manipulation.
- 6 Conclusions and Discussion: The approach establishes a route toward agents that reason about quantitative problems, alongside code-generating models and formal methods.The paper presents these as complementary approaches toward a common goal.
- 6.1 Limitations of Our Approach: The approach cannot automatically verify answer correctness, lacks calculators or Python for complicated calculations, and offers little direct control over acquired capabilities.These limitations arise from the model’s evaluation and training setup.
- 6.2 Societal Impact: Minerva’s potential societal impact is limited because performance remains well below human performance and outputs cannot be automatically verified.The paper gives an accessible, affordable math tutor as a possible future application if these issues are solved.
- Contributions: The paper reports contributions from dataset preparation, model training, infrastructure, evaluations, experiments, analysis, and writing across the project team.The listed roles include preparation of the Mathematical web pages and arXiv datasets and memorization experiments.
B Training Dataset Details
The training dataset combines arXiv papers with mathematics-containing web pages, using filtering and extraction procedures intended to preserve mathematical content. The resulting web corpus retains selected TeX or AsciiMath representations while discarding other MathML content.
- Sources: The two main training-data sources are arXiv papers and web pages containing mathematics.The section describes separate collection and processing procedures for each source.
- arXiv Dataset: The arXiv dataset contains 2M papers through February 2021 in LATEX format, filtered by length, token density, section structure, and document completeness.Metadata supplied the title and abstract, while comments and selected document regions were removed.
- Web Dataset: Mathematics web pages were initially filtered using raw-HTML markers such as <math or MathJax-Element-.The pages were dated through January 2022 and processed with heuristics designed to extract mathematical content.
- Web Dataset: About 80% of web documents represented mathematics in TeX or AsciiMath inside script tags or MathML annotation blocks.The extraction retained TeX or AsciiMath content from these forms while excluding other MathML block content.
- Web Dataset: The remaining roughly 20% of documents used MathML formats that were discarded, yielding a final English dataset of 60GB after filtering and processing.Other content inside mathematical markup blocks was also removed when it duplicated encoded notation.
C Model and Training Procedure Details
Minerva is obtained by unsupervised finetuning of pretrained PaLM models on a technical dataset. The models use a 2048-token context and differ in training duration, batch size, and compute allocation.
- Minerva models start from pretrained PaLM models and undergo unsupervised finetuning on a technical dataset.
- The models use context length 2048, batch size 128 except for the 540B model’s batch size 32, and no dropout.
- The learning rate follows reciprocal square-root decay continued from pretraining, with model-specific pretraining and finetuning step counts.
- After a 10x learning-rate reduction, all models receive 4% additional training steps.
- Training used the t5x framework and v4 TPUs, requiring 14 days for 8B, 17 days for 62B, and 29 days for 540B.
D MATH Evaluation Details
MATH evaluation extracts final answers from chain-of-thought responses, normalizes formatting, and checks mathematical equivalence with SymPy. The evaluation uses a four-example prompt and reports roughly a 1% accuracy gain from SymPy equivalence.
- Answer extraction and normalization: The normalization function removes formatting and textual variations, including LaTeX wrappers, punctuation, units, and common shorthand TeX forms.
- Answer extraction and normalization: Final answers are extracted from model responses using a required “Final Answer” format after chain-of-thought reasoning.
- Answer extraction and normalization: SymPy equivalence parses normalized answers as LaTeX, subtracts the resulting expressions, simplifies them, and treats 5-second timeouts as nonequivalent.
- Evaluation effect: Around 1%: SymPy equivalence improves overall accuracy on MATH problems by approximately 1%.
- Prompting: The MATH evaluation uses a four-example prompt selected to fit most problems within a context length of 1024.
E.1 Dependence of performance on number of generated samples
The evaluation studies how performance changes with the number of generated samples and compares pass@k, majority voting, and reranking. Pass@k keeps improving with more samples, whereas majority voting saturates relatively quickly.
- Sample-count dependence: Pass@k continues improving as k increases, while majority voting saturates faster on MATH and GSM8k.
- Reranking comparison: Majority voting performs significantly better than log-likelihood reranking.
- Majority-voting thresholds: The analysis models sampled answers with a multinomial distribution and notes that probabilities below 1/N cannot be resolved from N draws.
- Majority-voting thresholds: For k = 64, the estimated resolution for p1 − p2 is 0.25, although the bound is described as a very rough estimate.
F.5 OCWCourses evaluation
OCWCourses evaluation extracts answers from chain-of-thought responses and grades them according to annotated answer types. The dataset covers 18 MMLU-STEM subtopics and uses task-specific prompting, including multiple-choice chain-of-thought prompts.
- Answer evaluation: OCWCourses answers are extracted using the required “Final Answer” format from responses that may include chain-of-thought reasoning.
- Answer evaluation: Automatically verifiable solutions are annotated as symbolicexpression, symbolicequation, or numeric and graded with type-specific functions.
- Answer evaluation: Numeric answers are converted to floats after unit removal and compared using a threshold near zero or numpy.isclose() otherwise.
- Dataset and prompting: MMLU-STEM evaluation covers 18 subtopics spanning mathematics, physics, chemistry, biology, computer science, engineering, and related areas.
- Dataset and prompting: The standard MMLU setup uses a 5-shot development-set prompt and selects the option with the highest score for pass@1.
- Dataset and prompting: Chain-of-thought scoring estimates the most probable choice by majority voting over the reduced answer set {A, B, C, D}.
H Arithmetic Results
Minerva’s arithmetic evaluation probes operations across increasing digit lengths, while selected examples illustrate both successful reasoning and recurring failure categories. The 8B model’s errors chiefly involve incorrect reasoning or calculations, with occasional unsupported mathematical facts or hallucinated objects.
- Arithmetic evaluation: The arithmetic prompts cover addition, multiplication, subtraction, and variable-digit expressions.The prompt examples range from two-digit operations to larger-number arithmetic and a generic operation template.
- Arithmetic evaluation: 500 questions were sampled for each operation and each choice of number of digits to evaluate basic arithmetic performance.The results are shown in Figure 8.
- Selected examples: In one example, the 62B model correctly solves a piecewise-function evaluation, obtaining f(0) + f(1) + f(2) = 10.The model evaluates each branch before summing the three values.
- Failure analysis: The error taxonomy distinguishes incorrect reasoning, incorrect calculation, incorrect facts, misunderstood questions, answers that are too short, and hallucinated mathematical objects.Some labels depend on human judgment, and more than one label could apply in some cases.
- Selected examples: In another example, the 8B model reports 4 instead of the target value 34 after mishandling the squared negative input.The 62B output preserves the correct calculation and final answer.
I.3 False Positive for pass@k
The false-positive analysis examines tasks solved by pass@k but not by majority voting, focusing on correct answers that appear only once. Manual review finds that a substantial share of these apparent successes are false positives, while others are correct but insufficiently explained.
- Sample selection: The analysis targets correct samples from pass@k that are not the majority answer, especially answers appearing only once.For the 62B model, this subset contains 270 samples among 2,000 tasks solved by pass@k but not majority voting.
- Manual classification: 25 samples were manually classified into five categories of false positives and correct solutions.The categories distinguish graph-related errors, wrong methods, minor reasoning mistakes, unexplained correct answers, and plainly correct answers.
- Manual classification: 16% were clear false positives in which the model produced the right answer using the wrong approach or method.Another 16% had the right reasoning but minor mistakes whose effects were uncertain.
- Manual classification: 44% of reviewed samples were simply correct, while 16% were correct answers without explaining steps.The latter could lose points under proper grading despite correct reasoning.
- Estimated impact: The density of false positives in the reviewed set was roughly 30%.The analysis estimates that false-positive correction would raise the majority-voting accuracy to at least 65% under its stated assumptions.
J.1 Dataset overlap
The analysis probes whether Minerva’s MATH performance reflects training-set overlap or memorization of exact problem formulations. It finds no evidence of dataset contamination, while larger wording and number changes produce somewhat degraded performance that is difficult to interpret.
- Dataset overlap: BLEU overlap was computed between 100 high-confidence MATH questions and target answers and 500-word chunks of the training dataset.The 100 questions were selected based on majority-vote confidence.
- Dataset overlap: Manual inspection of the 500 most overlapping text segments found no evidence of dataset contamination.The documents containing these segments were provided in supplementary data.
- Question modifications: Models were tested on minor framing, number, larger framing, and combined modifications, comparing accuracy across 64 solution samples before and after each change.The modifications were designed to probe sensitivity to exact wording, numerical values, and distribution shift.
- Question modifications: Larger framing and number-related modifications produced somewhat degraded performance, but the difficulty changes were not controlled well enough to establish memorization.The authors present this result as motivation for further research rather than definitive evidence.
- Solution overlap: 160 of 5,000 MATH problems had at least one of 256 model samples with BLEU above 80, and high-similarity samples tended to be shorter than typical solutions.The overlap analysis also examined BLEU and ROUGE similarity between generated and target solutions.