PaperlayerView exact revision on arXiv

Source-linked AI summary

Solving Quantitative Reasoning Problems with Language Models

Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, Vedant Misra

arXiv:2206.14858v2cs.CLcs.AIcs.LG

TL;DR

Language models have struggled with college-level quantitative reasoning. Minerva trains on technical content and achieves state-of-the-art results on technical tasks without external tools, while correctly answering nearly a third of over 200 undergraduate science problems.

  • Problem

    Large language models have struggled with quantitative reasoning tasks involving mathematics, science, and engineering problems.

    Source: §1 · p. 1

  • Method

    Minerva is a large language model trained on mathematical and technical content that solves quantitative problems through natural-language mathematical reasoning without external tools.

    Source: §2.1 · p. 5, §6 · p. 12

  • Results

    Minerva achieves state-of-the-art results on all considered technical tasks except GSM8k, with considerable improvement over previous results on the other tasks.

    Source: §3 · p. 9

  • Takeaways & Limitations

    Training large language models on high-quality mathematical data can produce strong performance on logical reasoning, numerical calculation, and symbolic manipulation.

    Source: §6 · p. 12

    The approach lacks automatic answer verification and is limited on tasks requiring complicated numerical calculations because it has no access to external tools.

    Source: §6.1 · p. 12

Abstract

from arXiv · show

Language models have achieved remarkable performance on a wide range of tasks that require natural language understanding. Nevertheless, state-of-the-art models have generally struggled with tasks that require quantitative reasoning, such as solving mathematics, science, and engineering problems at the college level. To help close this gap, we introduce Minerva, a large language model pretrained on general natural language data and further trained on technical content. The model achieves state-of-the-art performance on technical benchmarks without the use of external tools. We also evaluate our model on over two hundred undergraduate-level problems in physics, biology, chemistry, economics, and other sciences that require quantitative reasoning, and find that the model can correctly answer nearly a third of them.

1 Introduction

Minerva targets the persistent difficulty of quantitative reasoning in language models by generating self-contained, step-by-step solutions to mathematical and scientific problems without external tools. It combines technical-data training with evaluations spanning standard benchmarks and more than 200 undergraduate science and mathematics questions.

1 Introduction

  • Minerva addresses language models’ weakness on quantitative reasoning, which requires parsing problems, recalling relevant knowledge, performing computations, and producing precise mathematical tokens.These capabilities are presented as a testbed for robust solvers that could support scientific and technical work.Source: §1 · p. 1

1.1 Our Contribution

  • Minerva generates self-contained, step-by-step solutions with correct LATEX notation for natural-language mathematics and science questions, including over 200 undergraduate-level OCW problems.The OCW evaluation extends assessment beyond purely mathematical settings into chain-of-thought quantitative reasoning.Source: §1.1 · pp. 2–3, §1 · p. 2
  • Minerva extends PaLM models with scientific and mathematical training data, achieving state-of-the-art performance on MATH, GSM8k, and STEM MMLU in few-shot settings.The models use 8B, 62B, and 540B parameter starting points and are not explicitly trained on the evaluation datasets.Source: §1.1 · p. 2
  • The paper’s main novelty is a large dataset pairing natural language with formal mathematical language, including equations and diagrams, to improve quantitative reasoning through data quality and model scale.The dataset combines arXiv and carefully processed web content while minimizing loss of mathematical information.Source: §1.1 · p. 2

1.2 Related Works

  • Unlike approaches using calculators, code, or other external tools, this work evaluates models’ ability to solve quantitative problems through their own reasoning, with majority voting improving over greedy decoding.Related work includes scratchpad and chain-of-thought prompting, verifier-based reranking, code generation, and formal mathematics systems.Source: §1.2 · pp. 3, 5

2 Training and Evaluation

Minerva is trained on mathematical and general-language data, then evaluated across established benchmarks and 272 undergraduate STEM problems. Inference-time majority voting improves over greedy decoding, while its performance saturates faster than pass@k and exceeds log-likelihood reranking.

2.1 Mathematical Training Dataset

  • The technical training dataset contains 38.5B tokens from mathematically filtered webpages, arXiv papers, and general natural-language data, preserving mathematical notation for calculation and symbolic manipulation.Mathematical webpages were collected using MathJax expressions and cleaned while retaining LATEX symbols and formatting.Source: §2.1 · p. 5

2.2 Models and Training Procedure

  • Minerva starts from pretrained PaLM decoder-only transformers and finetunes them autoregressively on the mathematical dataset; the largest 540B-parameter model uses 26B continued-training tokens despite being highly undertrained.The training setup and hyperparameters are summarized in Table 2.Source: §2.2 · p. 6

2.3 Evaluation Datasets

  • Evaluation uses few-shot prompting, up to 1024 input tokens and 512 generated tokens, with final answers checked against ground truth using SymPy for mathematical equivalence.The benchmarks include MATH, GSM8k without external tools, and MMLU-STEM with standard and chain-of-thought prompting.Source: §2.3 · pp. 6–7

2.4 Undergraduate-Level STEM Problems

  • OCWCourses contains 272 automatically verifiable undergraduate STEM problems from MIT OpenCourseWare, including 191 numeric and 81 symbolic problems across multiple scientific courses.Proof-based and open-ended short-answer problems were excluded, and problems were processed to be self-contained with clearly delineated final answers.Source: §2.4 · p. 7

2.5 Inference-Time Techniques

  • Majority voting over multiple sampled solutions outperforms greedy decoding, reaches 97% of large-k accuracy at k = 64 for MATH and k = 16 for GSM8k, and beats log-likelihood reranking.Majority-voting performance saturates faster than pass@k, whose gains continue as k increases.Source: §2.5 · pp. 7–8

3 Results

Minerva achieves state-of-the-art results across the considered technical tasks, with considerable improvements over previous results except on GSM8k. Additional evaluations show 62B matching Poland’s 2021 national average, while finetuning benefits PaLM but not Minerva and 540B handles basic arithmetic with limited accuracy on longer additions.

3 Results

  • Minerva 62B scores 57% on Poland’s National Math Exam, matching the 2021 national average.Source: §3 · p. 8
  • Minerva achieves state-of-the-art results on all considered technical tasks, with considerable improvement over previous results except on GSM8k.Source: §3 · p. 9
  • Finetuning Minerva on MATH yields no improvement, whereas finetuning PaLM produces a significant improvement.This suggests that standard finetuning has lower marginal utility as unsupervised training data become more capable and diverse.Source: §3 · p. 9

3.1 Basic arithmetic

  • Minerva 540B achieves over 80% accuracy on 10-digit addition but over 20% on 18-digit addition.Source: §3.1 · p. 9

4 Performance Analysis

The analysis characterizes model failure modes and assesses false positives in quantitative reasoning evaluations. Larger models improve reasoning and calculation robustness, while false positives remain low overall but increase with difficulty and affect pass@256 estimates.

4.1 Model Mistakes

  • Among 216 high-confidence disagreements, Minerva 8B was correct in 15 samples and Minerva 62B was correct in 201, with both models mainly failing through reasoning or calculation errors.The comparison required the top answer to receive at least 15% of votes.Source: §4.1 · p. 9
  • Minerva 8B also made benign arithmetic mistakes, rarely gave answers that were too short, and occasionally hallucinated nonexistent equations or mathematical facts.Table 4 summarizes the 8B failure modes among samples solved correctly by the 62B model.Source: §4.1 · p. 9, Table 4 · p. 9

4.2 False Positives

  • False positives were overall low for Minerva 62B on MATH but increased with difficulty level.The estimate was based on manually inspecting 100 random MATH questions, 20 from each difficulty level.Source: §4.2 · p. 10, Table 5 · p. 10
  • 84.5% pass@256 accuracy for Minerva 62B includes false positives; after removing them, the estimated accuracy is bigger than 68%, with a 30% false-positive rate among selected samples.The selected samples were those failing majority voting but passing through a single correct answer.Source: §4.2 · p. 10

5 Memorization

The authors investigate whether Minerva’s quantitative performance reflects memorization rather than genuine solution synthesis, using dataset-overlap searches, modified problems, and solution-similarity analyses. Across these tests, they find little evidence that performance is attributable to memorization.

5 Memorization

  • Minerva’s solutions rely on memorized intermediate facts, such as square roots and trigonometric identities, while strong performance would require combining recall with genuine solution synthesis.The section distinguishes strong memorization of explicit evaluation problems from weaker memorization of alternate answers.Source: §5 · p. 10
  • The memorization analyses provide little evidence that Minerva’s MATH performance is driven by recalling explicit problems, alternate answers, or highly similar ground-truth solutions.The evaluation combines training-corpus searches, modified-problem robustness, and BLEU-overlap analyses.Source: §5 · p. 10, §5.3 · p. 11, §5.1 · p. 11

5.1 Training and Evaluation Dataset Overlap

  • None of the 100 high-majority-vote MATH questions matched the 250 highest-BLEU training-corpus segments inspected, suggesting these questions did not enter the data-collection process.The analysis searched Math Web Pages for training-data overlap, while noting that some MATH problems can be found on the web.Source: §5.1 · p. 11

5.2 Performance on Modified MATH Problems

  • Modified-problem accuracy correlated with original-problem accuracy without a clear bias toward the original formulation, suggesting minimal memorization.The 62B model was tested on 20 correctly answered problems modified through wording or numerical changes.Source: §5.2 · p. 11, Figure 5 · p. 11

5.3 BLEU Score Between Ground Truth and Generated Solutions

  • 160 of 5,000 MATH test questions had a correct sample with BLEU ≥80, but majority-vote performance remained robust after removing high-similarity samples.The 62B model was analyzed using 256 samples per problem; high-BLEU solutions tended to be short.Source: §5.3 · p. 11

6 Conclusions and Discussion · Appendix · A Detailed Contributions

The paper concludes that natural-language mathematical reasoning can enable strong quantitative reasoning without external tools, while emphasizing limitations in verification, numerical calculation, and capability control. It also discusses limited societal impact and documents the contributors’ roles across datasets, training, evaluation, experiments, analysis, writing, and project leadership.

6 Conclusions and Discussion

  • Natural-language mathematical reasoning enables strong logical, numerical, and symbolic performance through autoregressive sampling without external tools, complementing code-generating and formal approaches.The authors argue that future quantitative reasoning agents should combine useful elements from these approaches.Source: §6 · p. 12

6.1 Limitations of Our Approach

  • The approach lacks automatic answer verification, external calculators or interpreters for complicated calculations, and direct control over capabilities acquired from large-scale training.These limitations distinguish the approach from formal methods and constrain difficult quantitative reasoning tasks.Source: §6.1 · p. 12

6.2 Societal Impact

  • Minerva remains far below human performance and its outputs cannot be automatically verified, limiting current societal impact despite potential for broadly positive effects if these issues are solved.The authors identify an accessible, affordable math tutor as a possible application that could improve educational inequalities.Source: §6.2 · p. 12

A Detailed Contributions

  • Aitor and David prepared the Mathematical web pages and arXiv training datasets, while Vedant and Vinay collected OCWCourses and supervised contractors.These contributions covered the principal training-data preparation and collection activities.Source: A Detailed Contributions · p. 16
  • Aitor trained Minerva and led model ablations with David and Vedant; broader teams built training and evaluation infrastructure, collected evaluation datasets, and conducted evaluations.The infrastructure and evaluation work involved Aitor, Ambrose, David, Anders, Ethan, Henryk, Vinay, and Vedant.Source: A Detailed Contributions · p. 16
  • Teams conducted inference-time, fine-tuning, and memorization experiments, including ablation studies led by contributors across these experimental programs.The named contributors included Aitor, Anders, Behnam, Cem, Ethan, Guy, Henryk, Imanol, Vedant, and Vinay.Source: A Detailed Contributions · p. 16
  • The contributors analyzed model results, categorized mistakes and false positives, wrote the paper, and advised and led the project throughout its life cycle.Analysis included sample explorations, while writing and leadership were distributed across the named project contributors.Source: A Detailed Contributions · p. 16

B Training Dataset Details · B.1 arXiv · B.2 Mathematical web pages

The training dataset combines processed arXiv papers with mathematics-containing web pages. The arXiv corpus was filtered for quality, while web pages were selected and heuristically processed to extract mathematical content.

B Training Dataset Details

B.1 arXiv

  • The arXiv collection began with 2M papers through February 2021 in LATEX format, concatenating multiple files when present.Source: §B.1 · p. 16
  • After removing comments and material outside the main paper, metadata titles and abstracts were added to each arXiv document.Source: §B.1 · p. 16
  • Quality filters removed arXiv papers exceeding 75k tokens, exceeding 0.6 tokens per character on average, lacking \section headers, or becoming empty.Source: §B.1 · p. 16
  • The processed arXiv dataset contains 1.2M papers totaling 58GB.Source: §B.1 · p. 16

B.2 Mathematical web pages

  • Mathematical web pages were initially identified by raw HTML containing either "<math" or "MathJax-Element-", using pages available as of January 2022.Source: §B.2 · p. 16
  • Heuristics extracted mathematical content from TeX or AsciiMath in script tags and from TeX annotations inside MathML blocks.Source: §B.2 · pp. 16–17
  • About 80% of documents used TeX or AsciiMath, while the remaining 20% generally used discarded MathML; after extraction, cleanup, filtering, and English selection, the dataset totaled 60GB.Source: §B.2 · pp. 16–17

C Model and Training Procedure Details

Minerva models were obtained by unsupervised technical-data finetuning of pretrained PaLM models, using a specified context length, batch configuration, and learning-rate schedule. Training used progressively different step counts and TPU configurations across model sizes.

Model initialization and finetuning

Optimization schedule

  • The reciprocal square-root learning-rate schedule continued pretraining, followed by a 10x learning-rate reduction and 4% additional training steps.The 8B, 62B, and 540B models received 600k, 400k, and 383k unsupervised finetuning steps, respectively, after 1M, 520k, and 257k pretraining steps.Source: C Model and Training Procedure Details · p. 17

Infrastructure and duration

D MATH Evaluation Details … E Additional Evaluation Experiments

The MATH evaluation extracts and normalizes model answers before testing SymPy equivalence, which improves measured accuracy by around 1%. It also uses a four-shot prompt selected to fit most problems within a 1024-token context.

D.1 MATH Answer Normalization

  • Normalization removes formatting and representational differences so equivalent answers are not falsely labeled incorrect.The procedure was developed by inspecting ground-truth targets and samples from Minerva and OpenAI davinci-002 while avoiding changes that could create false positives.Source: §D.1 · p. 17
  • Final answers are extracted from complete responses, including chain-of-thought reasoning, using the standardized “Final Answer” pattern.The few-shot prompt asks for “Final Answer: The final answer is ANSWER. I hope it is correct.”Source: §D.1 · p. 17
  • The normalization code handles mathematical formatting such as boxed, bold, overlined, and LaTeX expressions before comparison.It also standardizes shorthand forms including fractions and square roots.Source: §D.1 · p. 18
  • Around 1%, SymPy equivalence improves overall MATH accuracy compared with exact string matching.Answers are parsed with SymPy and judged equivalent when simplification of their difference gives zero, with a 5-second timeout.Source: §D.1 · p. 18, Table 6 · p. 19

D.2 MATH Prompt

  • The MATH evaluation uses a four-shot prompt selected from eight random examples that excluded Asymptote plotting commands.Four examples were chosen so most problems fit within a context length of 1024, enabling comparisons with many models.Source: §D.2 · p. 19
  • The passages include a domain-of-expression problem whose final answer is [2,5), a determinant problem involving det A = 2 and det B = 12, and a weight-lifting word problem.Source: §D.2 · p. 19

E.1 Dependence of performance on number of generated samples … F.4 Problems in OCWCourses

The supplementary analyses characterize sampling, reranking, finetuning, and majority-voting behavior, while documenting the construction and contents of the OCWCourses evaluation dataset.

E.1 Dependence of performance on number of generated samples

  • Pass@k continues improving slowly, whereas majority-voting accuracy saturates quickly as the number of generated samples increases.The comparison is shown for MATH and GSM8k, with exact-string-match accuracies without SymPy processing.Source: Figure 6 · p. 20, §E.1 · p. 20

E.2 Log-Likelihood Reranking

  • Majority voting performs significantly better than reranking responses by the model’s assigned log-likelihood.The comparison does not use SymPy processing.Source: §E.2 · p. 20, Table 8 · p. 21

E.3 Finetuning on MATH

  • Finetuning does not improve Minerva on MATH, while improving PaLM but leaving it behind Minerva.Models were finetuned on MATH for 3000 steps using 0-shot, custom 4-shot, or random-example prompts; PaLM’s best model was selected after 50 training steps.Source: §E.3 · p. 21

E.4 Majority Voting Thresholds

  • For k = 64, the majority-voting analysis gives a rough p1 − p2 resolution of 0.25 and probes probabilities down to p1 > 0.06.The multinomial approximation explains why majority voting saturates even when pass@k continues improving.Source: §E.4 · pp. 21–22

F.1 Breakdown of courses

  • OCWCourses problems are broken down by course, with solution types separately categorized in accompanying tables.Table 10 presents the course breakdown, while Table 11 covers answer types.Source: §F.1 · p. 22, Table 10 · p. 22, Table 11 · p. 23

F.2 Contractor instructions

  • Contractors were instructed to produce clean, self-contained STEM problems and solutions in correct LaTeX, removing items requiring external references.Referenced material was to be incorporated when readily available; otherwise, the problem should be deleted.Source: §F.2 · p. 23, Figure 7 · p. 23

F.3 OCWCourses Prompt

  • The OCWCourses prompt presents worked examples spanning net-charge, complex-number, Laplace-transform, and differential-equation problems with formatted final answers.Examples specify answer formats such as +n or -n and ordered pairs, and include explanatory solution steps before the final answer.Source: §F.3 · p. 24

F.4 Problems in OCWCourses

  • The OCWCourses problems are provided separately from the main paper materials.The separate-file provision supplies the evaluation problems used by the dataset documentation.Source: §F.4 · p. 24

F.5 OCWCourses evaluation · G MMLU-STEM Evaluation Details

The evaluation details specify answer extraction and normalization for OCWCourses, alongside standard and chain-of-thought evaluation procedures for the 18-subtopic MMLU-STEM benchmark.

F.5 OCWCourses evaluation

  • OCWCourses grades symbolic expressions and equations by converting answers to SymPy quantities and checking equality programmatically, while numeric answers are unit-stripped and converted to floats.Source: §F.5 · p. 24
  • For numeric answers, values near zero are compared using an absolute-difference threshold of 0.01 times their mean; otherwise, evaluation uses numpy.isclose().Source: §F.5 · pp. 25, 27
  • OCWCourses evaluation extracts the final answer from model responses using the pattern “Final Answer: The final answer is ANSWER. I hope it is correct.”Source: §F.5 · p. 25

G MMLU-STEM Evaluation Details

H Arithmetic Results … I.2 Categorizing Model Mistakes

This section evaluates Minerva’s arithmetic performance across operations, digit lengths, and example solutions, while examining mistakes in arithmetic and algebra outputs. It categorizes recurring reasoning and calculation errors, incomplete responses, and untraceable mathematical objects, including contrasts between 8B and 62B outputs.

H Arithmetic Results

  • Minerva’s basic-arithmetic performance was measured on 500 randomly sampled questions for each operation and digit length, with results shown in Figure 8.The prompt included addition, multiplication, subtraction, and variable-operation arithmetic questions.Source: H Arithmetic Results · p. 30, Figure 8 · p. 30

I.1 Additional Correct Samples

  • The section presents correctly solved examples spanning eigenvalue calculations and MATH problems involving piecewise functions and indistinguishable plants.The supplied examples include answers of 10 for the function problem and 28 for the window-sill arrangement problem.Source: §I.1 · p. 31, Additional examples from MATH that are… · pp. 31–32

Additional examples from MATH that are correctly solved by the 62B model

  • The 62B model’s additional MATH examples correctly derive f(0) + f(1) + f(2) = 10 and the plant arrangement count = 28.The plant solution enumerates distributions across three window sills, while the model output independently reaches 28.Source: Additional examples from MATH that are… · pp. 31–32

I.2 Categorizing Model Mistakes

  • Failure labels cover incorrect reasoning, calculation, facts, question interpretation, insufficient justification, and hallucinated mathematical objects.The analysis used two samples per main category; categorization sometimes relied on human judgment and allowed multiple labels, while incorrect facts were separated when reasoning depended on an invalid contextual premise.Source: §I.2 · p. 33
  • In the polynomial-degree and arithmetic-sequence examples, the 8B model answers 8 and 6, whereas the 62B model gives 5 and 9, respectively.For the parallel-line example, 8B reports −3, while the cited calculation derives a = −8.Source: §I.2 · pp. 37–38
  • For polynomial degree, the 8B output incorrectly adds the degrees of f(x) and g(x), whereas the 62B output correctly identifies the degree as 5.The supplied outputs give 8B’s answer as 8 and 62B’s answer as 5.Source: §I.2 · p. 37
  • For the parallel-line problem, the 8B model equates coefficients without accounting for the second equation’s factor of 2, while the 62B derivation gives a = −8/5.The 8B output gives a = −3; the 62B calculation states 6a + 4 = a −4 and reports a = −8/5.Source: §I.2 · p. 38
  • The analysis labels solutions as short and wrong when key reasoning steps are missing, and flags mathematical objects that cannot be easily traced to the original task formulation.These labels are assigned by human raters during solution review.Source: §I.2 · p. 38

I.3 False Positive for pass@k … J Memorization Analysis Details

This section examines the reliability and reasoning quality underlying the model’s quantitative problem-solving performance across mathematics evaluations, including Polish, GCSE, and JEE Main problems. It also analyzes false positives in pass@k and the relationship between correct answers and valid reasoning.

I.3 False Positive for pass@k

  • For the 62B model, 2,000 tasks were solved by pass@256 but not majority voting, including 270 unique correct samples selected as the hardest cases.The analysis targets samples labeled correct by pass@k but not the majority answer, focusing on correct answers appearing only once.Source: §I.3 · p. 40
  • The sampled pass@k cases contained 44% correct answers, 16% correct answers without explained steps, 16% minor-mistake false positives, 16% clear false positives, and 8% graph-related false positives.The categories came from manually labeling 25 samples.Source: §I.3 · p. 40
  • The estimated false-positive rate is roughly 30%; extrapolation gives at least 65% accuracy when combining pass@k-only and majority-voting results.The estimate focuses on tasks with only one correct answer, although 90% of correct tasks have multiple samples with the correct final answer.Source: §I.3 · p. 40

Examples of Correct Answers without Explaining Steps

  • Examples show correct answers without fully explained steps, including outputs reaching 9 100 and 512 despite abbreviated or formula-heavy reasoning.The examples illustrate that the final answer can be correct even when the solution omits explanatory steps.Source: Examples of Correct Answers without… · p. 41

Examples of False Positives with minor mistakes

I.4.1 Polish National exam in Mathematics 2022

  • 19 of 26 multiple-choice problems were solved correctly on the Polish National Mathematics Exam 2022, with mathematically valid reasoning supporting most correct solutions.The evaluation included six text problems and excluded three graph-dependent problems; broader evaluation with in-context learning and other exam iterations was left for future work.Source: §I.4.1 · pp. 44–45
  • On the Polish exam probability example, the medium model answered 1/81 incorrectly, whereas the large model identified four valid cases and answered 4/81 correctly.The four cases are (3,8), (4,6), (6,4), and (8,3), among 81 equally likely pairs.Source: §I.4.1 · p. 46

I.4.2 GSCE Exam 2019 — Mathematics Higher

  • Minerva 540B scored 38 of 46 points (82.6%) on selected image-independent problems from the June 2019 GCSE Mathematics Higher exam.The evaluation counted only the top answer in majority voting.Source: §I.4.2 · p. 47
  • GCSE samples show that straightforward one-point tasks were solved correctly except one, while some complex tasks were also solved correctly.The examples additionally include a LATEX table completion and failure cases involving a conceptual error and a recoverable calculation mistake.Source: §I.4.2 · p. 47

I.4.3 JEE Main Exam 2020

  • The JEE Main 2020 problems exceeded the model’s current abilities, so the evaluation presents selected failure cases rather than a complete study.One featured polynomial-coefficient problem nevertheless concludes with the answer n = 30.Source: §I.4.3 · p. 49

J.1 Dataset overlap … BLEU = 80.02

The paper probes dataset and solution overlap, finding no evidence of training-data contamination while cautioning that degraded performance under larger question modifications is not clear evidence of memorization. Generated solutions rarely show high BLEU overlap with MATH targets, and high-overlap examples tend to be short.

J.1 Dataset overlap

  • The dataset-overlap analysis measured BLEU between 100 highly confident MATH questions and answers and 500-word training-data chunks.The 100 questions were those for which the model had the highest majority-vote confidence.Source: §J.1 · pp. 49–50, Figure 10 · p. 50
  • Manual inspection of the 500 most overlapping training segments found no evidence of dataset contamination.The segments came from training data partitioned into 500-word chunks and were selected from the overlap analysis.Source: §J.1 · pp. 49–50

J.2 Question modifications

  • The question-modification probe tested minor framing, number, larger framing, and combined changes to assess sensitivity to exact wording and distribution shift.Each modification type was evaluated by comparing accuracy before and after modification.Source: §J.2 · p. 50
  • Larger framing changes and combined number-and-framing changes somewhat degraded performance, but differing task difficulty prevents interpreting this as clear evidence of memorization.Accuracy was compared across 64 solution samples before and after four modification types on a 20-question subset.Source: §J.2 · p. 50, Figure 11 · p. 51

J.3 Solution overlap

  • Only 160 of 5,000 MATH test problems had at least one of 256 generated samples with BLEU greater than 80.The analysis compared model-generated solutions with ground-truth MATH solutions using BLEU and ROUGE.Source: §J.3 · p. 52
  • High-BLEU generated solutions tended to be shorter than typical solutions.Figure 13 compares the word-length distribution of samples with BLEU greater than 80 against all samples.Source: §J.3 · p. 52, Figure 13 · p. 53

BLEU = 100

  • Examples with BLEU = 100 show exact-looking overlap between model output and target solution fragments.The supplied examples include repeated solution text and the model-output phrase “We have that.”Source: BLEU = 100 · p. 52

BLEU = 89.97

  • The excerpts show the same trigonometric-algebraic solution fragment twice, followed by equivalent continuation lines, while a separate excerpt begins with “Model output: We have that”.Source: BLEU = 89.97 · p. 53, BLEU = 80.02 · p. 54

Related papers

Loading 2206.14858v2…