Source-linked AI summary
Training Verifiers to Solve Math Word Problems
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, John Schulman
TL;DR
Large language models still struggle with robust multi-step mathematical reasoning. This paper introduces GSM8K and trains verifiers to rank sampled solutions, finding that 6B verification slightly outperforms a finetuned 175B model.
Problem
Even the largest language models struggle with robust multi-step mathematical reasoning, despite strong performance on diverse tasks.
Method
The paper introduces GSM8K and trains verifiers to judge sampled model solutions, selecting the candidate ranked highest by the verifier.
Results
6B verification slightly outperforms a finetuned 175B model, providing a boost approximately equivalent to a 30x model size increase.
Takeaways & Limitations
Verification offers a scalable alternative to finetuning for improving mathematical reasoning performance on GSM8K.
Abstract
from arXiv · showhide
State-of-the-art language models can match human performance on many tasks, but they still struggle to robustly perform multi-step mathematical reasoning. To diagnose the failures of current models and support research, we introduce GSM8K, a dataset of 8.5K high quality linguistically diverse grade school math word problems. We find that even the largest transformer models fail to achieve high test performance, despite the conceptual simplicity of this problem distribution. To increase performance, we propose training verifiers to judge the correctness of model completions. At test time, we generate many candidate solutions and select the one ranked highest by the verifier. We demonstrate that verification significantly improves performance on GSM8K, and we provide strong empirical evidence that verification scales more effectively with increased data than a finetuning baseline.
1 Introduction
Large language models still struggle with multi-step mathematical reasoning because individual errors are difficult to recover from. The paper introduces GSM8K and verifier-based selection, reporting improved scaling and performance relative to finetuning.
- Motivation: Large language models falter on multi-step mathematical reasoning, with samples frequently containing catastrophic mistakes.This motivates methods beyond simply increasing model size.
- Motivation: Autoregressive models cannot correct errors during generation, so solutions that veer off-course quickly become unrecoverable.The paper identifies high sensitivity to individual mistakes as a significant challenge in mathematical reasoning.
- Method: The proposed verifiers evaluate generated solutions, then select the highest-ranked candidate from a fixed number of sampled solutions at test time.The paper argues that verification is generally simpler than generation and benefits from optionality.
- Dataset: GSM8K contains 8.5K curated grade school math questions and natural language solutions designed with high linguistic diversity and elementary concepts.State-of-the-art language models struggle on GSM8K primarily because of its problem diversity.
- Results: Verifiers provide approximately the same performance boost as a 30x model size increase versus finetuning, while scaling significantly better with increased data.Dropout also acts as a strong regularizer, significantly improving both finetuning and verification.
2 Dataset
GSM8K is an 8.5K-problem dataset of human-written grade-school math word problems, split into 7.5K training and 1K test problems. It emphasizes high quality, linguistic diversity, moderate difficulty, and natural-language solutions.
- Dataset composition: 8.5K human-written grade-school math problems are split into 7.5K training and 1K test problems, with solutions requiring 2–8 elementary calculation steps.A bright middle school student should be able to solve every problem.
- Design principles: Less than 2 percent of problems are estimated to contain breaking errors after extensive quality control based on workers’ answer agreement.The dataset avoids error-prone scraping by relying on human workers to create problems.
- Design principles: Problems are designed to be relatively unique rather than drawn from shared linguistic templates or superficial variations, making held-out test performance more relevant.This diversity aims to avoid a common weakness in other datasets.
- Design principles: Problems require no concepts beyond early Algebra and are challenging for large state-of-the-art language models without being completely intractable.The vast majority can be solved without explicitly defining a variable.
- Design principles: Solutions are collected in natural language, with writers encouraged to explain their work while retaining diverse linguistic styles.This format is intended to support analysis of language models’ internal monologues.
3 Related Work
Prior math word-problem datasets vary in size, solution format, templating, language quality, and problem complexity, motivating GSM8K’s design. Earlier solving methods used seq2seq or specialized architectures, while recent work explored additional pretraining and ranking-based selection of model completions.
- Math word-problem datasets: Earlier datasets were small or had limited solution formats, while larger alternatives suffered from templating or natural-language solution quality issues.Dolphin18K contains 18K problems but provides only equations or final answers; AQuA-RAT contains 100K problems and has templating and quality-control issues, while MathQA retains data quality problems.
- Math word-problem datasets: ASDiv emphasizes diversity and quality, principles shared by GSM8K, whereas GSM8K is larger, includes natural-language solutions, and requires more solving steps on average.ASDiv contains 2.3K math word problems; the passage also contrasts GSM8K with the larger and more complex MATH dataset, but does not provide the full comparison.
- Related reasoning datasets: Other reasoning datasets target symbolic mathematics, reading comprehension, or commonsense question answering, while GSM8K combines basic background knowledge with reading comprehension and logical reasoning.The passage identifies symbolic math, LogiQA, and CommonsenseQA as related reasoning-dataset areas and describes GSM8K’s overlapping requirements.
- Methods for solving math word problems: Classic math word-problem benchmarks were approached with recurrent seq2seq models, followed by specialized encoder-decoder architectures whose strongest results often used pretrained BERT-family encoders.The passage names recurrent seq2seq work, several specialized architectures, and large pretrained encoders from the BERT family.
- Pretraining methods: Recent methods further improve mathematical reasoning through additional pretraining on educational corpora, curricula, or masked subexpression prediction.The passage cites AMPS from Khan Academy problems and Mathematica scripts, internet-extracted curricula, and masked subexpression prediction.
- Verification and ranking: Ranking-based selection has been used for language-model completions, including human-preference ranking for storytelling and concurrent joint generation-and-ranking for math word problems.The concurrent math-word-problem approach is described as closely related to this work, sharing many fundamental similarities.
4 Methods
The paper compares low-temperature single-sample finetuning with high-temperature sampling and verifier-based selection. Verifiers are trained from generator-produced solutions, with token-level prediction and language modeling objectives providing strong design choices.
- Finetuning: Finetuning minimizes cross-entropy over training tokens and evaluates each problem with one low-temperature sample.The baseline uses GPT-3-family initialization, focusing mainly on 175B and 6B models.
- Verification: Verification samples multiple high-temperature solutions, scores them, and returns the candidate with the highest verifier score.At test time, the method samples 100 completions per problem before ranking them.
- Sampling and coverage: Test@1 uses temperature T = 0, whereas test@100 uses T = 0.7; test@1 improves approximately monotonically while test@100 peaks within the first few epochs.The authors therefore use 2-epoch models to generate verifier-training samples, because later training reduces solution diversity.
- Verification: The verifier pipeline finetunes a generator for 2 epochs, labels 100 sampled training completions per problem, and trains a verifier for 1 epoch.Solutions are labeled solely by whether they reach the correct final answer.
- Verifier ablations: Token-level verifiers ultimately outperform solution-level verifiers, despite initially slower and noisier training.Token-level prediction remains improving late in training, while solution-level prediction quickly shows overfitting signs.
- Verifier ablations: Including a language modeling objective strictly improves verification over using only the verification objective.The authors motivate this improvement by arguing that language-distribution knowledge helps discriminate between sampled solutions.
- Verifier ablations: A large generator with a small verifier performs significantly better than a small generator with a large verifier.Verification remains effective even when the verifier is much smaller than the generator.
5 Additional Experiments
Additional experiments show that verifier performance improves with more generated completions only up to a point, while voting among top-ranked solutions offers another strategy. Dropout substantially benefits both finetuning and verifier training by improving performance and mitigating overfitting.
- Completion Search: 6B verifier performance improves as completions increase up to 400 per test problem, then decreases beyond 400.The decline suggests search benefits are eventually outweighed by adversarial solutions.
- Majority Voting: Majority voting selects the final answer receiving the most votes among the top verifier-ranked solutions.The voting process considers only the final answers reached by individual solutions.
- Majority Voting: With 100 samples, allowing only the top 3-5 samples to vote is optimal.With more starting samples, the voting pool can also be larger.
- Dropout Ablations: 20% residual dropout strongly improves finetuning and verification performance.The dropout rate was selected from a hyperparameter sweep, and additional pretraining with dropout was performed because GPT-3 models lack pretrained dropout.
- Dropout Ablations: Dropout significantly improves solution-level verifiers and mitigates overfitting in the unregularized baseline.The experiments compare dropout with solution-level and token-level verifier variants.
6 Conclusion
Verification significantly improves performance over finetuning: on the full dataset, a 6B verifier slightly outperforms a finetuned 175B model, while token-level verifiers are less prone to overfitting and residual dropout benefits all methods.
- Conclusion: 6B verification slightly outperforms a finetuned 175B model on the full dataset, providing a boost approximately equivalent to a 30x model size increase.Verification provides a significant performance boost relative to a finetuning baseline.
- Conclusion: Token-level verifiers are less prone to overfitting than solution-level verifiers.
- Conclusion: All methods benefit from regularization with residual dropout.
A Dataset Details
The dataset began with contractor-written math problems and solutions, then scaled collection through Surge AI and re-solving-based answer checks. Contractors received optional GPT-3-generated seed questions and guidance to produce descriptive, non-repetitive problems and solutions.
- Collection and validation: The authors initially collected 1,000 problems and natural-language solutions from freelance Upwork contractors before scaling collection with Surge AI.After full-dataset collection, workers re-solved all problems without re-solving problems they had originally written, and final answers were checked against the original solutions.
- Contractor guidance: Contractors could directly use, modify, or draw inspiration from seed questions generated by a few-shot-prompted 175B GPT-3 model.They were also instructed to write descriptive solutions and avoid reusing problem settings or templates across questions.
B Hyperparameters
The experiments used a fixed set of hyperparameters, with specified exceptions for selected figures. Sweeps and ablations found negligible gains from substantially changing learning rate, batch size, verifier temperature, or objective.
- Hyperparameter robustness: Learning-rate and batch-size sweeps over an order of magnitude in both directions produced no significant improvements.The sweeps were conducted around the tabulated values.
- Hyperparameter robustness: Alternative verifier temperatures, such as 1.0 instead of 0.7, had negligible effect in ablations.The reported comparison tested reasonable choices for verifier temperature.
- Hyperparameter robustness: Using cross-entropy instead of mean squared error also had negligible effect in ablations.This comparison concerns the verifier objective.
- Experimental exceptions: The tabulated hyperparameters applied to all experiments unless explicitly noted otherwise.Figure 8c used 4x more tokens per batch and 300 completions during training and testing; Figure 8 used 20% dropout, and Figure 7a trained verifiers on 100 completions while searching over more at test time.
C Calculator Annotations · D Example Model Solutions
The calculator annotations are automatically generated and used to override sampling during testing, but imperfect annotation coverage and minor implementation bugs slightly limit reported performance. The paper also presents selected examples comparing finetuning and verification at 6B and 175B scales.
- C Calculator Annotations: Calculator annotations combine hard-coded logic with a finetuned language model rather than human contractor annotations.The generation logic is highly unlikely to produce incorrect annotations but can omit lines that could be annotated.
- C Calculator Annotations: During training, annotated and unannotated tokens are treated identically, while testing overrides tokens after “=” and inside <<. . . >>.The override occurs only when a well-formatted annotation exists.
- C Calculator Annotations: Python eval simulates the calculator by evaluating expression tokens, with timeouts or errors causing annotations to be skipped.When annotations are skipped, the model is sampled normally.
- C Calculator Annotations: Less than 1% is the typical magnitude of underestimation from calculator bugs, while fixing them improves full-training-set verification performance by about 1%.The original calculator was used for all reported results, so those test results are slight underestimates.
- C Calculator Annotations: Figure 9 depicts the calculator sampling procedure used to integrate calculator annotations into model generation.The figure provides a diagram of the sampling process.
- D Example Model Solutions: The paper showcases slightly cherry-picked samples comparing finetuning and verification at both 6B and 175B scale.The samples were selected for diversity.
E Verifier Details
The verifier is a language model trained jointly to predict language and classify completion correctness. Its scalar prediction head uses a reserved vocabulary token, and verifiers are generally initialized from their corresponding generators.
- Architecture and objective: Verifiers jointly optimize language modeling and completion-correctness classification, using a per-token scalar prediction head.The verifier is a language model augmented with a small scalar head that predicts whether a model completion is correct or incorrect.
- Architecture and objective: The scalar head shifts and scales one special vocabulary-token logit with a bias and gain, leaving other-token logits available for language modeling.This implements verification without repurposing the logits for the remaining vocabulary tokens.
- Initialization: Verifier initialization from the generator performed slightly better in ablations and is the default for experiments.The alternative is initialization from the same pretrained language model used before generator finetuning.
- Training objective: The joint objective equally mixes language and verifier data, effectively upsampling original language data 100-fold.The verifier loss and language-modeling loss are added without weighting, and an epoch is defined by seeing each verifier example once.
- Training objective: Joint-objective training masks question tokens and computes loss only on solution tokens.This loss masking is illustrated in the joint training objective visualization.
F Verifier Visualization
Token-level verifier visualizations make verifier decisions interpretable by showing predicted correctness values for each token. The examples illustrate calibrated confidence, false positives and negatives, and errors involving ambiguous language or quantity binding.
- Visualization: Token-level verifier visualizations show predicted correctness values for each token, with green indicating high scores and red indicating low scores.The visualization uses a 175B token-level verifier trained on the full training set and presents five model completions.
- Visualization: The verifier’s prediction can be compared with actual completion correctness, and disagreements identify verifier errors.The second table column gives the verifier’s prediction, while the third reports whether the completion was actually correct.
- Examples: In a true-positive example, verifier confidence increases as the solution progresses, potentially because training included many incorrect model-generated samples.The first row shows the verifier correctly classifying a correct completion while initially remaining uncertain.
- Examples: The examples also show a false positive caused by subtracting 400 from the diamond’s price instead of the gold’s, reflecting occasional quantity-binding errors.In another incorrect completion, the verifier becomes highly confident after the solution explicitly states 64 dollars instead of 64 + 16 + 8 = 88 dollars.