Source-linked AI summary
A Careful Examination of Large Language Model Performance on Grade School Arithmetic
Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati, Summer Yue
TL;DR
Public math benchmarks may overstate LLM reasoning when benchmark-like data contaminates training. The paper constructs and evaluates the human-created GSM1k benchmark against GSM8k, finding drops of up to 8% for some models, systematic overfitting in several families, and minimal overfitting among frontier models while models still generalize to novel problems.
Problem
Public benchmarks such as GSM8k may contain training-like examples, limiting how confidently their scores measure genuine mathematical reasoning.
Method
The paper creates 1205 human-authored GSM1k problems matched to GSM8k and benchmarks open- and closed-source LLMs under standardized evaluation.
Results
Accuracy drops reach 8% on GSM1k for the worst models, several families show systematic overfitting, and frontier models show little evidence of overfitting while models still solve novel problems.
Takeaways & Limitations
GSM8k performance can include contamination-related overfitting, but overfitting does not imply that models lack the ability to reason on new grade-school problems.
Takeaways & Limitations
GSM1k and GSM8k are highly similar but not identically distributed, so the comparison approximates an uncontaminated GSM8k evaluation.
Abstract
from arXiv · showhide
Large language models (LLMs) have achieved impressive success on many benchmarks for mathematical reasoning. However, there is growing concern that some of this performance actually reflects dataset contamination, where data closely resembling benchmark questions leaks into the training data, instead of true reasoning ability. To investigate this claim rigorously, we commission Grade School Math 1000 (GSM1k). GSM1k is designed to mirror the style and complexity of the established GSM8k benchmark, the gold standard for measuring elementary mathematical reasoning. We ensure that the two benchmarks are comparable across important metrics such as human solve rates, number of steps in solution, answer magnitude, and more. When evaluating leading open- and closed-source LLMs on GSM1k, we observe accuracy drops of up to 8%, with several families of models showing evidence of systematic overfitting across almost all model sizes. Further analysis suggests a positive relationship (Spearman's r^2 = 0.36) between a model's probability of generating an example from GSM8k and its performance gap between GSM8k and GSM1k, suggesting that some models may have partially memorized GSM8k. Nevertheless, many models, especially those on the frontier, show minimal signs of overfitting, and all models broadly demonstrate generalization to novel math problems guaranteed to not be in their training data.
1 Introduction
The paper introduces GSM1k to test whether GSM8k performance reflects genuine reasoning or benchmark contamination. Evaluations reveal substantial drops for some models, while frontier models show little overfitting and models generally still solve novel problems.
- Motivation: Benchmark contamination may inflate apparent LLM reasoning performance when training data contains questions resembling public evaluations.The concern motivates testing established benchmarks such as GSM8k with newly constructed problems.
- Contribution: GSM1k is a human-created collection of 1205 grade-school math problems designed to mirror GSM8k’s difficulty and distribution.The dataset was created without LLM or synthetic-data assistance.
- Results: 8% worse accuracy was observed on GSM1k than GSM8k for the worst-performing models.The models evaluated included leading open- and closed-source families.
- Results: Several model families show consistent overfitting across nearly all versions and sizes, while frontier models show minimal signs of overfitting.The reported pattern distinguishes systematic overfitting from comparatively robust frontier performance.
- Release policy: GSM1k is being withheld temporarily to reduce the risk of future benchmark contamination, with evaluation code intended for public release.The authors specify release conditions for the benchmark.
2 Related Work
Prior work treats benchmark contamination as a serious threat to evaluating model reasoning and proposes similarity-based, functional, and newly generated variants to detect or reduce it.
- Benchmark comparison: This paper adapts the logic of prior ImageNet overfitting studies to compare GSM8k with a newly constructed grade-school mathematics benchmark.GSM8k contains approximately 8.5k grade-school math problems.
- Data contamination: Benchmark contamination is a recognized problem, prompting efforts to remove training data with high n-gram or embedding similarity to evaluation sets.These approaches aim to reduce the likelihood that benchmark-like material enters model training data.
- Detection and prevention: Related methods test whether models favor original benchmark wording as a contamination signal or generate fresh instances through functional evaluations.Functional evaluations create new problem instances that are not reused.
3 GSM1k
GSM1k was constructed by human annotators and processed to match GSM8k on problem difficulty and answer magnitude. Human and pre-2021 model checks support their similarity, while the benchmark remains unreleased to limit contamination.
- Dataset construction: 1205 GSM1k problems were created by human annotators to resemble GSM8k and require only elementary arithmetic.No language models were used to construct the dataset.
- Release policy: The benchmark is withheld until specified performance or date conditions are met, limiting opportunities for future contamination.The authors also commit to releasing the evaluation framework.
- Caveat: GSM1k and GSM8k are highly similar but not identically distributed, so comparisons approximate an evaluation setting without a publicly released GSM8k test set.The authors recommend interpreting results with this scope boundary in mind.
- Quality control: Three review layers checked problem correctness, grade-school content, formatting, and agreement between independent solutions.Problems with discrepant answers were discarded.
- Difficulty matching: Problem requests were allocated by estimated resolution-step difficulty to match GSM8k’s difficulty distribution.Difficulty was treated as an estimate rather than a perfectly precise quantity.
- Answer magnitude: Answer magnitudes were matched by discarding candidate problems until the surviving 1205 approximated GSM8k’s distribution.Initial generated problems tended to have slightly larger answers.
- Validation: Human annotators solved an average of 4.07 ± 0.93 GSM8k problems versus 4.36 ± 1.11 GSM1k problems in 15 minutes.The comparison was designed to assess whether the benchmarks had similar human solve rates.
- Validation: GPT-NeoX-20B and GPT-2, released before GSM8k, showed minimal differences between GSM8k and GSM1k solve rates.These models served as checks for contamination-free evaluation behavior.
4 Results
The evaluation compares models on GSM8k and GSM1k under standardized prompting and examines answer extraction and prompting variations. Many models fall below the no-overfit comparison, while trends remain consistent across ablations.
- Evaluation setup: Models were evaluated with the same five GSM8k training examples for both benchmarks, using up to 1000 generated tokens to avoid truncating reasoning.Open-source models were evaluated at temperature 0 for reproducibility.
- Benchmark comparison: Many models lie below the no-overfit line, whereas some Claude models exceed it because GSM1k is slightly easier than GSM8k.The figure focuses on models above 70% GSM8k accuracy.
- Scoring: Automatic scoring extracts the final numeric answer, and human answer extraction did not materially change findings for the examined models.The ablation addressed formatting mismatches that automatic evaluation marked incorrect.
- Robustness: Standardized prompting can produce GSM8k accuracies below individually reported benchmarks, but overfitting trends persist across prompt ablations.The authors varied prompt formulation and the number and source of n-shot examples.
- Results: Many lesser-known models near the top of the OpenLLMLeaderboard perform substantially worse on GSM1k, suggesting benchmark gaming rather than improved reasoning.Models were partitioned by GSM8k performance for fair comparisons.
5 Analysis
The analysis finds systematic benchmark overfitting in several model families, while frontier models generally show little overfitting and overfit models can still solve novel problems. A positive likelihood–performance-gap relationship supports partial GSM8k memorization, but outliers indicate contamination is not the full explanation.
- Lesson 1: Some Model Families are Systematically Overfit: Phi and Mistral models perform stronger on GSM8k than GSM1k across almost every release and model scale.Yi, Xwin, Gemma, and CodeLlama show the same pattern to a lesser extent.
- Lesson 2: Other Models, Especially Frontier Models, Show No Signs of Overfitting: Frontier and close-to-frontier models generally perform similarly on GSM8k and GSM1k, showing minimal signs of overfitting.The authors offer stronger reasoning and more careful contamination control as possible explanations.
- Lesson 2: Other Models, Especially Frontier Models, Show No Signs of Overfitting: Mistral Large is the only Mistral-family model reported without overfitting, supporting the hypothesis that stronger models can generalize despite benchmark exposure.The authors present this as evidence favoring learned elementary reasoning over uniquely careful contamination control, while acknowledging uncertainty about training data.
- Lesson 3: Overfit Models Are Still Capable of Reasoning: A 6% Phi-2 accuracy drop still leaves it solving over half of GSM1k, whose problems were guaranteed absent from its training distribution.This performance is similar to Llama2-70B despite Phi-2 having over 25 times fewer parameters.
- Lesson 4: Data Contamination Is Likely Not The Full Explanation for Overfitting: Spearman’s rank correlation of 0.36 links per-character GSM8k log-likelihood with the GSM8k–GSM1k performance gap, consistent with partial test-set memorization.Each percentage-point performance difference is associated with a 1.2 × 10^-2 increase in per-character log-likelihood; p = 0.03.
- Lesson 4: Data Contamination Is Likely Not The Full Explanation for Overfitting: Outliers show that contamination is likely not the full explanation: models with very different log-likelihoods can have similar overfit, and indirect training effects may matter.The examples include Mixtral variants, Math-Shepherd-Mistral-7B-RL, and Llema models.
6 Discussion
The discussion concludes that GSM1k reveals benchmark overfitting alongside substantial generalizable mathematical reasoning, with frontier models showing little evidence of overfitting. The paper also describes its scope, reproducibility plans, and staged release policy for the unreleased benchmark.
- 6 Discussion: GSM1k results show benchmark contamination evidence, systematic overfitting in several model families, minimal frontier-model overfitting, and strong reasoning on novel problems.These conclusions summarize the paper’s reported discussion and results.
- 6 Discussion: The paper states that its contributions and scope, limitations, experimental details, compute resources, and asset documentation are addressed in the submission materials.The checklist records code and data availability information, statistical reporting, and evaluation infrastructure.
- 6 Discussion: All GSM1k problems were created by hired human annotators, who were paid 20–25 dollars per hour, with approximately $180,000 spent on annotation.The checklist identifies Scale AI annotators as the problem creators.
B Dataset Documentation
GSM1k is a held-out dataset of 1205 elementary arithmetic problems created by human annotators to mirror GSM8k while avoiding model-generated contamination. Its instructions require original, grade-school-solvable problems with integer answers and basic arithmetic.
- Construction and intent: 1205 problems comprise GSM1k, a held-out version of GSM8k intended to measure data contamination.The dataset contains elementary mathematical reasoning problems and largely mimics GSM8k’s format and style.
- Creation process: Human annotators created GSM1k from scratch without using language models or chatbots.Annotators were shown GSM8k examples for guidance but were instructed to create original, non-paraphrased questions.
- Answer format: Final answers had to be single non-negative integers, with units specified in the question when relevant.Intermediate steps could include simple decimals, but final answers could not be negative or zero.
- Annotator instructions: Annotators were instructed to produce novel questions resembling 8th-grade math quizzes while avoiding repeated premises and example paraphrases.The workflow included reviewing examples, creating problems, and following step guidance.
- Problem constraints: Problems had to be solvable with addition, subtraction, multiplication, and division using calculations suitable for pen-and-paper work.The instructions targeted an 8th grader and limited solutions to elementary arithmetic operations.
- Evaluation prompt: Each evaluation question uses five randomly selected GSM8k training examples as n-shot prompts.The examples vary across questions, and the procedure follows a common GSM8k evaluation approach.
E Results with an Alternative Prompt
Under an alternative prompt scheme, model-specific results vary substantially, but the overall pattern of which model families appear overfit remains similar. The evaluation compares GSM8k and GSM1k performance under this prompt-based analysis.
- Prompt design: The alternative prompt differs from the standard n-shot setup mainly by using non-GSM8k guiding examples and a different answer format.It is described as a chain-of-thought prompt in the LM Evaluation Harness, but its principal differences are prompt examples and formatting.
- Alternative prompt results: Prompt choice introduces significant variance in model results, but the general trend of overfit model families is similar.The paper reports this pattern in Table F for the alternative prompt analysis.
F Results Table
Table F reports model performance on GSM8k and GSM1k, sorting models by their performance difference while emphasizing the comparison between the two benchmarks.
- Comparison basis: Models are sorted by the difference between GSM8k and GSM1k performance.The analysis focuses primarily on this difference rather than matching previously reported benchmark numbers.
- Table structure: Table F contains model difference, GSM8k accuracy, GSM1k accuracy, z-score, and p-value columns.The table header defines the reported fields for each model.
- Reported rows: For gemini-1.5-flash-preview-0514, GSM8k accuracy is 0.797 and GSM1k accuracy is 0.835, with a difference of -0.038.The row also reports a z-score of -2.507 and p-value of 0.994.
- Reported rows: For gemini-1.5-flash-preview-0514, GSM8k accuracy is 0.810 and GSM1k accuracy is 0.808, with a difference of 0.001.The row also reports a z-score of 0.110 and p-value of 0.456.
G 50 Examples from GSM1k
The GSM1k examples span diverse elementary arithmetic word problems, including budgeting, rates, proportions, time, counting, and multi-step totals. A corrected table addresses an earlier inclusion error.
- Correction: A previous paper version mistakenly included questions from a nonfinal GSM1k version, and the displayed table was corrected.The correction is explicitly identified before the example material.
- Counting and totals: The collection also contains counting and multi-step quantity problems involving books, chapters, paragraphs, points, jellybeans, boxes, and table legs.These problems combine several arithmetic operations while retaining an elementary word-problem format.
- Financial arithmetic: The examples include budgeting and shopping problems requiring arithmetic with prices, discounts, wages, and expenses.Examples cover groceries, clothing, shoes, flowers, pet supplies, chores, and saving for purchases.
- Rates and units: Several examples require rate and unit calculations, such as production, travel capacity, labor, and material requirements.Tasks involve coal carts, airline baggage fees, tile coverage, painting, lawn mowing, and excavation.
- Time and proportions: Other examples test time, averages, proportions, and sequential arithmetic over days or weeks.Examples include break duration, album schedules, yearly string costs, car savings, reading targets, and weekly earnings.
J Ablation with Human Answer Extraction
Automatic answer extraction can misclassify correct model outputs because it relies on brittle exact string matching. Human extraction changes absolute scores but not the measured overfitting difference between GSM8k and GSM1k for the analyzed models.
- Extraction limitations: Automatic extraction takes the last number produced and compares it with the gold answer using exact string matching.This can mark answers with extraneous trailing text as incorrect.
- Human extraction: Human annotators extracted answers for a subset of high-performing GSM1k models to measure extraction-error effects.The subset was selected for constructing a leaderboard of top-performing models.
- Human extraction: Human versus automatic extraction changed absolute performance numbers but produced no meaningful difference in GSM8k–GSM1k overfitting for this model subset.The comparison was limited to the analyzed high-performing models.
K Ablations with the Alternative Format
The alternative-format ablations test whether prompt wording and example selection affect apparent overfitting. Results vary substantially with the chosen few-shot examples, although the overall pattern remains largely consistent.
- Alternative format: The ablation reconstructs alternative chain-of-thought prompts using randomly selected GSM8k problems in the alternative answer format.This tests whether performance differences stem from using non-GSM8k examples in the alternative prompt.
- Results: Significant variance appears even when only the selected few-shot examples are changed.The broad finding shape remains largely consistent, but exact rankings and numerical values are highly prompt dependent.
- Results: Figure 14 arranges models from the most overfit families by their GSM8k-to-GSM1k performance drop under the first alternative-format prompt.Lower values indicate worse performance according to the figure caption.
L Ablations with Number of Fewshot Examples
The study varies the number of GSM8k few-shot examples to test sensitivity to prompt size. It evaluates the most overfit model families from 1 through 10 examples, with the primary analysis using five.
- Experimental design: The ablation evaluates the most overfit model families with 1 through 10 GSM8k few-shot examples in the standard prompt format.The number of examples is varied inclusively across n=1 to n=10.
- Experimental design: Few-shot examples are randomly selected from GSM8k training data and vary across GSM1k and GSM8k test questions.The primary results correspond to n=5.
- Results: Figure 16 compares GSM8k and GSM1k performance as the number of standard-format GSM8k few-shot examples changes.The figure focuses on models from the most overfit families.