PaperlayerView exact revision on arXiv

Source-linked AI summary

A Careful Examination of Large Language Model Performance on Grade School Arithmetic

Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati, Summer Yue

arXiv:2405.00332v3cs.CLcs.AIcs.LG

TL;DR

Benchmark contamination can make LLM reasoning scores difficult to interpret. The authors construct GSM1k, a human-created benchmark modeled on GSM8k, and find that some models perform worse on it while others show little evidence of overfitting. Their analysis also links GSM8k generation likelihood with the performance gap, suggesting partial memorization may contribute.

  • Problem

    Because benchmark examples may resemble training data, high scores on public reasoning benchmarks may not reflect reasoning ability alone.

    Source: §1 · p. 1

  • Method

    The authors create a 1,250-problem, human-authored GSM1k benchmark designed to mirror GSM8k and evaluate leading open- and closed-source models on it.

    Source: §1 · p. 2

  • Results

    Models show performance drops of up to 13% on GSM1k, while frontier models show little evidence of overfitting; GSM8k generation likelihood is positively related to the performance gap.

    Source: §6 · p. 11

  • Takeaways & Limitations

    Even heavily overfit models can still solve novel grade-school problems, so overfitting does not mean they are incapable of reasoning.

    Source: §5.3 · p. 8

    Reported GSM8k scores can be below model-specific benchmark results because the evaluation uses a standardized prompt rather than each model’s performance-maximizing prompt.

    Source: §4 · p. 6

Abstract

from arXiv · show

Large language models (LLMs) have achieved impressive success on many benchmarks for mathematical reasoning. However, there is growing concern that some of this performance actually reflects dataset contamination, where data closely resembling benchmark questions leaks into the training data, instead of true reasoning ability. To investigate this claim rigorously, we commission Grade School Math 1000 (GSM1k). GSM1k is designed to mirror the style and complexity of the established GSM8k benchmark, the gold standard for measuring elementary mathematical reasoning. We ensure that the two benchmarks are comparable across important metrics such as human solve rates, number of steps in solution, answer magnitude, and more. When evaluating leading open- and closed-source LLMs on GSM1k, we observe accuracy drops of up to 13%, with several families of models (e.g., Phi and Mistral) showing evidence of systematic overfitting across almost all model sizes. At the same time, many models, especially those on the frontier, (e.g., Gemini/GPT/Claude) show minimal signs of overfitting. Further analysis suggests a positive relationship (Spearman's r^2=0.32) between a model's probability of generating an example from GSM8k and its performance gap between GSM8k and GSM1k, suggesting that many models may have partially memorized GSM8k.

1 Introduction

The paper introduces GSM1k, a human-created benchmark designed to mirror GSM8k and probe whether benchmark performance reflects reasoning or contamination. Models show uneven drops on GSM1k, with evidence of overfitting in some families but little sign in frontier models.

1 Introduction

  • GSM1k is a 1,250-problem, human-created benchmark designed to mirror GSM8k and support an apples-to-apples comparison.Its creators used no LLM or other synthetic data source and took steps to match GSM8k’s difficulty distribution.Source: §1 · p. 2
  • Up to 13% worse performance on GSM1k than GSM8k was observed for the worst-performing model in the evaluation.The paper reports that Mistral and Phi show consistent evidence of overfitting across nearly all versions and sizes, while frontier models and all Llama2 sizes show minimal signs.Source: §1 · pp. 2–3
  • A model’s probability of generating GSM8k examples is positively related to its GSM8k–GSM1k performance gap (Spearman’s r2 = 0.32).The authors say this suggests partial memorization of GSM8k examples may contribute to overfitting.Source: §1 · p. 3
  • GSM1k is being withheld for now to reduce the risk of future data contamination, with release tied to a performance threshold or a date.The authors also plan recurring evaluations and release of the evaluation code.Source: §1 · p. 3

2 Related Work

The paper situates GSM1k among benchmark-based evaluations of mathematical reasoning and approaches to detecting or reducing data contamination. It contrasts its new-problem benchmark with methods that perturb existing questions or generate variants of them.

2 Related Work

  • GSM1k follows the ImageNet overfitting comparison strategy of Recht et al. [2019], using a newly created set to compare performance with GSM8k.GSM8k is described as a benchmark of 8.5k grade-school math problems, with top models reporting over 95% accuracy as of April 2024.Source: §2 · p. 3

2.1 Data Contamination

  • Common contamination-mitigation practices include removing benchmark-overlapping n-grams [Brown et al. [2020]] and highly similar embeddings [Shi et al. [2024]].The paper notes that data contamination is a recognized concern in the field [Balloccu et al. [2024], Magar and Schwartz [2022], Sainz et al. [2023], Jacovi et al. [2023], Xu et al. [2024]].Source: §2.1 · p. 3
  • Alternative detection methods perturb benchmark wording [Xu et al. [2024]] or generate fresh numerical variants [Srivastava et al. [2024]], while GSM1k uses new problems.The cited functional evaluations generate only a small portion of the full problem space, whereas their MATH results suggest substantial contamination, including for frontier models.Source: §2.1 · p. 3

3 GSM1k

GSM1k is a human-authored set of grade-school arithmetic problems built and checked to approximate GSM8k’s difficulty and answer-magnitude distributions. Human and pre-GSM8k model checks support its comparability, while the authors caution that the datasets are not identically distributed.

3 GSM1k

  • GSM1k contains 1,250 human-authored problems requiring elementary math, created from GSM8k examples and restricted to basic arithmetic with positive-integer solutions.No language models were used in dataset construction.Source: §3 · pp. 3–4
  • The authors describe GSM1k and GSM8k as highly similar but not identically distributed, making GSM1k an approximation of an ideal never-public GSM8k test set.The authors note that they undertook extensive efforts to maximize similarity.Source: §3 · pp. 4–5

3.1 Quality Checks

  • Three review layers checked problem correctness, grade-school appropriateness, formatting, independent solutions, and production quality.Of 2,108 initial problems, 1,419 passed the second-solve stage and 1,375 passed the general quality audit.Source: §3.1 · p. 5

3.2 Matching the Difficulty Distribution of GSM8k

  • Difficulty was approximated by solution-operation counts, and the final 1,250 problems were selected to match GSM8k’s answer-magnitude distribution as closely as possible.The operation-count measure is rough, and the number of steps does not necessarily correlate directly with true difficulty; matching answer magnitudes addresses the possibility that larger numbers affect solve rates.Source: Figure 3 · p. 4, §3.2 · p. 5, Figure 4 · p. 5

3.2.1 Human Differentiation Rates

  • 21.83% of human attempts identified the GSM1k item among five questions, near the 20% chance rate.The audit used 19 annotators not involved in problem creation and covered 1,205 attempts.Source: §3.2.1 · p. 6

3.2.2 Human Solve Rates

  • Annotators solved an average of 4.07 ± 0.93 GSM8k problems and 4.36 ± 1.11 GSM1k problems in 15 minutes.The authors conclude that GSM1k is comparable in difficulty and perhaps slightly easier, so difficulty differences are unlikely to explain substantial model-accuracy decreases.Source: §3.2.2 · p. 6

3.2.3 LLM Solve Rates

  • GPT-NeoX-20B and GPT-2, released before GSM8k, showed minimal differences in solve rates between GSM8k and GSM1k.The authors use these models as a sanity check because they are known not to be contaminated by GSM8k.Source: §3.2.3 · p. 6

4 Results

The results compare models under shared evaluation settings and report substantially worse GSM1k performance for some less-known leaderboard models. Figure 5 contrasts models’ GSM1k and GSM8k accuracy against the no-overfit line, with different patterns across model families.

4 Results

  • The standardized evaluation may yield GSM8k scores below model-reported benchmarks because it does not use each model’s individually optimized prompt.The authors focus on the GSM1k–GSM8k performance difference under a fixed evaluation strategy and describe this as a fair comparison.Source: §4 · p. 6
  • Many lesser-known models near the top of the OpenLLMLeaderboard perform substantially worse on GSM1k, suggesting they may be gaming GSM8k rather than improving reasoning.The authors select evaluated models based on popularity and report the full results in Appendix D.Source: §4 · p. 6
  • Figure 5 places many Mistral and Phi models below the no-overfit line, while some Claude models lie above it.The caption says the Claude pattern is consistent with GSM1k being slightly easier than GSM8k; the plot is zoomed to 70–100% GSM8k accuracy.Source: Figure 5 · p. 7, §4 · p. 7
  • Models are partitioned by GSM8k performance and compared with models at similar performance levels in Figures 5, 6, and 7.The evaluation uses a shared prompt with five randomly drawn GSM8k training examples; open-source models are evaluated at temperature 0.Source: §4 · pp. 6–7

5 Analysis

The analysis finds systematic GSM8k-over-GSM1k performance in several model families, while frontier models often show minimal overfitting and overfit models can still solve novel problems. A positive association between GSM8k generation likelihood and the performance gap supports partial memorization as one contributor, but outliers indicate contamination is not the full explanation.

5.1 Lesson 1: Some Model Families are Systematically Overfit

  • Phi and Mistral show systematic stronger performance on GSM8k than GSM1k across nearly every release and scale, with a weaker pattern in Yi, Xwin, Gemma, and CodeLlama.Examining model families rather than individual releases makes the overfitting pattern more conclusive, according to the authors.Source: §5.1 · p. 7

5.2 Lesson 2: Other Models, Especially Frontier Models, Show No Signs of Overfitting

  • Many models, including frontier and close-to-frontier models, show minimal overfitting, and no models in Figure 6’s 40–70% GSM8k-accuracy range lie on the no-overfit line.The authors propose stronger reasoning and more careful contamination prevention by frontier builders as possible explanations; they favor the reasoning hypothesis but note training sets are unavailable for certainty.Source: §5.2 · pp. 7–8, Figure 6 · p. 8

5.3 Lesson 3: Overfit Models Are Still Capable of Reasoning

  • Overfitting does not mean models cannot reason: Phi-3 solves over 68% of GSM1k despite an almost 10% accuracy drop from GSM8k, similar to dbrx-instruct with almost 35x as many parameters.The authors also report that Mistral models remain among the strongest open-source models even accounting for overfitting.Source: §5.3 · pp. 8–9

5.4 Lesson 4: Data Contamination Is Likely Not The Full Explanation for Overfitting

  • Spearman’s rank correlation of 0.32 links per-character GSM8k log-likelihood with the GSM8k–GSM1k performance gap, while outliers suggest contamination is not the full explanation.Each percentage-point performance difference is associated with a 7.9 × 10−3 increase in per-character log-likelihood; the authors also report Pearson r2 = 0.15 and Kendall’s τ = 0.28.Source: §5.4 · pp. 9–10

6 Discussion

The discussion presents GSM1k as a novel benchmark for measuring overfitting on GSM8k. Evaluations show substantial overfitting in some models, alongside minimal evidence in frontier models and signs of generalizable mathematical reasoning even among heavily overfit families.

6 Discussion

  • GSM1k is a novel dataset designed to measure LLM overfitting on GSM8k.The study benchmarks leading open- and closed-source models on the dataset.Source: §6 · p. 10
  • Models show performance drops of up to 13% accuracy, with Mistral and Phi consistently overfitting across almost all sizes and versions.The authors also report minimal overfitting among frontier models.Source: §6 · p. 11
  • A positive relationship between GSM8k example likelihood and the GSM8k–GSM1k performance gap suggests partial memorization, while many heavily overfit models still show strong generalizable mathematical reasoning.Source: §6 · p. 11

A Annotator Instructions

Annotators were asked to create original grade-school math questions and answers resembling the dataset, using examples as guidance but not chatbots or paraphrases.

A Annotator Instructions

  • Annotators had to create original questions and answers similar to the dataset, without chatbot assistance or paraphrasing the examples.The instructions say chatbot use could result in removal from the project.Source: A Annotator Instructions · p. 19
  • The workflow begins by reviewing example question-answer pairs, then creating problems according to each task’s guidance.Source: A Annotator Instructions · p. 19
  • Problems should not reuse premises, and answers must be positive integers rather than zero or negative values.Source: A Annotator Instructions · p. 19
  • Calculations should be solvable with pen and paper by an eighth grader and use only elementary arithmetic operations.Source: A Annotator Instructions · p. 19
  • Final answers must be single integers, with units specified in the question; intermediate steps may use simple decimals.Source: A Annotator Instructions · p. 19
  • Annotators are instructed to check their work, with a quality-control process also planned.Source: A Annotator Instructions · p. 19
  • Figure 9 shows annotator-facing material before three example prompts drawn from GSM8k.Source: Figure 9 · p. 20

B N-shot Prompt (examples selected randomly from GSM8k train)

C Results with An Alternative Prompt

An alternative prompt reduces the GSM8k–GSM1k gap by about 1% on average, though the effect is much larger for some models. The authors note that the prompt differs from the standard one in its examples and answer format.

D Results Table

Table D presents full results under standard and alternative prompts, focusing on the GSM8k–GSM1k performance difference while holding the evaluation setting constant.

D Results Table

  • The comparison emphasizes performance differences under a constant evaluation setting; the Z-score and p-value use a two-tailed two-proportion Z-test.Source: D Results Table · p. 23

Standard Prompt

  • Table D sorts models by the GSM8k–GSM1k performance difference and reports scores, Z-scores, and p-values for the standard prompt.The authors caution that harness format mismatches count as incorrect even when answers are otherwise correct.Source: Standard Prompt · p. 23, D Results Table · p. 23

Alternative Prompt

E 50 Examples from GSM1k

The examples are grade-school word problems built around everyday contexts and quantitative relationships, including costs, rates, time, ratios, and counts.

E 50 Examples from GSM1k

F Additional Plots From Log-Likelihood Experiments

The log-likelihood plots compare GSM1k with GSM8k and show almost no correlation between models’ probability of generating GSM1k and overfitting, which the caption attributes to GSM1k being newly created and absent from the internet.

F Additional Plots From Log-Likelihood Experiments

  • Figure 10 reports almost no correlation between models’ probability of generating GSM1k and their level of overfitting.The caption attributes this to GSM1k being newly created and not on the internet.Source: Figure 10 · p. 33
  • GSM1k’s new creation and absence from the internet are given as the reason its generation probability shows almost no correlation with overfitting.Source: Figure 10 · p. 33
  • Figure 11 plots the difference between models’ log-likelihood on GSM1k and GSM8k.Source: Figure 11 · p. 33

G Bar Chart of Performance Gaps Between GSM8k and GSM1k Across All Model Accuracies

The plots divide models by GSM8k accuracy: among models above 70%, some show overfitting while others show little to none; in the 40–70% range, all fall below the line, with Llama-2-70b doing better than some others.

G Bar Chart of Performance Gaps Between GSM8k and GSM1k Across All Model Accuracies

  • Among models above 70% GSM8k accuracy, Mistral and Phi are overfit, while other models show little to no evidence of overfitting.Source: Figure 12 · p. 34
  • For models with 40–70% GSM8k accuracy, all appear below the line, though Llama-2-70b does better than some others.Source: Figure 13 · p. 34
  • Figure 14 covers models with less than 40% GSM8k accuracy.Source: Figure 14 · p. 35
Loading 2405.00332v3…