PaperlayerView exact revision on arXiv

Source-linked AI summary

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models

Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, Mehrdad Farajtabar

arXiv:2410.05229v2cs.LGcs.AI

TL;DR

The paper asks whether high GSM8K performance demonstrates genuine mathematical reasoning and addresses the limitations of fixed, single-metric evaluation. It introduces GSM-Symbolic, a symbolic-template benchmark for controlled variants, and finds substantial variance, complexity sensitivity, and sharp degradation from irrelevant information. These results challenge the reliability of conventional GSM8K metrics and motivate more robust reasoning evaluations.

  • Problem

    It remains unclear whether strong GSM8K results reflect genuine mathematical reasoning, while the benchmark’s fixed questions and single metric limit controlled analysis and raise contamination concerns.

  • Method

    The paper introduces GSM-Symbolic, generates checked symbolic variants of GSM8K questions, and evaluates open and closed models across controlled instances and setups.

  • Results

    Models show substantial performance variability across question instantiations, greater degradation with increased complexity, and performance drops of up to 65% after adding irrelevant information.

  • Takeaways & Limitations

    The findings challenge single-point GSM8K accuracy as a sufficient measure and underscore the need for more reliable evaluations of mathematical reasoning.

  • Takeaways & Limitations

    Adding or removing a clause does not necessarily change the number of required reasoning steps by exactly one, so the difficulty analysis focuses on performance-distribution trends.

Abstract

from arXiv · show

Recent advancements in Large Language Models (LLMs) have sparked interest in their formal reasoning capabilities, particularly in mathematics. The GSM8K benchmark is widely used to assess the mathematical reasoning of models on grade-school-level questions. While the performance of LLMs on GSM8K has significantly improved in recent years, it remains unclear whether their mathematical reasoning capabilities have genuinely advanced, raising questions about the reliability of the reported metrics. To address these concerns, we conduct a large-scale study on several SOTA open and closed models. To overcome the limitations of existing evaluations, we introduce GSM-Symbolic, an improved benchmark created from symbolic templates that allow for the generation of a diverse set of questions. GSM-Symbolic enables more controllable evaluations, providing key insights and more reliable metrics for measuring the reasoning capabilities of models.Our findings reveal that LLMs exhibit noticeable variance when responding to different instantiations of the same question. Specifically, the performance of all models declines when only the numerical values in the question are altered in the GSM-Symbolic benchmark. Furthermore, we investigate the fragility of mathematical reasoning in these models and show that their performance significantly deteriorates as the number of clauses in a question increases. We hypothesize that this decline is because current LLMs cannot perform genuine logical reasoning; they replicate reasoning steps from their training data. Adding a single clause that seems relevant to the question causes significant performance drops (up to 65%) across all state-of-the-art models, even though the clause doesn't contribute to the reasoning chain needed for the final answer. Overall, our work offers a more nuanced understanding of LLMs' capabilities and limitations in mathematical reasoning.

1 INTRODUCTION

The paper questions whether strong GSM8K results reflect genuine mathematical reasoning and introduces controllable evaluations to examine this issue. Its findings show substantial performance variability, sensitivity to irrelevant information, and the need for more reliable evaluation methods.

  • Motivation: GSM8K’s fixed questions and single accuracy metric limit analysis of reasoning robustness, difficulty effects, and possible data contamination.The benchmark does not support controllable experiments across varied question conditions.
  • Contributions: GSM-Symbolic generates diverse GSM8K variants from symbolic templates, enabling more nuanced evaluation across multiple setups.The study evaluates 25 state-of-the-art open and closed models.
  • Findings: All models lose performance on GSM-Symbolic, while results vary across instantiations of the same question.The authors describe this variance as challenging the reliability of single-point GSM8K results.
  • Findings: Performance degradation and variance increase as the number of clauses in a question increases.This pattern indicates difficulty handling greater question complexity.
  • Findings: Adding seemingly relevant but irrelevant information causes performance drops of up to 65% across state-of-the-art models.The authors report that few-shot examples do not overcome the GSM-NoOp challenge.

2 BACKGROUND: REASONING & LANGUAGE MODELS

The background frames logical reasoning as producing novel goals through logical steps and reviews evidence that LLM reasoning is fragile, token-sensitive, and plausibly pattern-based. GSM-Symbolic extends this literature by evaluating distributions across controlled question variants and difficulty levels.

  • Reasoning and Language Models: Logical reasoning is defined as using logical steps to achieve a novel goal rather than memorizing or mimicking previously encountered solutions.Novelty is used to distinguish genuine reasoning from reproduction of familiar responses.
  • Reasoning and Language Models: Transformers fail to generalize on some non-regular tasks, with structured memory identified as necessary for handling complex tasks.This provides a computational perspective on limitations in complex reasoning.
  • Reasoning and Language Models: Prior work suggests that LLM reasoning is probabilistic pattern-matching rather than formal reasoning, despite limited ability to manipulate abstract symbols.The reviewed literature describes reasoning as fragile and non-formal.
  • Reasoning and Language Models: Small input-token changes can substantially alter model outputs, indicating strong token sensitivity and fragile reasoning behavior.This sensitivity is linked in the background literature to token bias.
  • GSM-Symbolic’s Context: GSM-Symbolic builds on these concerns by generating diverse instances and difficulty levels instead of relying on one performance metric.The framework is intended to provide a more comprehensive picture of reasoning capabilities.

3 GSM-SYMBOLIC

GSM-Symbolic converts GSM8K examples into checked symbolic templates that generate controlled question variants. The evaluation uses thousands of generated examples and standardized prompting to study model behavior across conditions.

  • Dataset Construction: GSM8K contains over 8000 grade-school math questions and answers, mostly requiring the four basic arithmetic operations.Its popularity as a single test set creates risks of contamination and sensitivity to minor question changes.
  • Template Generation: GSM-Symbolic represents GSM8K examples as parsable templates with variables, domains, and conditions preserving valid questions and answers.Common conditions include divisibility constraints ensuring whole-number answers.
  • Template Generation: Automated checks verify template values, conditions, and answers, followed by manual review of 10 random samples per template.Questions missed by model-based validation are reviewed again manually.
  • Template Generation: Numerical ranges are chosen to resemble GSM8Ktest ranges so evaluation emphasizes logical reasoning rather than arithmetic difficulty.Appendix analysis reports that expanded ranges remain within models’ arithmetic-accuracy boundaries.
  • Experimental Setup: Experiments cover more than 20 open models from 2B to 27B parameters alongside several state-of-the-art closed models.Selected results appear in the main experiments, with full results reported in the appendix.
  • Experimental Setup: The study uses 100 templates and 50 samples per template, producing 5000 examples for each benchmark.The default setup uses 8-shot Chain-of-Thought prompting with greedy decoding.

4 EXPERIMENTS & RESULTS

Across GSM-Symbolic evaluations, models show unstable mathematical performance: accuracy varies across equivalent instantiations, declines with altered numbers and added clauses, and can collapse when irrelevant information is introduced.

  • Reliability of GSM8K Results: All models exhibit non-negligible accuracy variance across 50 GSM-Symbolic datasets generated from the same templates.The underlying reasoning steps remain unchanged across instances, which differ only in names and values.
  • Reliability of GSM8K Results: 21 out of 25 models place original GSM8K performance more than one standard deviation from the center of the GSM-Symbolic distribution, often on its right side.The authors suggest data contamination as one possible explanation for this optimistic bias.
  • Fragility to Input Changes: Changing names produces less variance than changing numbers, while changing numbers or both names and numbers shifts performance lower and increases variance.Original GSM8K accuracy is closer to the changed-names distribution than to the changed-numbers distributions.
  • Difficulty and Number of Clauses: As clauses increase from GSM-M1 through GSM-P2, accuracy decreases, variance increases, and the rate of accuracy decline also accelerates.The authors relate the increasing variance to greater difficulty in searching and pattern-matching.
  • GSM-NoOp: Adding seemingly relevant but irrelevant clauses causes substantial GSM-NoOp failures because models often convert those statements into operations.Even eight same-question GSM-Symbolic shots do not recover the performance drop; Llama-3-8B remains unchanged and Phi-3 declines slightly.

5 CONCLUSION

The conclusion reports that GSM-Symbolic exposes substantial variability and fragility in LLM mathematical reasoning, while GSM-NoOp reveals sensitivity to irrelevant information. It argues that these findings challenge single-point GSM8K evaluation and motivate more reliable evaluation and further research.

  • Performance varies substantially across different instantiations of the same question, challenging the reliability of single-point GSM8K accuracy metrics.
  • LLM performance is more sensitive to numerical changes than to changes in proper names, and deteriorates as question complexity increases.
  • Adding seemingly relevant but inconsequential information causes substantial performance drops across models, even with multiple examples of the same question or similar irrelevant information.
  • The authors characterize LLM mathematical reasoning as fragile and potentially closer to sophisticated pattern matching than true logical reasoning.
  • The appendix includes full benchmark results, additional distributional and difficulty analyses, and discussions of o1-mini and o1-preview.

A.1 DETAILED EXPERIMENTAL SETUP

The evaluations use an 8-shot chain-of-thought prompt with a fixed problem-solving preamble and specified response-generation limits. Greedy decoding is used except for o1-mini and o1-preview, whose APIs do not expose decoding control.

  • All reported evaluations use 8-shot chain-of-thought prompting.
  • The prompt begins with an instruction to solve each mathematical question step by step as an expert problem solver.
  • The evaluation prompt format is documented as the format used for experiments.
  • Except in the final Sec. 4.4 experiment, evaluations use the original eight GSM8K shots and stop when context, end-token, or next-question conditions are reached.
  • Greedy decoding is used throughout except for o1-mini and o1-preview, whose APIs do not allow decoding control and whose repeated responses can change.

A.2 FULL RESULTS

The appendix reports full 8-shot results for open and closed models on GSM8K and GSM-Symbolic variants. GSM8K accuracy is presented for the full test set and a 100-question template subset, whose performance is reported as similar.

  • Full results cover Gemma, Gemma2, Phi, Mistral, Llama3, GPT-4o, and the o1 series on GSM8K and GSM-Symbolic variants.
  • GSM8K results include accuracy on the full 1,319-example test set and on a randomly selected 100-question subset used to generate GSM-Symbolic templates.
  • The two GSM8K evaluation sets show very similar performance levels, with no significant differences observed.
  • Table 1 contains the full 8-shot results for all models across GSM8K and different GSM-Symbolic variants.
  • Additional results on performance variation on GSM-Symbolic are presented in Figure 10.

A.3 ADDITIONAL RESULTS ON GSM-SYMBOLIC PERFORMANCE DISTRIBUTIONS

Additional analyses examine performance distributions and their statistical significance on GSM-Symbolic. The authors report high variance and drops for additional models, while noting that the significance conclusion depends on the testing assumptions.

  • Additional models also show high performance variance on GSM-Symbolic, including models associated with the largest drops.
  • For an overwhelming majority of models, the 50 GSM-Symbolic performance results differ statistically significantly from the original 100 GSM8K questions under the one-sample t-test assumptions.
  • A related critique argues that a two-sample z-test would be more suitable because the evaluation compares two datasets.
  • Under the alternative assumptions proposed by Ivanova et al., the results would likewise be statistically non-significant.

A.4 ABLATION: DOES FINE-TUNING ON EASIER TASKS HELP WITH MORE DIFFICULT TASKS?

Fine-tuning on the easier GSM-P1 task slightly improves performance on GSM-P1 but does not transfer to the harder GSM-P2 task. Adding GSM-P1 examples as in-context shots likewise does not improve GSM-P2 performance.

  • In-context learning: Including GSM-P1 shots does not improve GSM-P2 performance compared with using shots solely from GSM8K.This matches the observed lower performance on GSM-P2 than GSM-P1.
  • Fine-tuning: Fine-tuning Phi-3.5 on GSM-P1 slightly improves GSM-P1 performance while decreasing performance on GSM-P2.The fine-tuning set used 50 separate GSM-P1 templates and 10,000 generated examples.
  • Implication: Current results suggest that scaling training data will not improve language-model reasoning capabilities.The authors state that this direction warrants further research.

A.5 RESULTS ON O1-PREVIEW AND O1-MINI

o1-preview and o1-mini generally follow the study’s observed trends, although o1-preview remains robust across difficulty levels. Both models nevertheless show a significant performance drop on GSM-NoOp, including cases where irrelevant information is incorrectly used.

  • Difficulty and variance: o1-mini shows decreasing performance and increasing variance as difficulty increases, while o1-preview maintains robust performance across difficulty levels.For o1-preview, the distributions remain close to one another across difficulty levels.
  • GSM-NoOp: Both o1-preview and o1-mini experience a significant performance drop on GSM-NoOp.The models therefore share this limitation despite o1-preview’s strong performance across difficulty levels.
  • Statistical caveat: The statistical interpretation of evaluation results depends on assumptions that may not hold, including independence and model accuracy conditions.The text notes that t-tests also assume normality, although the reported distributions passed a normality test.
  • Failure cases: o1-preview naively applies the 10% inflation despite the prices being specified as current prices.The example indicates that the inflation information is irrelevant to the calculation.
  • Comparison with open models: o1-preview and o1-mini achieve stronger results than current open models but retain similar limitations.The authors associate the stronger results potentially with improved training data and post-training procedures.
  • Failure cases: On another GSM-NoOp example, o1-preview and o1-mini subtract an irrelevant donation amount from the price difference.The donation amount does not affect the requested comparison.

A.6 ABLATION: THE IMPACT OF ARITHMETIC ACCURACY

The authors examine whether arithmetic difficulty explains performance drops on GSM-Symbolic variants. They report that the evaluated models handle the relevant arithmetic ranges accurately, while GSM-P1 and GSM-P2 still show substantial declines.

  • Range selection: The template numerical ranges were chosen close to GSM8K values because logical reasoning was considered more important than arithmetic ability.The ranges were slightly increased to generate enough valid instances.
  • Final-answer distributions: GSM-Symbolic contains more 3-digit final answers and fewer 1-digit or 2-digit answers than GSM8K, but most final answers remain below 5 digits.Its overall numerical range does not increase significantly.
  • Performance comparison: GSM-P1 and GSM-P2 have digit distributions similar to GSM-Symbolic, yet their performance drops significantly.The reported arithmetic range is therefore within the models’ demonstrated capabilities.
  • Intermediate calculations: Intermediate answers involve digit-length distributions similar to those of final answers, and realistic response-extracted arithmetic also shows very high accuracy.The intermediate-operation analysis defines an intermediate answer as the result of each operation during a solution.
  • Arithmetic accuracy: Gemma2-9B and Phi3-Medium achieve nearly perfect addition and subtraction accuracy up to 4 digits, with very high multiplication and division accuracy.These results come from zero-shot arithmetic evaluations across answer digit lengths.
  • Interpretation: The authors judge arithmetic difficulty unlikely to explain GSM-P2’s significant performance drop.GSM-P2 is mostly within the fewer-than-5-digit range handled accurately by models such as Gemma2-9B.

B DISCUSSION ON STATISTICAL ANALYSIS

The authors defend a conditional-distribution analysis of renamed GSM8K questions while acknowledging that its interpretation depends on debatable assumptions. They argue that model responses do not exhibit the behavior expected under their proposed true-reasoner view.

  • Reasoning model: The proposed interpretation assumes that an ideal reasoner’s performance on a single question follows a degenerate distribution rather than a Bernoulli distribution.The authors argue that this reflects consistent question-level competence rather than aggregate score variability.
  • Conditional analysis: The analysis defines P(o = true) for correctly answering an original GSM8K question and P(x = true) for correctly answering its renamed variant.The renamed questions preserve the variables while changing names, reducing the influence of arithmetic mistakes.
  • Conditional analysis: The authors focus on P(x = true | o = true) and P(x = true | o = false) rather than treating renamed-question accuracy as an independent Bernoulli trial.They expect a true reasoner’s conditional behavior to follow a specific Bernoulli pattern.
  • Observed behavior: Figure 18 shows that Gemma2-9B and other models do not exhibit the expected conditional behavior.The figure reports deviations from both the uniform assumption and the expected p = 1 behavior conditioned on original-question correctness.
  • Limitation: The authors acknowledge that their assumptions were not articulated clearly enough and leave readers to judge them against Ivanova et al.’s alternatives.They explicitly present the validity of the competing statistical foundations as unresolved.
  • Assumption: The degenerate-distribution argument assumes away non-reasoning factors such as arithmetic mistakes and overlooking information.The authors acknowledge that humans can exhibit non-zero variance, although they expect it to be very small on simple grade-school arithmetic.
Loading 2410.05229v2…