Source-linked AI summary
MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
Kaixuan Huang, Jiacheng Guo, Zihao Li, Xiang Ji, Jiawei Ge, Wenzhe Li, Yingqing Guo, Tianle Cai, Hui Yuan, Runzhe Wang, Yue Wu, Ming Yin, Shange Tang, Yangsibo Huang, Chi Jin, Xinyun Chen, Chiyuan Zhang, Mengdi Wang
TL;DR
Existing perturbation benchmarks largely preserve solution reasoning, leaving unclear whether strong mathematical performance reflects understanding or memorization. This paper constructs MATH-P-Simple and MATH-P-Hard from level-5 MATH problems to test generalization when problem formulations fundamentally change. All evaluated models show performance degradation on MATH-P-Hard, and the authors identify memorization in blindly applying techniques, with original-problem demonstrations potentially worsening the issue.
Problem
Prior perturbation benchmarks preserve underlying reasoning patterns, leaving hard perturbations that make original solution steps inapplicable unexplored.
Method
The authors create MATH-P-Simple and MATH-P-Hard by applying simple and hard perturbations to 279 level-5 MATH problems.
Results
All evaluated models show performance degradation on MATH-P-Hard, and errors include blindly applying techniques without judging whether modified settings remain suitable.
Takeaways & Limitations
Hard perturbations expose a memorization pattern in which models reuse learned problem-solving techniques without assessing their applicability to modified contexts.
Takeaways & Limitations
The study prohibits tool usage, including code interpreters, because brute-force programs can trivially solve many problems.
Abstract
from arXiv · showhide
Large language models have demonstrated impressive performance on challenging mathematical reasoning tasks, which has triggered the discussion of whether the performance is achieved by true reasoning capability or memorization. To investigate this question, prior work has constructed mathematical benchmarks when questions undergo simple perturbations -- modifications that still preserve the underlying reasoning patterns of the solutions. However, no work has explored hard perturbations, which fundamentally change the nature of the problem so that the original solution steps do not apply. To bridge the gap, we construct MATH-P-Simple and MATH-P-Hard via simple perturbation and hard perturbation, respectively. Each consists of 279 perturbed math problems derived from level-5 (hardest) problems in the MATH dataset (Hendrycksmath et. al., 2021). We observe significant performance drops on MATH-P-Hard across various models, including o1-mini (-16.49%) and gemini-2.0-flash-thinking (-12.9%). We also raise concerns about a novel form of memorization where models blindly apply learned problem-solving skills without assessing their applicability to modified contexts. This issue is amplified when using original problems for in-context learning. We call for research efforts to address this challenge, which is critical for developing more robust and reliable reasoning models.
1. Introduction
Existing math benchmarks test robustness to simple perturbations that preserve reasoning patterns, leaving open whether models understand when those patterns cease to apply. MATH-Perturb addresses this gap with hard perturbations that fundamentally change problem formulations and tests whether models assess applicability rather than blindly reuse learned solutions.
- High benchmark performance may reflect data contamination, pattern recognition, or memorized solution steps rather than understanding underlying mathematical rationale.
- Prior perturbation studies mainly preserve the fundamental reasoning patterns, so they do not directly test generalization across substantial reasoning shifts.
- Hard perturbations keep lexical changes similar to simple perturbations while making original solution paths inapplicable to the modified problems.
- The benchmark evaluates whether models can judge when changed formulations require different reasoning instead of applying learned skills indiscriminately.
- Robustness to hard perturbations may become a bottleneck for real-world LLM use as average-case capabilities improve.
- MATH-P-Simple and MATH-P-Hard each contain 279 perturbed problems derived from the hardest level-5 problems in MATH.
2. Dataset Curation
The authors curate paired simple and hard perturbations of level-5 MATH problems under minimal-edit and changed-answer constraints. Expert annotation, cross-validation, and similarity analyses are used to construct and characterize the benchmark.
- Origin of the Dataset: The benchmark starts from level-5 MATH problems spanning seven mathematical subjects and uses both train- and test-split seed problems.
- Annotation Criterion: MATH-P-Simple uses non-essential modifications that preserve the original solution method, whereas MATH-P-Hard uses fundamental changes requiring deeper mathematical understanding.
- Additional Requirements: Annotators make minimal textual modifications while ensuring both perturbed answers differ from the original answer.
- Quality Control: Twelve mathematically trained PhD annotators create the problems, with double-checking, independent cross-validation, and manual verification of disagreements.
- Benchmark Overview and Statistics: 279 modification pairs remain after quality checks, including 164 examples from the train split and 115 from the test split.
- Similarity Analysis: Embedding-based similarity analysis uses normalized edit distance and OpenAI’s text-embedding-3-large, while retrieval yields MRRs of 0.995 for simple and 0.986 for hard perturbations.
- Common Strategies for Perturbations: Hard perturbations increase mathematical complexity, enlarge key values, relax constraints, or remove essential properties such as symmetry, reducibility, and linearity.
3. Experimental Results
Across zero-shot evaluations, MATH-P-Hard substantially lowers accuracy relative to original problems, while MATH-P-Simple usually causes smaller declines. Failure analysis identifies both ordinary capability errors and memorization-driven reasoning failures, and original-problem demonstrations can help or mislead models on hard perturbations.
- Benchmarking the performance of LLMs: MATH-P-Hard performance is significantly lower than performance on original problems for every evaluated model.The authors interpret this gap as evidence that MATH-P-Hard is more difficult.
- Benchmarking the performance of LLMs: Most models show a slight performance drop on MATH-P-Simple, with the main declines coming from problems originating in the training split.Generalization errors remain even when test examples preserve the training problems’ reasoning patterns.
- Benchmarking the performance of LLMs: Test-split results for MATH-P-Simple are mixed: several models drop by more than 5%, while Phi-3.5-mini-instruct improves.For most models, modified and original test accuracies are close, unlike the larger gaps reported in earlier work.
- Failure Mode Analysis: Common errors on harder problems include numerical and algebraic mistakes, unjustified claims, missed cases, and missing mathematical knowledge, especially in weaker models.The analysis focuses on cases where models fail MATH-P-Hard but solve the original or simple-perturbation problem.
- Failure Mode Analysis: Models also ignore changed assumptions, reuse unsuitable original techniques, or output the original desired outcome; memorization accounts for 40% of o1-mini errors and 25% of Claude-3.5-Sonnet errors.These memorization issues are pervasive, often coupled with other errors, and become more prominent as general capability errors diminish.
- Does In-context Learning Help or Hurt?: Original problem-solution demonstrations boost nearly all models on MATH-P-Simple but leave MATH-P-Hard improvements below 5% for most models.On MATH-P-Hard, demonstrations both correct some errors and introduce misleading errors, with larger effects among large models.
4. Related Work
Prior benchmark perturbations generally preserve the original solution pattern, whereas MATH-P-Hard changes the reasoning pattern and supports studying memorization beyond verbatim recall.
- Perturbations to Existing Mathematical Benchmarks: Existing perturbation benchmarks generate variants that retain the original problems’ solution steps and reasoning patterns.Examples include GSM-IC, GSM-Plus, GSM-Symbolic, Functional MATH, and Putnam-AXIOM.
- Memorization: The paper investigates mathematical-reasoning memorization through counterfactual tests using perturbed problems rather than existing benchmark items.The authors report subtle memorization forms beyond naive verbatim memorization.
- Perturbations to Existing Mathematical Benchmarks: MATH-P-Hard applies hard perturbations under which the original reasoning pattern does not apply, while preserving direct links to the original MATH problems.This makes the benchmark suitable for evaluating harder skills and memorization effects.
- Comparison with MATH2: Unlike MATH2, MATH-P-Hard retains natural original problems as references while requiring harder skills after modification.This supports both harder-math evaluation and memorization analysis.
5. Conclusion
The paper studies LLM math-reasoning generalization under hard perturbations by converting 279 level-5 MATH problems into simple and hard variants, finding performance degradation on MATH-P-Hard.
- 5. Conclusion: The study modifies 279 level-5 MATH problems into MATH-P-Simple and MATH-P-Hard using simple and hard perturbations.MATH-P-Simple serves as a control experiment.
- 5. Conclusion: All evaluated models show performance degradations on MATH-P-Hard.
- 5. Conclusion: Many MATH-P-Hard errors are traced to a new form of memorization.The supplied conclusion passage introduces this finding but does not specify its mechanism fully.
A. Version Information of the Models
The appendix identifies the evaluated model groups and provides version information for those models.
- A. Version Information of the Models: The evaluation includes long-CoT, closed-source, open-sourced general-purpose, and math-specific model groups.
- A. Version Information of the Models: Table 2 contains version information for the models.
B. Benchmark Statistics
The appendix provides benchmark statistics through tables reporting subject-level problem counts and response-category counts and percentages.
- B. Benchmark Statistics: Table 3 reports the number of problems corresponding to different subjects.
- B. Benchmark Statistics: Table 4 reports the number and percentage of model responses in each of four categories.
C. Additional Experimental Results
The analysis categorizes model responses across original, simple-perturbation, and hard-perturbation problems, revealing how correctness varies across these related formulations.
- Response Categories: The three problems yield four response cases after accounting for correctness fluctuations across MATH-P-Simple variations.Cases distinguish whether models solve the original or simple variant and whether they also solve the hard variant.
- Response Categories: Stronger models show more Case I responses and fewer Case II responses.Case I solves at least one of the original and simple problems plus the hard problem; Case II solves none correctly.
- Response Categories: Less than 10% of responses fall into Case III, where models solve the hard variant but not the easier original or simple variants.Manual inspection attributes these cases to misalignment between model capabilities and annotators’ perceived problem difficulty.
- Response Categories: Table 4 reports the percentage of each model’s responses assigned to the defined cases.The table supports comparisons of response patterns across models.
C.3. The Effect of In-Context Learning
This section examines in-context learning with each problem’s original unmodified problem-and-solution pair, including its separate helpful and misleading effects on MATH-P-Hard.
- In-Context Learning Setup: The experiment measures performance with the corresponding original problem and solution supplied as the in-context learning example.Table 6 reports comparisons with and without this original example.
- Effects on MATH-P-Hard: For MATH-P-Hard, the analysis separates in-context learning into an ICL effect that reduces error rates and a misleading effect that increases them.The decomposition is reported in Table 7 and visualized for representative models in Figure 8.
C.4. Ablation Study: In-Context Learning with the Original Example v.s. In-Context Learning with a Random Example
The ablation compares using the corresponding original example against using a random same-category example to assess how example choice affects one-shot in-context learning.
- Example Selection: One-shot in-context learning with the corresponding original problem and solution consistently outperforms learning with a random same-category example except in one case.Table 8 presents the performance comparison between the two example-selection strategies.
- Example Selection: The comparison evaluates original-example and random-example in-context learning under the same category-matched query setting.The random example is selected from the same category as the query problem.
C.5. Inference-time Scaling Behaviors
This section studies inference-time scaling by generating multiple solutions and evaluating both pass@k and self-consistency as the number of sampled solutions changes.
- Pass@k: For each problem, the study independently generates N solutions and computes pass@k for each 1 ≤ k ≤ N.In the formula, c denotes the number of correct answers among the runs.
- Self-Consistency: Self-consistency uses majority voting over k randomly sampled responses and reports the mean and standard deviation across five random draws.The evaluation covers o1-mini, Llama-3.1-8B-Instruct, and Qwen2.5-Math-7B-Instruct.
- Experimental Configuration: The experiments use N = 64 for Llama-3.1-8B-Instruct and Qwen2.5-Math-7B-Instruct, but N = 8 for o1-mini.Results are plotted in Figure 9.