PaperlayerView exact revision on arXiv

Source-linked AI summary

Climbing the Ladder of Reasoning: What LLMs Can-and Still Can't-Solve after SFT?

Yiyou Sun, Georgia Zhou, Haoyue Bai, Hao Wang, Dacheng Li, Nouha Dziri, Dawn Song

arXiv:2504.11741v2cs.AIcs.CLcs.LG

TL;DR

Small-scale SFT improves mathematical reasoning, but which capabilities it enhances and which limitations remain are not well understood. The paper analyzes AIME24 by categorizing questions into four difficulty tiers and examining the requirements for advancing between them. It finds that R1-style reasoning and limited SFT support progress to Medium, while Hard performance plateaus and Exh remains challenging.

  • Problem

    Although small-scale SFT can improve mathematical reasoning, the specific capabilities it enhances and the limitations that persist remain incompletely understood.

    Source: §1 · p. 2

  • Method

    The paper analyzes AIME24, categorizes questions into four difficulty tiers, and examines requirements for advancing between tiers.

    Source: §1 · p. 2

  • Results

    AIME24 results form four tiers: 500–1K R1-style instances support Medium proficiency, Hard accuracy plateaus around 65%, and Exh remains challenging; careful curation adds little, while scale matters.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Dataset scale remains important, but simply expanding SFT data does not effectively address Exh-level questions.

    Source: §3 · p. 9

    SFT-trained models tend to adopt similar solution strategies, leaving open whether SFT alone can develop higher-level reasoning.

    Source: §1 · p. 3

Abstract

from arXiv · show

Recent supervised fine-tuning (SFT) approaches have significantly improved language models' performance on mathematical reasoning tasks, even when models are trained at a small scale. However, the specific capabilities enhanced through such fine-tuning remain poorly understood. In this paper, we conduct a detailed analysis of model performance on the AIME24 dataset to understand how reasoning capabilities evolve. We discover a ladder-like structure in problem difficulty, categorize questions into four tiers (Easy, Medium, Hard, and Extremely Hard (Exh)), and identify the specific requirements for advancing between tiers. We find that progression from Easy to Medium tier requires adopting an R1 reasoning style with minimal SFT (500-1K instances), while Hard-level questions suffer from frequent model's errors at each step of the reasoning chain, with accuracy plateauing at around 65% despite logarithmic scaling. Exh-level questions present a fundamentally different challenge; they require unconventional problem-solving skills that current models uniformly struggle with. Additional findings reveal that carefully curated small-scale datasets offer limited advantage-scaling dataset size proves far more effective. Our analysis provides a clearer roadmap for advancing language model capabilities in mathematical reasoning.

1 Introduction

The paper examines which mathematical reasoning capabilities small-scale SFT improves and what limitations remain. It analyzes AIME24 as a four-tier reasoning ladder and identifies conditions for progress between tiers.

1 Introduction

  • The study addresses the incomplete understanding of which capabilities small-scale SFT enhances and whether its gains generalize beyond training trajectories.It also asks which questions remain unsolved despite SFT and whether improvements reflect transfer of problem-solving strategies.Source: §1 · p. 2
  • AIME24 questions form four tiers—Easy, Medium, Hard, and Exh—and higher-tier solvers generally succeed on lower-tier questions.The analysis identifies requirements for advancing from one tier to the next.Source: §1 · p. 2
  • R1-style reasoning with 500–1K SFT instances supports progression to Medium proficiency, while Hard accuracy plateaus around 65% and current models struggle uniformly at Exh.Hard-level gains follow logarithmic scaling; Exh-level problems call for unconventional strategies such as out-of-the-box insights or strong geometric intuition.Source: §1 · p. 2
  • Careful SFT-dataset curation yields marginal gains, while scaling dataset size remains important despite diminishing benefits on Hard questions.Performance across math categories stays within 55±4%, and similar versus randomly constructed datasets differ by about 1%.Source: §1 · p. 2
  • SFT-trained models tend to adopt similar solution strategies, leaving open whether SFT alone can develop higher-level reasoning capabilities.The authors frame this as a question for future research.Source: §1 · p. 3

2 Climbing the ladder of reasoning

Using AIME24 to trace a ladder of mathematical reasoning difficulty, the analysis finds that small-scale SFT can raise Medium-level performance, while Hard-level accuracy plateaus and Exh-level questions remain unsolved by fine-tuned models. The study identifies training conditions and persistent reasoning barriers that distinguish these tiers.

2.1 Experimental Setup

  • The study uses AIME24’s varied, hierarchically difficult questions to assess reasoning, reporting average pass rate across multiple generations and manually assigning four tiers based on model performance.The benchmark was selected to assess reasoning with mainly high-school mathematics, and the default evaluation uses avg@8.Source: §2.1 · pp. 3–4

2.2 The First Ladder: From Easy-Level To Med-Level Questions

  • About 90% Medium accuracy follows small-scale SFT, Hard-level accuracy plateaus near 65%, and fine-tuned models score 0% on Exh-level questions.Small-scale SFT models initially address Hard questions at about 25%; the Exh result holds across the varying SFT dataset sizes examined.Source: §2.2 · p. 4, §2.4 · p. 5, §2.4.2 · p. 7

2.3 All You need is SFT on 1K random R1-style trajectories in any math categories

  • At least 500 long, randomly selected R1-style trajectories meet the 90% Medium-level passline, regardless of math category.Shorter trajectories, fewer examples, or Gemini-style trajectories lower accuracy.Source: §2.3 · pp. 4–5, Figure 2 · p. 5

2.3.1 SFT leads models to similar problem-solving strategies

  • Across diverse math-training categories, about half of generated trajectories are rated almost identical to DeepSeek-R1 and the rest mostly similar.Similarity ratings use a 0–5 scale on Medium-level problems.Source: §2.3.1 · p. 5, Figure 3 · p. 5

2.4.1 Why models fail: instability from exploration and computation of the task

  • Hard problems compound errors across sequential reasoning steps, while computationally intensive substeps can become the primary obstacle for small-scale SFT models.The analysis models overall success across steps as the product of individual step success rates; a tetrahedron-volume calculation is identified as a key obstacle in one example.Source: §2.4.1 · p. 6

2.4.2 SFT data scaling shows logarithmic trend in Hard-level question accuracy

  • Hard-level accuracy scales logarithmically with SFT data size and plateaus near 65%, while models using reinforcement learning or external tools surpass that ceiling.The comparison includes public SFT models trained on 114K and 1M trajectories.Source: Figure 4 · p. 7, §2.4.2 · p. 7

2.4.3 Carefully curated small-scale SFT dataset does not deviate from the scaling trend

  • A test-aware curated 1K dataset scores 33.6% on Hard questions, 5 points above a random 1K dataset, but increasing size from 1K to 2K yields a larger 7% gain.The authors note that the curated-versus-random comparison is somewhat unfair because the curated data was built with knowledge of test questions.Source: §2.4.3 · p. 7

2.5 The third ladder: from Hard-level to Exh-level questions

  • Exh-level failures include rigid familiar strategies, limited geometric intuition, and insufficient exploration of substeps; the authors note that these limitations extend beyond their examples.For instance, R1 may solve a subcase of a problem but rush to an incorrect conclusion on the full problem, despite long reasoning.Source: §2.5 · p. 8

3 Summary and implications for future study

Small-scale SFT models can reach Deepseek-R1-like question coverage with enough attempts, but their overall accuracy remains substantially lower. The findings also indicate logarithmic gains from more SFT data, marginal differences from curation, and a ceiling on scaling for Exh-level questions.

3 Summary and implications for future study

  • With cov@8×4, geometry-SFT models’ best possible scores all reach 90, but overall accuracy remains more than 20% lower.The authors identify instability in deep exploration and computational limitations as the primary obstacles.Source: §3 · p. 8, Table 2 · p. 9
  • Performance across SFT datasets for math categories stays within 55±4%, and a curated dataset beats a randomly selected diverse dataset by only 1%.The 1% gap comes from the comparison of a dataset selected for similarity to test questions with a random diverse dataset.Source: §3 · pp. 8–9, Table 2 · p. 9
  • Comparisons with literature results require caution because single sampling runs can inflate reported performance; the paper notes S1’s published result is about 20% higher than its own.The caveat concerns discrepancies between Table 2(a) and numbers reported in the literature.Source: §3 · p. 8, Table 2 · p. 9
  • Increasing SFT data yields logarithmic-like performance gains, while scaling alone cannot effectively address Exh-level questions.The paper emphasizes that dataset size remains important but its effect reaches a ceiling on the hardest tier.Source: §3 · p. 9

A Small-scale SFT Results in Literature

The section presents a comparison of existing reasoning models’ math performance and defines a four-tier classification for AIME24 questions based on model accuracy.

A Small-scale SFT Results in Literature

  • Table 3 compares math performance across existing reasoning models, using results reported by Team (2025b) and Muennighoff et al. (2025).The supplied passage identifies the comparison and its cited result sources, but does not provide table values.Source: Table 3 · p. 12

B Difficulty Classification of AIME24 Mathematical Problems

  • AIME24’s 30 questions are divided into Easy, Medium, Hard, and Exh tiers using base-model and small-scale-SFT accuracy thresholds.Easy has 4 questions, Medium 10, Hard 12, and Exh 4; the tiers use above-50%, over-50%, and below-10% criteria as specified for each group.Source: B Difficulty Classification of AIME24… · p. 12
  • The multi-level classification enables more fine-grained analysis of model performance, reasoning capabilities, and limitations.Source: B Difficulty Classification of AIME24… · p. 12

C Examples of carefully-selected small-scale SFT dataset

The examples show curated training questions selected for similarity to AIME24 problems, including integer-triple and triangle-geometry problems.

C Examples of carefully-selected small-scale SFT dataset

  • For AIME24 question #17, one curated integer-triple problem has a reported similarity of 0.60.The AIME24 problem asks for nonnegative triples satisfying a sum constraint and a cubic-expression constraint; the curated example uses a sum-of-squares constraint and ordered positive integers.Source: C Examples of carefully-selected… · p. 12
  • A second problem similar to AIME24 question #17 concerns minimizing a cubic expression over positive integer triples and has similarity 0.58.Source: C Examples of carefully-selected… · p. 12
  • For AIME24 question #16 on triangle geometry, two curated triangle problems are reported with similarities of 0.69 and 0.61.The examples involve circumcenter/incircle conditions and triangle constructions.Source: C Examples of carefully-selected… · p. 12

D Extended Analysis on AIME25

On AIME25, the paper finds a ladder-like progression across public model categories and extends its difficulty analysis to Medium, Hard, and Exh questions. It reports near-perfect Medium-level performance with 1K R1-style trajectories, logarithmic scaling on Hard questions, and comparable outcomes for curated and non-curated data.

D Extended Analysis on AIME25

  • AIME25 performance follows a ladder from the base model through small-scale fine-tuning to large-scale training or tool use, and the analysis omits an Easy tier.The base model does not exceed 50% accuracy on all questions except the first, which is included in Medium; Figure 5 sorts questions by increasing overall difficulty.Source: D Extended Analysis on AIME25 · p. 13, Figure 5 · p. 13
  • SFT on 1K R1-style trajectories across diverse categories achieves near-perfect accuracy on Medium-level AIME25 questions.The finding aligns with the authors’ conclusion that Medium-level problems primarily require a longer R1-style chain of thought, regardless of SFT category.Source: D Extended Analysis on AIME25 · p. 13, Figure 6 · p. 14
  • Hard-level AIME25 accuracy scales logarithmically with SFT dataset size across the tested range from 50 to 20K trajectories.The evaluation also included Openthinker2-32B, trained on 1M trajectories, along with QwQ-32B and Deepseek-R1.Source: D Extended Analysis on AIME25 · p. 13, Figure 7 · p. 14
  • On AIME25, broadly sampled non-curated SFT data performs comparably to carefully curated datasets such as S1.1 and LIMO.The authors report dataset size as having a more substantial impact than careful curation in small-scale SFT setups.Source: D Extended Analysis on AIME25 · p. 14, Figure 8 · p. 14

E Is small-scale SFT just overfitting to the train/test data or it’s true generalization?

The authors test whether small-scale SFT generalizes beyond familiar math benchmarks using perturbed questions and more challenging, diverse problems. They also describe a prompt for comparing solution trajectories by their high-level strategies.

E Is small-scale SFT just overfitting to the train/test data or it’s true generalization?

  • The evaluation tests robustness to question-context perturbations on GSM-Plus and performance on more challenging, diverse problems from Humanity’s Last Exam.The models are benchmarked against Qwen2.5-32B-Instruct, with all baselines fine-tuned from it.Source: E Is small-scale SFT just overfitting… · p. 15
  • SFT models’ performance drop on perturbed questions is similar to that of other popular models, while HLE results show no significant degradation and sometimes slight gains over the base model.The HLE gains are less pronounced than those on traditional math benchmarks.Source: E Is small-scale SFT just overfitting… · p. 15, Table 4 · p. 15

F Prompt used in trajectory similarity comparison

  • Trajectory comparisons assess shared high-level strategies, key differences, and similarity on a six-point scale from almost identical to totally different.The prompt asks for differences in main approaches and intermediate steps as well as a strategy-similarity rating.Source: F Prompt used in trajectory similarity… · p. 15

G Example of similar trajectories in solve Med-level problems

The worked box problem illustrates similar solution structure across model trajectories, including the use of constraints and symmetry to maximize the box’s space diagonal. The human solution also uses symmetry, while the paper notes that its approach differs from the model solutions.

G Example of similar trajectories in solve Med-level problems

  • The example asks for the squared radius of the smallest sphere enclosing a box with surface area 54 and volume 23, and reports 721 as the answer.The sphere’s diameter is the box’s space diagonal, so its radius is half that diagonal.Source: G Example of similar trajectories in… · pp. 16, 18
  • The DeepSeek-R1 trajectory reduces the constrained maximization by assuming two sides are equal, then solves a cubic to identify candidate dimensions.It sets a = b and uses the volume and surface-area constraints to determine c and form the cubic.Source: G Example of similar trajectories in… · pp. 17, 19
  • The human solution uses symmetry to set the width and height equal, then maximizes the remaining length under the volume and surface-area constraints.It expresses the length as 23/a^2 and uses the surface-area equation to solve for a.Source: G Example of similar trajectories in… · p. 18
  • The base model’s algebra trajectory tests rational roots, uses synthetic division, and then applies the quadratic formula to solve the cubic.The passage identifies a = 2 as a root before reducing the polynomial.Source: G Example of similar trajectories in… · p. 18

H Detailed Analysis of Exh-level Problems

The octagon-coloring example contrasts the intended casework solution with a model response that chooses a computationally demanding rotation-based approach. The accompanying error analysis attributes the failure to selecting an unsuitable case split and calculation load.

H Detailed Analysis of Exh-level Problems

  • The intended octagon solution counts colorings by the number of blue vertices and reduces the four-blue case using symmetries, yielding 115 favorable colorings out of 256 and m + n = 371.The model instead gives 41/256 and 297 after struggling with overlaps among rotations.Source: H Detailed Analysis of Exh-level… · p. 21
  • The model’s rotation-based case split leads it toward inclusion–exclusion, whose calculation load it recognizes as complex but does not replace with another method.The passage says it does not consider other ways to break down the problem.Source: H Detailed Analysis of Exh-level… · p. 21
  • The analysis attributes the wrong case-splitting choice to difficulty sketching solution paths at the right granularity and to suppressing enumeration even when needed.The model instead explores a more computationally demanding strategy beyond its capacity.Source: H Detailed Analysis of Exh-level… · p. 21

AIME24 Problem #2: Error Analysis: Case Splitting

  • When asked whether it knows alternative case-splitting properties, the model is reported to be aware of splitting by the number of blue vertices.The passage concludes that it knows this option but chooses not to use it.Source: AIME24 Problem #2: Error Analysis:… · p. 22

2024 AIME24 Problem #2: Error Analysis: Case Splitting (continued)

Follow-up prompts probe whether the model can identify a tractable case split and avoid unnecessary inclusion–exclusion. The analysis reports partial recognition but continued difficulty selecting and pursuing the efficient solution path.

2024 AIME24 Problem #2: Error Analysis: Case Splitting (continued)

  • The model identifies the greatest common divisor of the rotation shift and 8 as the simplifying property, but the analysis says this route still requires complex inclusion–exclusion.It misjudges blue-vertex configurations as more complicated, overlooking that cases with more than four blues need not be considered.Source: 2024 AIME24 Problem #2: Error… · p. 23
  • Even after prompting, the model fails to notice that the exactly-four-blue case does not require inclusion–exclusion, though the authors caution that this does not establish the insight is beyond its capabilities.The model had correctly ruled out cases with fewer than four blue vertices.Source: 2024 AIME24 Problem #2: Error… · p. 23
  • In a later response, the model recognizes that inclusion–exclusion is unnecessary for the case under discussion, yet still overlooks key simplifications and selects an overly demanding approach.The authors note that prompting for the right level of specificity remains challenging without concrete examples.Source: 2024 AIME24 Problem #2: Error… · p. 24

2024 AIME24 Problem #2: Error Analysis: Enumeration

The error analyses isolate distinct failures: enumeration breaks inside a longer reasoning chain, while the dodecagon problem exposes difficulty using symmetry to reduce geometric casework. In the latter, explicit prompting does not produce significant use of rotational symmetry.

2024 AIME24 Problem #2: Error Analysis: Enumeration

  • Isolated tests show the model can count four-blue-vertex configurations, but it mis-enumerates them in the full reasoning chain and needs more reasoning time.Reliable enumeration also required precise prompting that explicitly specified x == 4.Source: 2024 AIME24 Problem #2: Error… · p. 25

2024 AIME24 Problem #21 (continued)

  • DeepSeek-R1 revises its chord analysis after recognizing that step-size differences do not determine direction; enumerating perpendicular chord pairs remains complex.The excerpts show the model exploring chord directions and revising its initial approach.Source: 2024 AIME24 Problem #21 (continued) · p. 27

2024 AIME24 Problem #21: Error Analysis: Failure to Utilize Symmetry

  • Even strong hints do not make the model use dodecagon rotational symmetry: it handles axis-aligned rectangle subproblems but not the non-axis-aligned second subproblem.The analysis describes this as a case where spatial reasoning offers a serious advantage.Source: 2024 AIME24 Problem #21: Error… · p. 28

2024 AIME24 Problem #21: Error Analysis: Failure to Utilize Symmetry (continued)

The probes show that DeepSeek-R1 can handle some isolated counting and symmetry tasks, but more complex problems expose failures in applying those skills. In the function-intersection problem, it struggled to connect geometric reasoning with the functions’ domain restrictions and to locate a special case without prompting.

2024 AIME24 Problem #21: Error Analysis: Failure to Utilize Symmetry (continued)

  • Despite understanding 90-degree rotational equivalence, the model omitted valid rectangles after repeated rechecking, leading to the conclusion that it applied symmetry only technically, not effectively.The discussion suggests that limits on thinking time constrain the use of capabilities in more complex settings.Source: 2024 AIME24 Problem #21: Error… · p. 30

2024 AIME24 Problem #3: Error Analysis: Confusion of concept

  • The model traced each graph’s waves correctly in isolation but failed to solve the intersection problem reliably, discarding a correct count without adequate justification across five runs.Its proposed segment-pair count and grid analogy did not establish the intersections, and it missed the special case needed for the final answer.Source: 2024 AIME24 Problem #3: Error… · p. 32

2024 AIME24 Problem #3: Error Analysis: Confusion of concept (continued)

  • The model misapplied geometric reasoning to the two function graphs, misunderstanding their domain restrictions and how their differently oriented segments intersect.The analysis says it relied on geometric intuition rather than solving algebraically, but did not correctly apply the domain constraints.Source: 2024 AIME24 Problem #3: Error… · p. 33

2024 AIME24 Problem #3: Error Analysis: Edge case detection

  • The model could identify the two intersections near (1,1) when directed to examine that location, but failed to find the special case on its own.It also found the coordinate (1,1) and repeatedly rejected it without explanation.Source: 2024 AIME24 Problem #3: Error… · p. 34

I Sub-question decomposition for Hard-level questions

The section presents sub-question decompositions for hard geometry problems, breaking each target into intermediate calculations. The examples cover a triangle-and-circle problem and a tetrahedron problem.

I Sub-question decomposition for Hard-level questions

  • The triangle problem is decomposed into finding A’s coordinates, determining the circle’s center and radius, and finding the intersection of the tangents at B and C before addressing AP.The original question asks for AP after AD intersects the circle again at P.Source: I Sub-question decomposition for… · p. 35
  • For the tetrahedron problem, the listed subquestions ask for the volume and the total surface area from its triangular faces.The source problem specifies equal edge lengths AB and CD and asks for a distance in the form m√n from a point equidistant from all four faces.Source: I Sub-question decomposition for… · p. 35
  • A further tetrahedron subquestion asks for the common distance from the interior point to the faces, expressed as an inradius in simplest radical form.The requested form is m√n/p, with the stated integer conditions on m, n, and p.Source: I Sub-question decomposition for… · p. 35
Loading 2504.11741v2…