PaperlayerView exact revision on arXiv

Source-linked AI summary

SHAPE of Chain-of-Thought in Math Reasoning

Jonghyun Song, Sangjun Song, Minjae Oh, Haesung Pyun, Sungsik Lee, Yohan Jo

arXiv:2608.28600v1cs.AI

TL;DR

Mathematical CoT can be accurate without revealing the mathematical skills or interpretations that organize it. SHAPE analyzes trajectories through heuristics and semantic spaces, finding stronger correctness signals in heuristic usage and focused semantic-space engagement, while post-training narrows heuristic strategies and heuristic-guided RLVR improves performance.

  • Problem

    Final-answer accuracy and existing CoT features provide limited insight into the specific mathematical interpretations and actions underlying LLM reasoning.

  • Method

    SHAPE represents CoT trajectories as sequences of semantic spaces and heuristics, then automatically annotates heuristic-bearing units and tracks interpretation changes.

  • Results

    Heuristic features predict correctness more reliably than existing CoT representations; correct trajectories focus on fewer spaces, post-training narrows heuristic usage, and heuristic-guided RLVR improves performance.

  • Takeaways & Limitations

    Process-level structure offers a diagnostic lens for LLM math reasoning and a supported route for improving post-training through heuristic information.

Abstract

from arXiv · show

Large language models (LLMs) achieve strong performance on mathematical reasoning benchmarks, yet the mathematically meaningful skills underlying their reasoning remain underexplored. We introduce \texttt{SHAPE}, a framework that analyzes Chain-of-Thought (CoT) trajectories through two lenses developed in mathematics education: (1) semantic spaces: the model's evolving mathematical interpretations of a problem (e.g., algebraic, geometric), and (2) heuristics: the specific mathematical actions taken within those spaces (e.g., simplifying the problem, working backward). We first use \texttt{SHAPE} to analyze the reasoning patterns of various models. Our findings reveal that the mathematical heuristics employed by a model better explain final answer correctness than traditional CoT features. Furthermore, models are likely to reach correct solutions by concentrating their reasoning effort within a few semantic spaces rather than exploring many disparate ones -- a pattern consistent with human behavior. Next, we utilize the \texttt{SHAPE} lens to evaluate whether post-training truly enhances mathematical proficiency. We find that reinforcement learning induces mode-seeking in heuristic usage. Lastly, we post-train LLMs by promoting diverse heuristics and demonstrate its effectiveness in improving accuracy. Overall, \texttt{SHAPE} provides a theoretically-grounded diagnostic framework for decoding LLM reasoning and offers a new path toward post-training LLMs for math reasoning. The code for our model is available at https://github.com/holi-lab/SHAPE-of-CoT

1. Introduction

SHAPE addresses the limited insight provided by final-answer accuracy and surface-level CoT features by analyzing mathematical reasoning through semantic spaces and heuristics. It shows that heuristic patterns better predict correctness, while successful reasoning concentrates effort within relatively few interpretations.

  • Motivation and framework: Existing CoT analyses examine length, lexical markers, reasoning episodes, or structural patterns, but do not track specific mathematical interpretations.
  • Motivation and framework: SHAPE represents LLM reasoning as evolving semantic spaces and purposeful heuristics, capturing both mathematical interpretation and problem-solving actions.A semantic space is defined by adopted objects, goals, and constraints; heuristics include actions such as introducing representations or exploring cases.
  • Findings: Heuristic-level features predict answer correctness more reliably and consistently across models than conventional CoT features.
  • Findings: Correct reasoning is associated with focused heuristic activity in a small number of semantic spaces, whereas incorrect trajectories scatter activity across disparate spaces.
  • Study scope: SHAPE is used to analyze model reasoning, assess post-training effects, and guide heuristic-informed post-training improvements.

2. SHAPE: Framework and Pipeline

SHAPE converts CoT trajectories into structured sequences of heuristics and semantic spaces using automated annotation and state tracking. Its metrics distinguish how reasoning effort is distributed across spaces from how often the trajectory transitions between contiguous space segments.

  • SHAPE Framework: SHAPE treats a CoT trajectory as an observable record of intermediate mathematical moves, interpretations, and changes during problem solving.
  • SHAPE Framework: Heuristics are purposeful problem-solving devices, including simplifying problems, working backward, changing representations, making cases, and arguing by contradiction.
  • Automated pipeline: The automated pipeline segments CoT text into content units, assigns heuristic labels, and tracks semantic-space changes as MAINTAIN, NEW, or RETURN.
  • SHAPE Framework: A semantic space captures the solver’s current mathematical interpretation, including its considered constraints, goals, and available actions.
  • Automated pipeline: The pipeline is grounded in a manually adjudicated gold set of 48 trajectories yielding 1,598 content units, with Qwen3.5-27B selected as the primary annotator model.
  • Metrics: SHAPE represents each trajectory with a semantic-space sequence S and heuristic sets H, then derives space and segment distributions from them.
  • Metrics: The space distribution merges visits to the same semantic space, while the segment distribution treats each contiguous visit separately.
  • Metrics: Effective-number metrics quantify how many semantic spaces or segments are meaningfully used, accounting for the distribution of heuristic activity.

3. Analysis on the Reasoning Patterns of LLMs

SHAPE evaluates whether heuristic-level and semantic-space features reveal mathematical reasoning patterns more effectively than conventional CoT statistics. Heuristic frequencies provide the strongest correctness signal, while reasoning models and incorrect trajectories exhibit distinct semantic-space traversal patterns.

  • Heuristic-Level Features: Heuristic features predict answer correctness more reliably than conventional CoT representations such as length, lexical markers, and episode labels.SHAPE is validated by comparing logistic-regression predictors built from these feature families.
  • Heuristic-Level Features: 0.504 ±0.03 AUROC for CoT length alone and 0.503 ±0.03 for length plus reasoning features indicate little correctness signal in surface-level token statistics.Self-revision markers are also evaluated as a conventional feature family.
  • Heuristic-Level Features: 0.664 ±0.02 AUROC is achieved by SHAPE frequency features, outperforming all compared baselines.ThinkARM episode frequencies reach 0.618 ±0.02, while the SHAPE result uses three features rather than eight.
  • Semantic-Space Metrics: Reasoning models consistently show larger semantic-space coverage and higher transition density than instruction-tuned models.The reported semantic-space coverage for one group is 1.81–2.53, with transition density ρ of 0.40–0.51, versus 1.37–1.71 and 0.19–0.32 for other models.
  • Semantic-Space Metrics: Incorrect trajectories generally have higher effective semantic-space transitions and transition ratio ρ than correct trajectories, reflecting more intensive revisitation rather than forward progress.This pattern is described as potentially related to overthinking.

4. Analysis on the Effects of Post-Training

SHAPE reveals that perturbed problems trigger structural changes in heuristic choices and semantic-space dynamics, but hard perturbations still produce large accuracy losses. It also shows that RL post-training narrows successful reasoning into a concentrated subset of the base model’s heuristic space.

  • 4.1. Perturbed Problems Induce Adaptive but Error-Like Reasoning: Hard perturbations substantially reduce Pass@1 across all four models.
  • 4.1. Perturbed Problems Induce Adaptive but Error-Like Reasoning: Models change local heuristics and global semantic-space dynamics when the required solution approach changes.Heuristic-frequency divergence appears within the first five and ten content units, indicating that adaptation is not merely a late-stage correction.
  • 4.1. Perturbed Problems Induce Adaptive but Error-Like Reasoning: Hard perturbations open more semantic spaces and revisit them more often without committing to a successful interpretation.
  • 4.2. Post-Training Narrows Heuristics Usage: Post-training concentrates successful trajectories in the dense core of the base heuristic distribution while leaving some base patterns unreproduced.Every post-trained model has Density above 1.0 and Coverage below 1.0; the unrelated-base baseline has Density 0.520 and Coverage 0.437.
  • 4.2. Post-Training Narrows Heuristics Usage: The high-Density and moderate-Coverage pattern is specific to base–post-trained pairs rather than generic overlap between models.

5. Heuristic-Augmented Reinforcement Learning

The authors test whether adding mathematical heuristics to planning-based RLVR improves training. Heuristic-augmented planning produces higher reported performance than planning alone across the evaluated splits.

  • Training Setup: Plan-GRPO adds planning, while HA-Plan-GRPO additionally supplies eleven mathematical heuristics during rollout.
  • Results: Both planning variants improve over the base model, confirming that the planning structure contributes to performance.
  • Results: HA-Plan-GRPO yields substantially higher Avg@64 and Pass@64 than Plan-GRPO.The variants differ only in the rollout prompt, so the reported gain is attributed to adding heuristic information there.

6. Related Work

Prior work analyzes CoT through surface and structural features and studies reinforcement learning’s effects on reasoning diversity. SHAPE instead applies mathematical problem-solving concepts to track both actions and the interpretations under which they occur.

  • Chain-of-thought analysis: Existing CoT analyses examine length, self-revision markers, cognitive episodes, graph structure, and faithfulness to internal computation.
  • Chain-of-thought analysis: SHAPE uses heuristics and semantic spaces to track mathematical actions together with the interpretations governing those actions.
  • Post-training for reasoning: Recent post-training studies report that RLVR may improve sampling efficiency without expanding reasoning repertoires and may induce diversity collapse or mode-seeking.

7. Conclusion

SHAPE is a process-level framework for analyzing mathematical CoT through semantic spaces and heuristics. The paper reports diagnostic and training benefits, while noting that the framework currently analyzes observable CoT traces on mathematical benchmarks.

  • 7. Conclusion: SHAPE analyzes CoT trajectories through semantic spaces and heuristics drawn from mathematical problem-solving research.
  • 7. Conclusion: Heuristic features predict answer correctness more reliably than existing CoT representations, while correct trajectories remain focused within fewer semantic spaces.
  • 7. Conclusion: For novel solution approaches, models change heuristics without expanding semantic-space scope, while post-training concentrates successful trajectories in the base distribution’s dense heuristic core.
  • 7. Conclusion: Adding heuristic information to RLVR improves performance across difficulty conditions.
  • 7. Conclusion: SHAPE currently analyzes observable CoT traces and has been validated only on mathematical benchmarks.Extending the framework to other domains is identified as future work.

B. Gold Heuristic Set Constructions

The gold heuristic set validates SHAPE’s annotation pipeline through a deliberately varied trajectory sample, explicit content-unit segmentation, and consensus adjudication. Borderline cases retain multiple labels when distinct heuristic interpretations are simultaneously supported.

  • Gold-set sampling: The gold set comprises 48 CoT trajectories from six MATH-Perturb seed problems, each paired with original and hard-perturbed versions across four models.The selection covers structurally perturbed settings and models with different reasoning modes and sizes.
  • Content-unit segmentation: Content units are the largest contiguous trajectory spans supporting one heuristic interpretation, with boundaries introduced when mathematical goals, actions, or problem representations change.Planning-to-execution, computation-to-checking, and new attempts at transformation are explicit boundary triggers.
  • Gold-set validation: Table 8 evaluates candidate heuristic annotators using weighted F1 for overall agreement and macro F1 for rare heuristic classes.The supplied table caption defines the two metrics but does not provide their numerical values.
  • Consensus coding: Four annotators coded trajectories at the content-unit level and resolved disagreements through discussion, contextual comparison, and consensus adjudication.This interpretive procedure follows prior think-aloud coding practice rather than treating labels as purely mechanical.
  • Borderline annotation: A representative counting-method example received H9a, H11b, and H5 because it supported numerical exploration, method verification, and reduction to smaller ranges.The adjudication retained all three labels because each captured a distinct role of the same unit.

C. Tagging Pipelines

The tagging pipeline separates heuristic segmentation and annotation from semantic-space tracking. It uses strategy-sensitive content units, a constrained heuristic code set, and a prompted state machine calibrated on annotated trajectories.

  • Heuristic tagging: Grok-4.1-Fast achieves the strongest overall heuristic-tagging performance, while open-weight Qwen3.5-27B performs comparably.Table 8 reports weighted and macro F1, while the supplied passage does not provide their values.
  • Heuristic tagging: Qwen3.5-27B shows particularly strong agreement on frequent classes H8, H11, and N, but lower performance on rare classes H7 and H10.Table 9 reports precision, recall, F1, Cohen’s kappa, and gold and predicted instance counts by class.
  • Semantic-space tracking: Semantic-space tracking is implemented as a prompted state machine because semantic spaces evolve relative to each trajectory and lack an objective gold standard.The prompt is calibrated through iterative manual review on 48 annotated trajectories.
  • Pipeline implementation: The automated pipeline segments indexed sentence windows within target ranges and returns only unit indices, codes, and concise reasoning.Guidebook placeholders are expanded at runtime from supplementary files, while prompt listings retain placeholders for readability.
  • Content-unit segmentation: Segmentation groups consecutive sentences by strategic role and starts a new content unit only when the active strategy or mathematical intent changes.Plans are merged with immediate execution, while weak monitoring phrases are merged with neighboring substantive units.
  • Heuristic annotation: The pipeline uses heuristic codes H1–H11, permits multi-tagging, and requires specific sub-codes when available.Non-heuristic codes are reserved for units containing no heuristic strategy, and the segmentation stage excludes N1–N4.

E. Early-Stage Heuristic Divergence Under Perturbation

Early-stage analyses show that hard perturbations alter heuristic-frequency distributions more than simple perturbations, indicating that models adjust mathematical actions near the beginning of solutions. This tactical shift does not broaden semantic-space scope.

  • Early-stage divergence: Hard perturbations produce larger heuristic-frequency divergence than simple perturbations in both the first 5 and first 10 content units.The analysis truncates each trajectory prefix and compares original–simple or original–hard heuristic distributions using Jensen–Shannon divergence.
  • Statistical interpretation: All hard-versus-simple comparisons pass a one-sided paired test at p < .05, supporting an early rather than merely late heuristic shift.The significance testing compares hard and simple perturbations across post-trained models.
  • Quantitative results: .19–.21 versus .15–.17: first-5-unit DfreqJS is higher under hard than simple perturbations.These ranges are reported for hard and simple perturbations, respectively.
  • Interpretation: Models change their mathematical actions near the start of a solution but continue reasoning within a similar semantic-space scope.Thus early tactical adaptation does not translate into broader semantic-space expansion.

F. Experiment Details

The experiments use explicit planning prompts and a heuristic-guided variant for GRPO training. The latter asks models to select and justify genuinely useful heuristics before carrying out each major reasoning step.

  • Training setup: GRPO training uses 2×NVIDIA B200 GPUs and replaces a {problem} placeholder with training-dataset problems.The experimental details specify the hardware and prompt substitution.
  • Planning prompt: The explicit-planning prompt requires a short goal-and-subgoal plan before step-by-step execution.The plan is intended to guide the next action rather than describe reasoning retrospectively.
  • Heuristic-guided planning: The heuristic-guided prompt requires choosing one mathematical heuristic, explaining its usefulness, and performing the corresponding reasoning move at each major step.The prompt provides eleven available heuristics spanning representation changes, decomposition, contradiction, exploration, backward reasoning, and verification.
  • Heuristic-guided planning: The heuristic list is not a checklist: prompts instruct models to use only heuristics that genuinely help solve the problem.This constraint is intended to prevent irrelevant heuristic insertion.

F.2. Training Details

The training setup uses GRPO for 200 steps on the MATH training splits, with a 2048-token maximum generation length. Training takes approximately 3–3.5 hours, and GRPO is implemented in verl.

  • 200 training steps are run on the MATH training splits.
  • Training takes approximately 3–3.5 hours.
  • The maximum generation length is set to 2048 for training efficiency.
  • GRPO is implemented in verl.
Loading 2608.28600v1…