PaperlayerView exact revision on arXiv

Source-linked AI summary

The Missing Primitive: Diagnosing and Repairing Mathematical Reasoning in Large Language Models

Shuo Xing, Zilin Dai, Chengyuan Qian, Fangzhou Lin, Wenjing Chen, Ping He, Pan Lu, Alvaro Velasquez, Mohit Bansal, Zhengzhong Tu

arXiv:2610.02191v1cs.LG

TL;DR

Outcome-level measures leave unresolved whether LLMs understand the mathematical structure behind their solutions. The paper introduces PRIM to diagnose that understanding and ABSORB to transfer primitive-guided reasoning, reporting improved average performance across scales and challenging benchmarks.

  • Problem

    Outcome-level accuracy and proof verifiability provide limited insight into whether mathematical success reflects discovering solution structure or executing a derivation.

    Source: §1 · p. 1

  • Method

    The paper defines Mathematical Primitives as compact structural explanations, evaluates four reasoning dimensions with PRIM, and uses ABSORB to selectively transfer primitive-guided reasoning.

    Source: §1 · p. 2, §2.1 · p. 3, §2.2 · p. 3

  • Results

    ABSORB improves average performance by 2.42–4.48 points across model scales and challenging mathematical reasoning benchmarks, while independent Discovery is the dominant bottleneck.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results highlight the value of explicitly diagnosing structural mathematical understanding and leveraging that structure to improve LLM reasoning.

    Source: §6 · p. 10

Abstract

from arXiv · show

While Large Language Models (LLMs) have demonstrated striking capabilities on frontier mathematical problems, it remains unclear whether they possess the structural mathematical understanding underlying their solutions. In this paper, we take a first step toward systematically studying mathematical understanding in LLMs, from diagnosing its distinct capabilities to leveraging these findings to improve post-training. First, we introduce the notion of Mathematical Primitive to probe structural mathematical understanding and propose \hlei{}, a novel benchmark that evaluates mathematical reasoning along four distinct dimensions: Discovery, Generation, Digestion, and Execution. Second, our systematic diagnosis shows that solution accuracy masks distinct capability profiles, primitives unlock substantial latent execution capacity, and Discovery is the dominant bottleneck in mathematical reasoning. Our post-training analysis further shows that discovery-limited failures are particularly amenable to repair. Finally, building on these findings, we introduce \abs{}, a primitive-privileged self-distillation framework that selectively transfers primitive-guided reasoning into the student model. Extensive experiments demonstrate that \abs{} consistently improves mathematical reasoning over baselines across model scales and challenging benchmarks.

1 INTRODUCTION

The paper frames structural mathematical understanding as distinct from obtaining correct answers, a gap obscured by outcome-level metrics. It introduces PRIM to diagnose that structure and ABSORB to transfer primitive-guided reasoning, reporting average gains across model scales and challenging benchmarks.

1 INTRODUCTION

  • Outcome-level accuracy and proof verifiability leave unclear whether success reflects discovering solution structure or merely executing a derivation.Source: §1 · p. 1
  • Diagnosis finds that similar solving accuracy can mask different capability profiles, correct primitives unlock latent execution, and independent Discovery is the dominant bottleneck.Source: §1 · p. 2
  • Discovery-limited failures are more amenable to post-training, while existing methods can introduce regressions on problems the model already solves.Source: §1 · p. 2
  • ABSORB improves average performance by 2.42–4.48 points across multiple model scales and challenging mathematical reasoning benchmarks.Source: §1 · p. 2
  • PRIM evaluates structural mathematical understanding through Discovery, Generation, Digestion, and Execution, while ABSORB selectively transfers primitive-guided reasoning into the student.Source: §1 · p. 2

2 MATHEMATICAL PRIMITIVES AS STRUCTURAL UNDERSTANDING

The paper separates structural understanding from procedural execution by defining Mathematical Primitives as load-bearing conceptual observations and evaluating reasoning across four dimensions. PRIM operationalizes this evaluation with a curated benchmark and a rubric for judging predicted primitives.

2 MATHEMATICAL PRIMITIVES AS STRUCTURAL UNDERSTANDING

  • PRIM evaluates mathematical reasoning across Generation, Discovery, Digestion, and Execution, separating direct solving from identifying, extracting, or using a primitive.Source: §2 · p. 3, §2.2 · p. 3

2.1 MATHEMATICAL PRIMITIVE

  • A Mathematical Primitive is a concise, problem-specific conceptual observation that exposes the structure making a solution possible, rather than generic advice, routine calculation, or a full proof.Source: §2.1 · p. 3

2.2 FOUR DIMENSIONS OF MATHEMATICAL UNDERSTANDING

  • The four dimensions assess independent primitive discovery, direct solution generation, primitive extraction from a correct solution, and solving when the primitive is supplied.Source: §2.2 · p. 3

2.3 THE PRIM BENCHMARK

  • The evaluation set contains 182 problems sampled from HLE-VERIFIED Gold and Revision subsets after three graduate-level mathematics experts reviewed annotations and excluded 18 unsuitable problems.Source: §2.3 · p. 3
  • PRIM assigns problems to RECAST, WITNESS, or ARGUMENT primitive families and scores predictions for validity, essential structure, and explanation of how that structure enables the solution.Source: §2.3 · p. 3
  • PrimitiveAcc counts a prediction as accurate at a score threshold of τ = 0.8, with validity and structural identification required and mechanism quality refining the score.Source: §2.3 · p. 3

3 DIAGNOSING MATHEMATICAL REASONING

Across 12 models, PRIM separates structural discovery from execution and reveals capability profiles obscured by final-answer accuracy. Primitive guidance exposes latent solving capacity, while discovery-limited failures are repaired more often than failures with execution deficits.

3.1 EVALUATION SETUP

  • The evaluation covers 12 open- and closed-source models across three lineages, measuring Generation, Discovery, Digestion, and Execution on PRIM.Discovery and Digestion assess primitive identification; Generation and Execution use final-answer accuracy.Source: §3 · p. 4, §3.1 · p. 4, Table 1 · p. 4

3.2 RESULTS ANALYSIS

  • 17.58 to 29.67 percentage points of improvement across all 12 models followed primitive guidance, with Qwen execution competitive with closed-source GPT-5.4 variants.The gains indicate that direct-generation failures can mask downstream problem-solving ability when the relevant structure is supplied.Source: Table 1 · p. 4, §3.2 · p. 5
  • Similar Generation accuracy can conceal distinct profiles: OpenAI models are more primitive-forward, Qwen shows more D−G+ cases, and distilled DeepSeek models more joint failures.Discovery and Generation are positively associated overall, but models with comparable solving accuracy can differ sharply in Discovery.Source: Figure 2 · p. 4, §3.2 · p. 5
  • Teacher Primitives yield larger gains than Teacher Plans, while conditioning on a model’s own primitive provides little benefit or can degrade performance.Gold Primitives further maximize the effect, supporting a distinction between exposing load-bearing structure and providing generic procedural scaffolding.Source: Table 2 · p. 5, §3.2 · pp. 5–6
  • Qwen3.6-27B rises from 24.73% Discovery to 92.31% Digestion, and R1-Distill-32B from 6.04% to 65.38%, showing stronger retrospective recognition than independent discovery.Models often recover the organizing primitive from a correct solution even when they struggle to identify it from the problem alone.Source: §3.2 · p. 6
  • 83.6% of Generation failures fall in the D− regime, including many D−E+ cases where supplying the primitive enables success.The decomposition locates failed independent discovery as the dominant failure pattern, with latent execution capacity present in many cases.Source: Figure 4 · p. 6, §3.2 · p. 6

3.3 FURTHER ANALYSIS: WHICH FAILURES ARE REPAIRABLE?

  • The post-training study uses SFT and OPSD on three Qwen-3.5 scales, with 709 Mathematics Ph.D. qualifying problems paired with human-authored proofs.The authors focus on text-supervised training because these proof-oriented problems generally lack algorithmically verifiable short answers.Source: §3.3 · p. 6
  • 20.4% and 21.1% of D−E+ cases were repaired by SFT and OPSD, versus 6.8% and 8.2% of D−E− cases; net Generation gains remained modest.This ordering holds at each model scale; regressions in initially correct cases offset some repairs.Source: Table 3 · p. 7, §3.3 · p. 7

4 INTERNALIZING PRIMITIVE-GUIDED REASONING

ABSORB uses mathematical primitives as privileged teacher information to guide, but not replace, the student’s reasoning. Across model scales and benchmarks, it improves Generation while its gains need not appear as better explicit Discovery.

4.1 PRIMITIVES AS PRIVILEGED INFORMATION

  • Primitive-conditioned guidance targets missing structural information without prescribing execution steps the student can already perform.Primitives guide the student’s own generation process without being required at inference time.Source: §4.1 · p. 7, §4 · p. 7

4.4 EXPERIMENTAL RESULTS

  • ABSORB improves Generation at all three Qwen3.5 model scales, with a 5.49-point gain at 9B and a 3.73-point improvement in average benchmark performance.Source: Table 4 · p. 8, §4.4 · p. 9

4.2 BOUNDED OVERRIDE

  • ABSORB transfers positive teacher guidance fully while bounding strong suppression of student-preferred choices that may reflect privileged information.Positive guidance toward better alternatives remains active, so bounding negative pressure does not freeze incorrect preferences.Source: §4.2 · pp. 7–8

4.3 OVERALL OBJECTIVE

  • The objective conditions the teacher on the primitive and bounds how strongly its guidance can override the student’s reasoning.At each step, it uses the teacher’s top-K token support and student and teacher distributions renormalized over that support.Source: §4.3 · p. 8

5 RELATED WORK

The related work surveys outcome-based and process-oriented mathematical reasoning evaluation, alongside on-policy distillation methods. ABSORB is distinguished by using a mathematical primitive as privileged information and selectively transferring its guidance.

Mathematical Reasoning in LLMs

  • Mathematical benchmarks have grown more difficult, while evaluation remains predominantly anchored to final-answer correctness.Process-oriented alternatives include cognitive-dimension evaluation, minimal procedural skeletons, proof sketches, and executable abstractions; Mathematical Primitives target the concise, non-procedural idea underlying a solution.Source: §5 · p. 9

On-Policy Self-Distillation

  • On-policy self-distillation supervises student-generated trajectories with a teacher conditioned on privileged information, but naive versions can leak privileges and suppress exploration or self-correction.Prior approaches address transferable supervision, high-entropy decisions, teacher–student disagreement, or multiple privileged views.Source: §5 · p. 9
  • ABSORB uses a primitive rather than a full solution as privileged information and selectively transfers that guidance through bounded override.Source: §5 · p. 9

6 CONCLUSION

The paper introduces PRIM to evaluate structural mathematical understanding and reports that Discovery is a dominant bottleneck despite latent execution ability. It proposes ABSORB, which selectively transfers primitive-guided reasoning and improves over strong post-training baselines.

6 CONCLUSION

  • PRIM evaluates structural mathematical understanding across Discovery, Generation, Digestion, and Execution.The paper defines Mathematical Primitives as compact structural ideas underlying solutions.Source: §6 · p. 9
  • Similar solution accuracy can mask distinct capability profiles, correct primitives can unlock latent execution capacity, and independent Discovery is a dominant bottleneck.Discovery-limited failures are reported as more amenable to post-training, while existing methods can introduce regressions on problems already solved.Source: §6 · p. 9
  • ABSORB selectively transfers primitive-guided reasoning through bounded override and consistently improves over strong post-training baselines on challenging benchmarks.The paper frames these results as supporting the value of diagnosing structural understanding and leveraging it to improve reasoning.Source: §6 · pp. 9–10

A.1 DATASET FILTERING AND CORRECTION

The appendix documents benchmark filtering and correction, primitive-family and scoring procedures, and supplementary analyses of performance and model behavior. It also describes teacher prompts and explains why curated HLE-subset accuracy differs from full-HLE performance.

A.1 DATASET FILTERING AND CORRECTION

  • 11 predictions flipped from incorrect to correct after answer corrections, with no reverse flips.Expert review corrected 15 retained problems’ official answers, including 13 mathematical errors and 2 corrupted answer strings; the final benchmark contains 182 problems.Source: §A.1 · p. 14

A.2 PRIMITIVE TYPE AND FAMILY

  • The rubric routes ten frozen primitive types into RECAST, WITNESS, or ARGUMENT families, each defined by its load-bearing mathematical object.The families assess whether a model identifies the relevant target and explains how it enables the solution.Source: §A.2 · pp. 14–15, Table 6 · p. 15

A.3 PRIMITIVE SCORING PROTOCOL

  • Scoring separates primitive validity from whether the response identifies the essential gate and explains its operative mechanism.Judge–human agreement was 95.5% (Cohen’s κ = 0.91); one valid alternative primitive was found among 100 audited primitive-incorrect cases.Source: §A.3 · pp. 15–16

A.4 ILLUSTRATIVE EXAMPLE OF MATHEMATICAL PRIMITIVES

  • A primitive is presented as a compact structural idea that makes a derivation possible, not as a shortened proof.The illustrative figure shows only the core_concept field, contrasting it with the full rationale.Source: §A.4 · p. 16

A.5 PROMPT ON EXTRACTING A MATHEMATICAL PRIMITIVE AND PRIMITIVE SCORING

  • Curation prompts ask a teacher to identify and classify the essential mathematical property and its solution principle while avoiding calculations or solution steps.Primitive validity, gate, and mechanism are assessed separately, with semantic agreement prioritized over exact wording or downstream execution details.Source: §A.5 · p. 16, Figure 7 · p. 18, Figure 8 · p. 19

B.1 COMPARISON WITH FULL-HLE PERFORMANCE

  • Curated PRIM HLE-subset accuracy is not directly comparable to full-HLE performance because the subset excludes noisy or ambiguous examples.The discussion cites incorrect reference answers, rationale gaps, and irregular formatting as examples of removed issues.Source: §B.1 · p. 16

B.2 PROMPT ON GENERATING THE PRIMITIVE AND PLAN USING GPT-5.4

  • GPT-5.4 prompts generate either a mathematical primitive or a step-by-step solution plan for language-only problems.The primitive prompt defines it as the essential property and explanation connecting a problem to a valid solution principle.Source: §B.2 · pp. 16, 18, Figure 10 · p. 20, Figure 11 · p. 20

B.3.1 PERFORMANCE ACROSS PRIMITIVE FAMILIES; B.3.2 REASONING LENGTH AND GENERATION BEHAVIOR

  • Supplementary analyses stratify PRIM results by primitive family and compare Generation output-token distributions across models and answer correctness.The family breakdown covers Recast, Witness, and Argument; the distribution figure separates all rollouts from rollouts by final-answer correctness.Source: §B.3.1 · p. 21, Table 7 · p. 21, §B.3.2 · p. 21

B.3.3 EXECUTION UNDER DIFFERENT PRIMITIVE SOURCES

Externally supplied teacher primitives substantially improve Execution across model families, with gold primitives yielding the largest gains. The results indicate that supplying relevant mathematical structure—not merely splitting generation into discovery and execution—is key to the rescue effect.

B.3.3 EXECUTION UNDER DIFFERENT PRIMITIVE SOURCES

  • Externally supplied teacher primitives substantially improve Execution across model families, with gold primitives yielding the largest gains.Conditioning on self-generated primitives provides little or no benefit and often degrades performance.Source: §B.3.3 · p. 22, Table 8 · p. 22
  • The rescue effect is not explained by simply decomposing generation into a discover-then-execute pipeline; it depends on supplying relevant mathematical structure.Models often fail to identify that structure on their own.Source: §B.3.3 · p. 22
  • Table 8 compares Generation and Execution under self-generated, teacher-provided, and gold primitives, reporting percentage-point changes relative to Generation.The teacher-primitive result for gpt-5.4 is omitted because gpt-5.4 serves as the teacher model.Source: Table 8 · p. 22

B.3.4 FAILURE DECOMPOSITION AND RESCUE ANALYSIS

Generation failures are concentrated in discovery-negative regimes, and discovery-limited failures are more repairable than capability-limited failures. Digestion also relates to rescue under teacher-primitive guidance, though the strength of that relationship varies across models.

B.3.4 FAILURE DECOMPOSITION AND RESCUE ANALYSIS

  • Across model families, Generation failures are concentrated in D− regimes, including cases where models can execute a primitive once it is supplied.Table 9 categorizes direct Generation failures by primitive discovery and execution.Source: §B.3.4 · p. 23
  • For most models, digestible failures are more likely to be rescued by teacher primitives; Qwen3.5 shows 17.6–28.6 percentage-point advantages.The relationship is weaker for DeepSeek-R1 distill models and noisy when there are very few undigestible examples; Table 10 flags estimates based on fewer than 20 such problems as high variance.Source: §B.3.4 · p. 23, Table 10 · p. 23

B.3.5 REPAIRABILITY ACROSS FAILURE MODES

  • Across SFT, OPSD, and ABSORB, discovery-limited D−E+ failures are repaired substantially more often than capability-limited D−E− failures.Utilization-limited D+E+ cases have relatively high repair rates, while execution-limited D+E− rates vary more across methods.Source: §B.3.5 · p. 23, Table 11 · p. 24

C POST-TRAINING DATASET DETAILS

The post-training corpus comprises 709 Mathematics Ph.D. qualifying-examination problems spanning multiple universities and mathematical domains. It uses open-ended proof problems paired with human-authored proofs and extracted mathematical primitives.

C POST-TRAINING DATASET DETAILS

  • The corpus contains 709 Mathematics Ph.D. qualifying-examination problems spanning 1991–2026, categorized by source university and mathematical domain.The problems are strictly parsed into LATEX.Source: C POST-TRAINING DATASET DETAILS · p. 24, Table 12 · p. 24
  • Every problem is an open-ended proof task without an algorithmically verifiable short-answer label, paired with a human-authored reference proof.Reference proofs have a median length of 118 words; each problem also has a one-sentence primitive extracted by GPT-5.4-High using Figure 7's curation prompt.Source: C POST-TRAINING DATASET DETAILS · p. 24
  • Qualifying-examination problems were selected because their difficulty and open-ended reasoning demands closely match PRIM and require mathematical maturity and structural reasoning.Source: C POST-TRAINING DATASET DETAILS · p. 24

D POST-TRAINING SETUP

ABSORB and OPSD share a training recipe but differ in the privileged context provided to the teacher. Both train on student on-policy rollouts from the 709-problem corpus, with ABSORB optimized for one epoch.

D POST-TRAINING SETUP

  • ABSORB conditions the teacher on the mathematical primitive, whereas OPSD conditions it on the reference solution; both use the same training recipe and 709-problem corpus.Source: D POST-TRAINING SETUP · p. 24
  • For each problem, the student generates one on-policy rollout from the problem alone at temperature 1.0, top-p 0.95, top-k 20, with a 4096-token completion limit and 6144-token context.Source: D POST-TRAINING SETUP · p. 24
  • The teacher uses the same base model with its adapter disabled and receives the corresponding privileged context.Source: D POST-TRAINING SETUP · p. 24
  • ABSORB uses top-K support K = 128 and clamp threshold τ = 0.06; LoRA adapters are trained for one epoch, or 22–23 optimizer steps.The setup also specifies rank 64, α = 128, learning rate 5 × 10^-6, gradient clipping at 0.1, and effective batch size 32.Source: D POST-TRAINING SETUP · p. 24

E POST-TRAINING ABLATION STUDIES

Ablations on Qwen3.5-9B using PRIM (n = 182) test which teacher privilege, transfer mechanism, and student rollout target support ABSORB.

Teacher Conditioning

Transfer Mechanism

Divergence Control

  • Masking about 5% of tokens removes approximately 86% of divergence mass and partially recovers Generation from 31.87 to 34.62.The divergence-based mask uses an offline 95th-percentile threshold.Source: E POST-TRAINING ABLATION STUDIES · pp. 25–26

Entropy Control

  • Generation returns to 31.87 despite masking fewer tokens, suggesting neither divergence magnitude nor student confidence cleanly separates beneficial from harmful teacher intervention.This variant masks only high-divergence, low-entropy positions.Source: E POST-TRAINING ABLATION STUDIES · p. 26

Transfer Mechanism

  • 43.96 Generation is the only transfer-mechanism result that improves over the base model, with bounded override retaining supervision at every position while limiting teacher suppression of student-supported choices.The comparison fixes primitive-conditioned teacher guidance and solution-generation rollouts while varying transfer mechanisms.Source: Table 14 · p. 25, E POST-TRAINING ABLATION STUDIES · p. 26

Rollout Target

  • 39.01 Generation results when the student generates primitives under the ABSORB objective, versus 43.96 when primitives guide the teacher and the student generates solutions.Primitive-target rollouts remain weak across the three tested objectives.Source: E POST-TRAINING ABLATION STUDIES · p. 26, Table 15 · p. 26

Alternative Training Objectives

  • TOP-D yields 31.87 Generation and 9.34 Discovery; solution SFT yields 32.42 Generation, while primitive SFT yields 28.02 Generation and 2.75 Discovery.TOP-D uses the same privileged signal, training corpus, and self-teacher as ABSORB; the SFT variants directly supervise solutions or primitives.Source: E POST-TRAINING ABLATION STUDIES · pp. 26–27, Table 16 · p. 27
  • The dual-channel alternative substantially underperforms both the base model and ABSORB, but uses a larger training pool, full-length primitive annotations, and 8k-token completions.It is presented as a reference alternative rather than a strict single-variable ablation.Source: E POST-TRAINING ABLATION STUDIES · p. 27, Table 16 · p. 27
Loading 2610.02191v1…