PaperlayerView exact revision on arXiv

Source-linked AI summary

INSPIRE: An Internalize-Then-Improve Approach for Example-Driven Mathematical Reasoning

Shuai Wang, Jiayi Kuang, Yinghui Li, Haojing Huang, Xinnian Liang, Ying Shen, Liang Lin

arXiv:2608.27501v1cs.CL

TL;DR

Most LLM mathematical-reasoning methods prioritize final-answer correctness, while example-based reasoning remains underdeveloped. INSPIRE combines reference-guided internalization with staged preference training, and experiments report consistent improvements across model scales and families with maintained or improved out-of-distribution performance.

  • Problem

    Most mathematical-reasoning methods optimize final-answer correctness, while example-based reasoning—a marker of conceptual understanding—remains underdeveloped in current LLMs.

    Source: §1 · p. 1

  • Method

    INSPIRE uses Reference-Guided Student Internalization to produce preference candidates while avoiding distribution shift, followed by staged training for strategy adoption and correctness refinement.

    Source: §1 · p. 2

  • Results

    Experiments across model scales and families show consistent improvements, with maintained or improved performance on out-of-distribution benchmarks.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Decomposing capability acquisition into strategy adoption and correctness refinement is more effective than joint optimization.

    Source: §5 · p. 8

    Training uses 1,275 BrokenMath problems from limited mathematical domains, and effectiveness at larger scales has not been examined.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Mathematical reasoning has seen rapid progress in large language models (LLMs), yet existing methods optimize predominantly for final-answer correctness, raising the question whether models truly internalize mathematical concepts or merely memorize solution patterns. In human mathematics education, example-based reasoning such as constructing counterexamples to test theorem boundaries reflects deep conceptual understanding, but remains underdeveloped in current LLMs. Enhancing this capability through preference optimization presents two key challenges: (1) the model's limited example-based reasoning ability makes constructing effective preference pairs inherently difficult; and (2) capability acquisition is progressive, as the model must first learn to adopt this strategy before learning to apply it correctly. Therefore we propose INSPIRE, an Internalize-Then-Improve approach combining Reference-Guided Student Internalization (RGSI), which produces high-quality preference candidates under the policy model's own distribution, with a stage-wise rubric preference training strategy that decomposes learning into method-oriented and correctness-oriented stages. Experiments across multiple model scales and families demonstrate consistent improvements, even surpassing larger open-source models, while evaluations on out-of-distribution benchmarks confirm no degradation in general mathematical reasoning ability.

1 Introduction

INSPIRE addresses the limited example-driven mathematical reasoning ability of LLMs with a two-stage preference-optimization approach. It aims to first teach models to adopt example-based reasoning and then improve answer correctness.

1 Introduction

  • Most prior methods optimize final-answer correctness, while example-based reasoning—a skill associated with deep conceptual understanding—remains underdeveloped in LLMs.Models often rely on abstract theorem application rather than constructing concrete examples to test their reasoning.Source: §1 · p. 1
  • Preference optimization faces low-quality self-sampled candidates and distribution shift from external preferred references, while the model must learn to adopt example-based reasoning before applying it correctly.Source: §1 · pp. 1–2
  • Experiments show consistent improvements across multiple model scales and families, with ablations and analyses supporting the design choices.Source: §1 · p. 2
  • INSPIRE combines Reference-Guided Student Internalization to create high-quality preference candidates in the model’s own style with staged training for reasoning adoption and correctness.RGSI uses reference reasoning approaches to help the model regenerate responses in its own expression style, avoiding distribution shift; training first encourages example-based reasoning, then refines correctness.Source: §1 · p. 2

2 Related Work

Prior mathematical-reasoning methods largely optimize final-answer correctness, leaving concept-level reasoning through examples and counterexamples underdeveloped. Preference optimization also faces challenges in constructing useful preference pairs and moving beyond correctness-only signals.

Mathematical Reasoning in LLMs

  • Prior math-reasoning work scales supervised fine-tuning and data augmentation (Yu et al., 2024) and uses reinforcement learning to elicit self-verification and reflection.The passage identifies instruction synthesis, question rephrasing, and corpus expansion as data-scaling strategies, and cites GRPO, OpenAI o1, DeepSeek-R1, and DAPO as reinforcement-learning efforts.Source: §2 · p. 2
  • ConceptMath (Wu et al., 2024) and CounterMath (Li et al., 2025b) reveal gaps in concept-level reasoning, while targeted training to elicit example-based analysis remains largely unexplored.The cited work highlights reasoning about concepts and discriminating through examples and counterexamples as areas needing improvement.Source: §2 · p. 2

Preference Optimization for Reasoning

  • Mathematical preference methods include DPO and variants (Rafailov et al., 2023; Azar et al., 2024; Meng et al., 2024), step-level optimization (Lai et al., 2024; Xu et al., 2025), process rewards (Wang et al., 2024), and rubric-based criteria (Gunjal et al., 2025; Huang et al., 2025).The passage notes that step-based methods report notable results on long-chain reasoning tasks and rubric-based evaluation provides richer signals than single-dimensional rewards.Source: §2 · p. 2
  • Preference-pair construction remains difficult: rejection sampling (Yuan et al., 2024) and iterative online DPO (Xu et al., 2024) are common strategies.The passage identifies generating high-quality preference data under the policy model’s own distribution as a key challenge.Source: §2 · p. 2
  • Weak target models may yield too little on-policy quality variation, stronger off-policy data risks distribution shift, and correctness-only signals leave multidimensional evaluation and progressive training unexplored.The passage describes these limitations as challenges for preference optimization in reasoning.Source: §2 · p. 2

3 Methodology

INSPIRE builds preference data with a three-dimensional rubric and trains in two stages, first promoting example-based reasoning and then refining correctness while preserving the acquired strategy.

3.1 Rubric-based Preference Data Construction

  • The rubric scores Method, Reasoning, and Correctness separately, distinguishing example-based analysis from coherent argumentation and final-answer accuracy.DeepseekV3.2 independently scores each response on the three dimensions.Source: §3.1 · p. 3
  • RGSI conditions policy-model generation on both the problem and reference solution, producing on-policy candidates with stronger method use and reasoning than self-sampling.This addresses the lack of useful quality differences when the base model's self-sampled candidates are uniformly weak in example-based reasoning.Source: §3.1 · p. 4

3.2 Stage-wise Rubric Preference Training

  • M-DPO uses method-based preference pairs with a clear method-score separation and reasoning-quality gates, prioritizing preferred responses with stronger reasoning.The SFT model π0 initializes and anchors this stage; its output π1 starts Stage 2.Source: §3.2 · p. 4
  • C-DPO uses correct-versus-incorrect pairs with a sound-reasoning constraint, adds reasoning-quality comparisons among correct answers, and retains some Stage 1 pairs to limit forgetting.Method score remains a secondary ranking criterion to reinforce the acquired strategy.Source: §3.2 · pp. 4–5

1: // Data Preparation

Data preparation scores responses with rubric J over method, reasoning, and correctness, then organizes preference learning into method-oriented and correctness-oriented stages.

Data Preparation

  • Stage 1 constructs D1(d) pairs where the preferred response leads in method score by at least 1 and both responses have reasoning scores of at least 1.Pairs prioritize higher reasoning scores for the preferred response.Source: §1 · p. 5
  • Responses are scored with J on (m, r, c) for both Stage 1 and Stage 2 preference learning.Stage 2 is labeled Correctness-Oriented Preference Refinement (C-DPO).Source: §1 · p. 5

4 Experiments

Across three model configurations, staged training improves example-driven reasoning on CounterMath, while the final models maintain or improve performance on general mathematical reasoning benchmarks. Ablations and analyses examine the training stages, RGSI, example usage, and judge robustness.

4.2 Main Results

  • Stage 1 raises 7B Examples from 77.63% to 81.33% and F1 from 39.06 to 43.05; Stage 2 reaches F1 45.91 and Examples 84.79%.The same progressive pattern holds at 1.5B (F1: +2.66, Ex.: +7.08) and for Llama-3.1-8B-Instruct (F1: +7.29, Ex.: +12.99); alignment metrics improve steadily, while SFT gains are marginal.Source: §4.2 · p. 6, Table 2 · p. 6
  • The trained 7B surpasses Qwen2.5-Math-72B-Instruct (F1: 41.89), approaches Qwen3-32B (F1: 45.93), and both trained models exceed DeepSeek-R1 in Examples.Qwen-7B reaches 84.79% Examples and Llama-8B 93.09%, both above DeepSeek-R1 (80.59%); the fully trained 1.5B model (F1: 39.02) is comparable to the 7B base model (F1: 38.67).Source: §4.2 · p. 6, Table 2 · p. 6

4.3 Out-of-Distribution Evaluation

  • The final model maintains or improves results across five out-of-distribution benchmarks and all three configurations; Qwen2.5-Math-7B gains +6.67 on AIME 2024.The benchmarks are GSM8K, MATH500, AIME 2024, GAOKAOmathQA, and MMLU-collegeMath; unchanged AIME scores for the two smaller configurations are attributed to the benchmark’s 30-problem granularity and difficulty.Source: §4.3 · p. 7, Table 3 · p. 7

4.4 Ablation Studies

  • Method-oriented M-DPO alone (43.05) outperforms correctness-oriented C-DPO alone (40.27), while the full sequential pipeline yields the best results across evaluated metrics.Mixed DPO scores 42.72 versus 43.05 for Stage 1 despite using more data; chosen SFT reaches F1: 41.75 versus 45.91 for the full pipeline, and correctness-only DPO reduces example usage (73.19% vs. 77.63%).Source: §4.4 · p. 7, Table 4 · p. 7
  • RGSI produces useful preference signals: self-sampling yields marginal improvement, while teacher rewriting fails because its generation style shifts from the student’s.Self-sampling lacks sufficient candidate-quality variation; teacher rewriting creates trivially separable pairs, whereas RGSI maintains moderate margins and gradual accuracy growth during training.Source: §4.4 · pp. 7–8, Table 5 · p. 7, Figure 3 · p. 8

4.5 Analysis

  • Hint prompting increases example usage but yields only marginal judgment-accuracy improvement (F1: +0.46), compared with +7.24 F1 from training.The analysis attributes the difference to training learning to construct appropriate examples that contribute to sound judgments, whereas prompted examples are insufficiently targeted.Source: §4.5 · p. 8, Table 6 · p. 8
  • After Stage 1, substantive example usage (m = 2) rises from 30.74% to 39.30%, while purely abstract reasoning (m = 0) falls from 56.58% to 48.47%.A case study illustrates a progressive shift from abstract reasoning to targeted counterexample construction.Source: §4.5 · p. 8
  • Progressive improvement across stages also holds under GPT-4o evaluation, despite absolute score shifts from judge calibration.The study re-evaluates all stages with GPT-4o to check for bias from the primary DeepSeek-V3.2 judge.Source: §4.5 · p. 8, Table 7 · p. 8

5 Conclusion … Hint Prompt

INSPIRE combines Reference-Guided Student Internalization with stage-wise rubric preference training, and the paper reports consistent improvements while maintaining or improving out-of-distribution performance. The conclusion attributes this to separating strategy adoption from correctness refinement, while noting limits in data scale, model scale, and evaluation.

5 Conclusion

  • INSPIRE improves example-driven mathematical reasoning across model scales and families, maintains or improves out-of-distribution performance, and benefits from separating strategy adoption from correctness refinement.The authors propose exploring larger datasets and models and extending the approach to other underexplored reasoning strategies.Source: §5 · p. 8

Limitations

  • The study uses 1,275 BrokenMath problems, evaluates models only up to 7B, relies on DeepSeek-V3.2 as judge, and reports single runs with fixed seeds.Generalization to broader domains and 32B or 72B models remains unexamined; GPT-4o cross-validation confirmed consistent trends, and three model configurations showed consistent improvements.Source: Limitations · p. 9

A Evaluation Rubric Prompts; Scoring Prompt

Scoring Prompt

  • Method scores range from 0 for no concrete objects to 2 for objects playing a critical role, while Reasoning scores range from 0 for chaotic logic to 3 for complete, clear justification.The scoring prompt first identifies concrete mathematical objects and assesses whether they advance the proof through existence, counterexamples, key properties, or reasoning steps.Source: Scoring Prompt · p. 12

B Problem Filtering Criteria; Filtering Prompt

  • Problem filtering uses DeepSeek-R1 to find BrokenMath candidates, retaining proofs with concrete instances that directly aid understanding or validation through an essential reasoning role.The criteria include demonstrating existence, refuting universality, illustrating key properties, or validating intermediate steps by computation.Source: B Problem Filtering Criteria · p. 13, Filtering Prompt · p. 13

C Reference-Guided Student Internalization

Rewriting Prompt

  • The rewriting prompt asks the model to analyze step by step, use the hint as inspiration, write the full solution in its own words, and box the final answer.Source: Rewriting Prompt · p. 14

D Inference Prompt Templates; Standard Prompt; Hint Prompt

  • Evaluation uses a standard step-by-step prompt for main experiments and a hint prompt that explicitly asks for example-based reasoning.The prompts respectively ask whether the statement is true or false through step-by-step reasoning or by giving examples, with the answer boxed.Source: D Inference Prompt Templates · p. 14, Hint Prompt · p. 14

E Training and Details

The section specifies how example-based reasoning is evaluated and reports the SFT and DPO training settings shared across model scales.

E Training and Details

  • Evaluation follows CounterMath (Li et al., 2025b), using Macro-F1, Examples, Strict Align, and Loose Align, computed with DeepSeek-V3.2 as judge.The evaluation uses the same prompts as CounterMath.Source: E Training and Details · p. 14
  • SFT uses a 1 × 10−5 learning rate for 3 epochs; DPO uses 5 × 10−6, β = 0.2, for 3 epochs at both model scales.Source: E Training and Details · p. 14

F Additional Analysis · G Detailed Rubric Criteria

The additional analysis finds that Stage 1 increases substantive example use, while Stage 2 supports targeted counterexamples; the appendix also provides detailed scoring criteria for the evaluation rubric.

F Additional Analysis

  • The analysis samples 16 responses for each of 1,275 problems from both π0 and π1, retaining 20,245 and 20,171 valid responses, respectively.Responses are self-sampled without RGSI responses and scored and parsed using DeepSeek-V3.2.Source: F Additional Analysis · p. 15
  • Substantive example usage rises from 30.74% to 39.30% after Stage 1, while purely abstract reasoning falls from 56.58% to 48.47%.Table 8 reports method-score distributions before and after Stage 1.Source: Table 8 · p. 15, F Additional Analysis · p. 15
  • The base model relies on abstract theorem application without checking the continuity-from-above precondition or providing a concrete example.The excerpt identifies the missing precondition as m(E_1) < ∞.Source: F Additional Analysis · p. 15
  • Stage 1 learns to construct examples, but its example confirms the theorem rather than testing its boundary.The chosen sequence satisfies m(E_1) < ∞, so it does not challenge the theorem’s precondition.Source: F Additional Analysis · p. 15
  • Stage 2 further enables the model to identify theorem preconditions and construct targeted counterexamples, leading to correct reasoning.The analysis contrasts this with examples that only confirm existing theorems.Source: F Additional Analysis · p. 15

G Detailed Rubric Criteria

H Additional Experiments and Reproducibility Details · H.1 Evaluation on a Recent Backbone

This subsection evaluates INSPIRE’s full two-stage pipeline on the recent Qwen3-4B-Instruct-2507 backbone using the same training and evaluation pipeline as the main experiments.

H.1 Evaluation on a Recent Backbone

  • INSPIRE’s full two-stage pipeline is evaluated on Qwen3-4B-Instruct-2507 using the main experiments’ training and evaluation pipeline.Table 10 is identified as the evaluation on this recent backbone, but the supplied passage does not include its cell values.Source: §H.1 · p. 16, Table 10 · p. 16

H.2 Correctness-Only DPO Sensitivity

Correctness-only DPO remains close to the SFT baseline across β values and consistently reduces example usage, suggesting its limited effectiveness is not specific to one regularization setting.

H.2 Correctness-Only DPO Sensitivity

  • The baseline uses 10,539 preference pairs based solely on final-answer correctness, with LoRA training for three epochs at a 5 × 10−6 learning rate while β varies.It is trained from the same SFT checkpoint and candidate pool as the main pipeline, with all other settings fixed.Source: §H.2 · p. 16
  • Correctness-only DPO stays close to SFT across all tested β values and consistently reduces example usage, suggesting its limited effectiveness is not tied to one regularization setting.Table 11 reports the β sensitivity of this baseline on Qwen2.5-Math-7B-Instruct.Source: Table 11 · p. 16, §H.2 · p. 16

Related papers

Loading 2608.27501v1…