PaperlayerView exact revision on arXiv

Source-linked AI summary

INSPIRE: An Internalize-Then-Improve Approach for Example-Driven Mathematical Reasoning

Shuai Wang, Jiayi Kuang, Yinghui Li, Haojing Huang, Xinnian Liang, Ying Shen, Liang Lin

arXiv:2608.27501v1cs.CL

TL;DR

LLM mathematical reasoning methods largely optimize final-answer correctness, while example-based reasoning remains underdeveloped and difficult to train because models lack strong candidate-generation ability and acquire the capability progressively. INSPIRE combines Reference-Guided Student Internalization with stage-wise rubric preference training, producing consistent improvements across model scales and families while maintaining or improving out-of-distribution performance.

  • Problem

    Existing methods mainly optimize final-answer correctness, leaving example-based reasoning underdeveloped and raising whether models internalize concepts or rely on solution patterns.

  • Method

    INSPIRE uses Reference-Guided Student Internalization to generate preference candidates under the policy distribution, followed by method-oriented and correctness-oriented preference-training stages.

  • Results

    Experiments across model scales and families show consistent improvements, with maintained or improved performance on out-of-distribution benchmarks.

  • Takeaways & Limitations

    Decomposing capability acquisition into strategy adoption and correctness refinement is more effective than joint optimization for example-driven mathematical reasoning.

  • Takeaways & Limitations

    The study uses 1,275 BrokenMath problems from limited domains and validates models only up to 7B parameters, leaving broader data and larger-scale generalization unexplored.

Abstract

from arXiv · show

Mathematical reasoning has seen rapid progress in large language models (LLMs), yet existing methods optimize predominantly for final-answer correctness, raising the question whether models truly internalize mathematical concepts or merely memorize solution patterns. In human mathematics education, example-based reasoning such as constructing counterexamples to test theorem boundaries reflects deep conceptual understanding, but remains underdeveloped in current LLMs. Enhancing this capability through preference optimization presents two key challenges: (1) the model's limited example-based reasoning ability makes constructing effective preference pairs inherently difficult; and (2) capability acquisition is progressive, as the model must first learn to adopt this strategy before learning to apply it correctly. Therefore we propose INSPIRE, an Internalize-Then-Improve approach combining Reference-Guided Student Internalization (RGSI), which produces high-quality preference candidates under the policy model's own distribution, with a stage-wise rubric preference training strategy that decomposes learning into method-oriented and correctness-oriented stages. Experiments across multiple model scales and families demonstrate consistent improvements, even surpassing larger open-source models, while evaluations on out-of-distribution benchmarks confirm no degradation in general mathematical reasoning ability.

1 Introduction

Existing LLM methods emphasize final-answer correctness, while example-based reasoning remains underdeveloped despite reflecting conceptual understanding. INSPIRE addresses limited data quality and progressive capability acquisition through internalization and staged preference training.

  • Most existing methods optimize final-answer correctness, leaving open whether models internalize mathematical concepts or rely on surface-level solution patterns.
  • Example-based reasoning, such as constructing concrete examples to test theorem boundaries, reflects deep conceptual understanding but remains underdeveloped in current LLMs.
  • Preference optimization is difficult because self-sampled candidates are weak at example-based reasoning, while external references can introduce distribution shift.
  • INSPIRE combines Reference-Guided Student Internalization with stage-wise preference training that separates method-oriented learning from correctness refinement.

2 Related Work

Prior mathematical-reasoning work primarily uses supervised data expansion or reinforcement-learning post-training. These approaches generally optimize final-answer correctness, motivating INSPIRE’s focus on example-driven reasoning.

  • Existing approaches mainly strengthen mathematical problem solving through supervised fine-tuning and data augmentation.
  • Supervised methods expand training data through instruction synthesis, question rephrasing, and corpus expansion.
  • A second line of work uses reinforcement learning to elicit deeper reasoning, including self-verification and reflection behaviors.
  • Despite these advances, existing methods generally optimize final-answer correctness and pay limited attention to deeper reasoning strategies.

3 Methodology

INSPIRE constructs on-policy preference candidates with reference guidance and trains them in stages, first acquiring example-based reasoning and then refining correctness.

  • 3 Methodology: INSPIRE combines reference-guided candidate generation with multi-dimensional rubric evaluation to build preference data under the policy model’s distribution.Reference guidance improves method and reasoning quality while keeping generated responses on-policy.
  • 3.2 Stage-wise Rubric Preference Training: Stage-wise training separates method acquisition from correctness refinement because jointly optimizing both is suboptimal for the base model.M-DPO precedes C-DPO in the training pipeline.
  • Stage 1: Method-Oriented Preference Learning: M-DPO selects pairs using method separation and reasoning-quality gates, prioritizing preferred responses with higher reasoning scores.Both preferred and dispreferred responses must meet reasoning-quality constraints, while the preferred response must use the target method more strongly.
  • Stage 2: Correctness-Oriented Preference Refinement: The resulting model π1 is used to resample and score candidates before correctness-oriented preference refinement in Stage 2.Stage 2 shifts optimization toward answer correctness while using the same evaluation rubric.
  • Stage 2: Correctness-Oriented Preference Refinement: C-DPO compares correct with incorrect responses, also rewarding more rigorous reasoning among multiple correct candidates.Correctness pairs require c(yw)=1 and c(yl)=0, while quality pairs can compare correct responses using reasoning-score differences.
  • Stage 2: Correctness-Oriented Preference Refinement: C-DPO augments correctness-oriented pairs with some Stage 1 pairs to prevent forgetting the acquired reasoning strategy.π1 serves as both initialization and reference policy for the second stage.

1: // Data Preparation

The data-preparation procedure scores responses with a rubric and constructs stage-specific preference pairs for method acquisition followed by correctness refinement.

  • 1: // Data Preparation: Responses are scored with J using method, reasoning, and correctness dimensions.The rubric is applied during both data-preparation stages.
  • 1: // Data Preparation: Stage 1 constructs D1 pairs with a method-score gap and reasoning-quality thresholds, prioritizing pairs with higher preferred-response reasoning scores.The constraints require m(yw)−m(yl)≥1 and both reasoning scores to be at least 1.

4 Experiments

Experiments evaluate INSPIRE across model scales and families, benchmark generalization, and ablations of its training strategy and preference-data construction. Results support progressive method-then-correctness training, RGSI-based candidates, and preserved general mathematical reasoning.

  • Experimental setup: The study evaluates Qwen2.5-Math models at 1.5B and 7B scales plus Llama-3.1-8B-Instruct, using CounterMath and out-of-distribution benchmarks.The implementation uses Qwen2.5-Math-7B-Instruct as the primary base model and applies the same pipeline to smaller and cross-family models.
  • Main results: Stage 1 raises the 7B model’s Examples rate from 77.63% to 81.33% and F1 from 39.06 to 43.05, while Stage 2 further increases them to 84.79% and 45.91.The same progressive pattern holds for the 1.5B model and Llama-3.1-8B-Instruct, with F1 gains of +2.66 and +7.29, respectively.
  • Main results: The fully trained 7B model surpasses Qwen2.5-Math-72B-Instruct at F1 41.89 and approaches Qwen3-32B at F1 45.93.The 1.5B fully trained model reaches F1 39.02, comparable to the 7B base model’s F1 38.67.
  • Generalization: Out-of-distribution evaluation shows the final model consistently maintains or improves performance across GSM8K, MATH500, AIME 2024, GAOKAOmathQA, and MMLU-collegeMath.Qwen2.5-Math-7B gains +6.67 on AIME 2024, while unchanged scores for the two smaller configurations are attributed to the benchmark’s 30-problem granularity and difficulty.
  • Training strategy: Ablations show that method-oriented training is the critical first step: M-DPO alone reaches F1 43.05 versus 40.27 for C-DPO alone, while the sequential pipeline reaches 45.91.Mixed DPO underperforms Stage 1 alone, and chosen-response SFT reaches F1 41.75, below the full pipeline.
  • Preference construction: RGSI provides more useful preference signals than self-sampling or teacher rewriting by maintaining moderate margins and gradual accuracy growth during DPO.Self-sampling produces insufficient quality variation, whereas teacher rewriting creates distribution shift and trivially separable pairs.
  • Behavioral analysis: Training improves substantive example usage from 30.74% to 39.30% and reduces purely abstract reasoning from 56.58% to 48.47% after Stage 1.Cross-judge evaluation with GPT-4o preserves progressive improvement across stages despite shifts in absolute score calibration.

5 Conclusion

INSPIRE enhances example-driven mathematical reasoning through reference-guided internalization and stage-wise preference training. The paper reports consistent improvements while identifying limitations in training-domain coverage, model scale, judging, and run design.

  • 5 Conclusion: INSPIRE combines Reference-Guided Student Internalization with stage-wise rubric preference training for example-driven mathematical reasoning.RGSI regenerates reference solutions in the model’s own expression style, while training first encourages example-based reasoning and then refines correctness.
  • 5 Conclusion: The framework decomposes capability acquisition into strategy adoption and correctness refinement rather than optimizing both objectives jointly.This ordering addresses the progressive nature of learning example-based reasoning described in the paper.
  • Limitations: The training data contains 1,275 BrokenMath problems covering a limited set of mathematical domains.Generalization to larger-scale data and broader areas such as number theory and combinatorics remains unexplored.
  • Limitations: Validation is limited to models up to 7B parameters, leaving effectiveness at 32B or 72B scales unexamined.The authors also note that data construction and evaluation rely on DeepSeek-V3.2 as judge, although GPT-4o cross-validation showed consistent trends.
  • Limitations: Reported results use single runs with fixed random seeds, while consistent improvements across three model configurations suggest robustness.The paper identifies judge-model bias as another limitation and reports mitigation through GPT-4o cross-validation.

E Training and Details

Evaluation uses CounterMath metrics for correctness, example use, and alignment, while training applies fixed SFT and DPO settings across model scales.

  • Evaluation Metrics: Macro-F1 measures true/false judgment accuracy with macro averaging, while Examples measures how often models use examples during reasoning.Strict Align measures whether provided examples are fully consistent with the reference, and Loose Align measures a broader alignment condition.
  • Hyperparameters: SFT uses a 1 × 10−5 learning rate for 3 epochs, while DPO uses 5 × 10−6, β = 0.2, and 3 epochs.The same hyperparameters are used for both model scales.

F Additional Analysis

Additional analyses examine method-score distributions, a measure-theory case study, and the detailed rubric used to score responses.

  • Method Score Distribution: Method-score distributions are computed from 16 responses per problem for each of 1,275 problems sampled from π0 and π1.After scoring and parsing, 20,245 π0 responses and 20,171 π1 responses remained valid.
  • Method Score Distribution: Table 8 reports the distribution of self-sampled response method scores before and after Stage 1.The comparison isolates changes associated with the first training stage.
  • Case Study: One analyzed response applies continuity from above without checking the finite-measure precondition and gives no concrete example.The response therefore relies entirely on abstract theorem application for the stated judgment.
  • Case Study: A measure-theory case study compares reasoning across training stages on a CounterMath problem.Figure 4 is intended to show how reasoning changes through the pipeline.
  • Evaluation Rubric: Table 9 enumerates the scoring criteria and score levels for each evaluation dimension.The rubric covers method, reasoning, and correctness-related evaluation structure described in the paper.

H.1 Evaluation on a Recent Backbone

The full two-stage INSPIRE pipeline is also evaluated on the Qwen3-4B-Instruct-2507 backbone using the same training and evaluation procedure.

  • H.1 Evaluation on a Recent Backbone: INSPIRE is applied to Qwen3-4B-Instruct-2507 following the same training and evaluation pipeline as the main experiments.Table 10 reports the evaluation results, with bold formatting indicating the best result.

H.2 Correctness-Only DPO Sensitivity

The correctness-only DPO baseline is evaluated across β values using preference pairs based solely on final-answer correctness. Its performance remains close to SFT while consistently reducing example usage, indicating limited effectiveness across regularization settings.

  • 10,539 preference pairs train the correctness-only DPO baseline using final-answer correctness alone.The baseline shares the main pipeline’s SFT checkpoint and candidate pool.
  • Table 11 measures the baseline’s sensitivity to β on Qwen2.5-Math-7B-Instruct, with bold marking the best result.
  • Correctness-only DPO remains close to the SFT baseline across all β values and consistently reduces example usage.The result suggests its limited effectiveness is not specific to a particular regularization setting.
Loading 2608.27501v1…