PaperlayerView exact revision on arXiv

Source-linked AI summary

Mathematical Transfer in LLMs Follows Reasoning Approach More Than Topic

Sajad Goudarzi, Samaneh Zamanifard, Seyed Amin Seyed Haeri, Moloud Nasiri, Hamed Rahimian

arXiv:2610.00331v1cs.AI

TL;DR

The paper asks whether mathematical training data transfer better when they share a target’s topic or reasoning approach. It compares these relations through counterbalanced fine-tuning designs and finds approach-matched supervision wins across both evaluated designs. The results support reasoning approach as a more effective matching criterion than topic within the tested mathematical settings.

  • Problem

    The paper asks whether mathematical fine-tuning transfers more effectively from sources sharing the target’s topic or its reasoning approach.

    Source: §1 · pp. 1–2

  • Method

    Two counterbalanced 2 × 2 designs rotate each topic–approach cell through the target role and compare same-approach sources with same-topic sources.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    40/40 seed-pooled model–target comparisons favor same-approach supervision across two designs, with mean advantages of 10.8 and 14.3 percentage points.

    Source: §8 · p. 10

  • Takeaways & Limitations

    Reasoning approach is a more effective matching criterion than topic for transfer-oriented mathematical supervision in the evaluated settings.

    Source: §8 · p. 10

    Target-wise contrasts cannot distinguish source quality from a genuine target–approach interaction, so the pattern does not show that invariant reasoning is intrinsically more transferable.

    Source: §5.3 · p. 7

Abstract

from arXiv · show

When selecting mathematical training data for LLMs, a natural organizing principle is topic: probability examples for probability targets. An alternative is reasoning approach: worked solutions that share a solution method with the target, even when the mathematical domain differs. We ask which relation produces greater transfer after fine-tuning. We evaluate two counterbalanced $2\times2$ designs: probability and combinatorics crossed with invariant reasoning and double counting (2,000 problems), and number theory and geometry crossed with complement and pigeonhole reasoning (800 problems). In each design, every cell serves as the held-out target in turn: same-approach (SA) sources share the target's method but change the topic, while same-topic (ST) sources share the topic but change the method. Every source appears once in each role, so additive source-quality effects cancel from the equally weighted aggregate contrast. Across five base models and three training seeds per design, SA outperforms ST in all 40 seed-pooled model--target comparisons. Model-level advantages range from 8.2 to 16.2 percentage points in the primary design (mean: 10.8) and from 12.0 to 16.0 in the second design (mean: 14.3); all ten model-level 95% confidence intervals exclude zero. In both designs, ST sources are more similar to targets under embedding and lexical measures, so the SA advantage runs opposite to the measured ordering of statement-level resemblance. These findings identify reasoning approach as a more effective matching criterion than topic for mathematical transfer across the evaluated topic--approach combinations.

1 INTRODUCTION

The paper tests whether mathematical fine-tuning transfers better when source examples share the target’s topic or its reasoning approach. Across two counterbalanced designs, approach-matched sources outperform topic-matched sources, even though topic-matched problems are more similar to targets under measured similarity metrics.

1 INTRODUCTION

  • The introduction frames topic and reasoning approach as competing ways to select mathematical fine-tuning data and asks which produces greater transfer.Approach examples include invariant reasoning and double counting, which can apply across mathematical domains.Source: §1 · pp. 1–2
  • The rotational design compares sources that share the target’s approach but change topic with sources that share its topic but change approach.Each of the four topic–approach cells serves as the held-out target in turn.Source: Figure 1 · p. 2, §1 · p. 2
  • In the primary design, same-approach sources outperform same-topic sources in all 20 model–target comparisons, averaging 10.8 percentage points.Same-topic problems are more similar to targets under both embedding and lexical measures, yet transfer less effectively.Source: §1 · p. 2
  • The ordering replicates across 20 model–target contrasts in a second design using number theory, geometry, complement reasoning, and pigeonhole reasoning.This design extends the finding beyond the primary topic and approach choices.Source: §1 · p. 2
  • The paper identifies reasoning approach as a more effective matching criterion than topic in the evaluated mathematical setting.Its contribution is a controlled, counterbalanced comparison across five models, three seeds, and four target rotations.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work motivates comparing structural reasoning with surface topic, while leaving open whether skill-sensitive transfer persists under fine-tuning. The paper situates its controlled comparison alongside work on skills, mathematical supervision, benchmarks, and evaluation criteria.

2 RELATED WORK

  • Human problem-solving and analogy research motivate the hypothesis that shared structural principles can support transfer across surface-dissimilar problems.The paper cautions that human categorization and learning through LLM gradient updates are different processes.Source: §2 · p. 3
  • Didolkar et al. (2024) find skill-matched exemplars improve in-context prompting, while this study tests fine-tuning with a counterbalanced source rotation.Guo et al. (2024) report that skill-sensitive prompting effects need not persist in all instruction-tuning conditions.Source: §2 · p. 3
  • Skill-guided training-data methods use taxonomies to select data, whereas this study compares topic and reasoning approach as matching axes.Examples include Skill-it (Chen et al., 2023), LESS (Xia et al., 2024), STAT, and skill-aware reasoning distillation (Zhang et al., 2026).Source: §2 · p. 3
  • Existing mathematical benchmarks test forms of generalization such as compositional splits, template robustness, and exploratory or transformative generalization.This evaluation instead holds out a cell from a shared 2 × 2 construction and compares transfer from the other cells.Source: §2 · p. 3
  • The paper uses final-answer accuracy as its transfer endpoint and treats lexical and embedding similarity as descriptive diagnostics rather than measures of strategy.The endpoint accepts any valid solution method, not only the reference approach.Source: §2 · p. 3

3 A ROTATIONAL TEST OF MATHEMATICAL TRANSFER

The study crosses mathematical topic with solution approach, rotates each cell into the target role, and compares same-approach with same-topic sources. Its estimand averages target-level accuracy contrasts, while rotation cancels additive source and target effects but not source–target interactions.

3.1 CELLS AND SOURCE–TARGET RELATIONS

  • The four cells cross probability or combinatorics with invariant reasoning or double counting, and each target is compared with sources sharing its approach or its topic.For a target, the SA source changes topic and the ST source changes approach; neither comparison source is trained on the target cell.Source: §3.1 · p. 4

3.2 OUTCOME AND ESTIMAND

  • The outcome is final-answer accuracy, and the model-level estimand equally averages target-specific SA–ST contrasts across four targets and three seeds.A positive contrast does not imply that either condition improves over the unadapted model; the five-model mean is not a population-level estimate.Source: §3.2 · p. 4

3.3 WHAT ROTATION CONTROLS

  • Rotation cancels additive source effects in the aggregate contrast, but it does not eliminate source–target interactions such as statement-level resemblance.The design controls for fixed source identity, not every leftover difference.Source: §3.3 · p. 4

4 DATASET AND EXPERIMENTAL PROTOCOL

The dataset comprises 2,000 problems across four topic–approach cells. Problems were generated conditioned on topic and approach.

4 DATASET AND EXPERIMENTAL PROTOCOL

  • The dataset contains 2,000 problems, with 500 in each cell, generated using gpt-oss-120b at high reasoning effort conditioned on topic and approach.The passage begins describing accompanying materials but does not specify them in the supplied text.Source: §4.1 · p. 4

DATASET CONSTRUCTION

The study validates problems for a counterbalanced SA–ST comparison across target rotations, then evaluates accuracy with confidence intervals conditional on trained checkpoints.

DATASET CONSTRUCTION

  • Problems were retained when unconstrained re-solutions and three judges agreed on approach, then answer-checked and corrected through final auditing.Auditing found 83 of 2,000 entries inconsistent across solvers; they were replaced, and all 50 manually reviewed problems were correct.Source: DATASET CONSTRUCTION · p. 5, Figure 2 · p. 5

4.2 MODELS, TRAINING, AND EVALUATION

  • Five base or pretrained checkpoints were adapted with rank-16 LoRA on 450 source examples per run, using three seeds and evaluating on 500 held-out target items.The protocol comprised 60 source-training jobs; final-answer accuracy was the outcome, without requiring reproduction of reference wording or approach.Source: §4.2 · p. 5
  • +10.8 percentage points is the equally weighted mean model-level SA–ST advantage; all five models favor SA and every reported 95% CI lies above zero.The figure averages four held-out targets and seeds 7, 17, and 27 for each point.Source: Figure 3 · p. 6

4.3 UNCERTAINTY AND REPORTING

  • The 95% confidence intervals use paired, cell-stratified bootstrap resampling and quantify finite evaluation-set uncertainty conditional on trained checkpoints, not training-seed variability.Seed-to-seed variation is reported separately through per-seed SA–ST contrasts.Source: §4.3 · p. 6

5 RESULTS

Across both designs, same-approach training outperforms same-topic training, though target-level contrasts vary. The advantage persists across models, topics, approaches, and training budgets, with interpretation bounded by the design’s scope and subgroup caveats.

5.1 SA EXCEEDS ST ACROSS ALL FIVE MODELS

  • +10.8 percentage points was the equally weighted mean model-level SA–ST advantage in the primary design, and all five model-level 95% confidence intervals excluded zero.All 20 seed-pooled model–target contrasts favored SA; every reported target-level interval excluded zero.Source: §5.1 · p. 6

5.2 THE ADVANTAGE IS NOT ONLY LESS NEGATIVE TRANSFER

  • Across Llama, DeepSeek, Mistral, and Gemma, both SA and ST improved accuracy over base in all four targets, while SA gains averaged roughly 11–17 points versus 2–7 for ST.Qwen showed a different regime: SA averaged +9.9 points over base, while ST averaged −6.3 points and had negative transfer in every target. Low base accuracy depends on prompting format, generation strategy, and answer extraction; the finding concerns adaptation, not learning from zero knowledge.Source: §5.2 · p. 6

5.3 TARGET-WISE EFFECTS ARE CONSISTENT BUT HETEROGENEOUS

  • 12.0, 5.6, 19.5, and 6.2 points were the average SA–ST gaps for probability/invariant, probability/double counting, combinatorics/invariant, and combinatorics/double counting, respectively.Combinatorics/invariant was strongest for every model, while both double-counting targets retained positive contrasts. Target-wise contrasts cannot distinguish source quality from a target–approach interaction; the study does not infer intrinsic transferability of invariant reasoning.Source: §5.3 · p. 7, Figure 5 · p. 7

5.4 THE ORDERING REPLICATES WITH DIFFERENT TOPICS AND APPROACHES

  • +14.3 percentage points was the mean model-level SA–ST advantage in the 800-problem number theory/geometry design, with all five model-level 95% confidence intervals excluding zero.All 20 seed-pooled model–target point estimates favored SA; ST sources were more similar to targets under both embedding and TF–IDF measures.Source: §5.4 · pp. 7–8

5.5 THE ORDERING PERSISTS UNDER FULL FINE-TUNING

  • +11.1 points was the mean SA–ST advantage under full fine-tuning across four models, positive in all 16 model–target comparisons.Mistral did not converge under full fine-tuning and was excluded from these summaries; the matched LoRA mean was +12.0 points.Source: §5.5 · p. 8

5.6 THE ADVANTAGE APPEARS AT SMALLER TRAINING BUDGETS

  • +7.5, +9.7, and +11.4 points were the suite-mean SA–ST advantages at 150, 300, and 450 examples, respectively.All five model aggregates favored SA at each budget. SA–base grew from +9.7 to +13.3 points, while ST–base remained relatively flat at +2.2, +1.3, and +1.9; this single-seed ablation supports directional rather than precise effect-size conclusions.Source: §5.6 · p. 8

6 STATEMENT SIMILARITY AND THE TRANSFER ORDERING

The statement-similarity analysis compares source–target resemblance using embedding and lexical measures, including all-pairs and nearest-neighbor comparisons. Same-topic sources are more similar by these measures, although the transfer ordering favors same-approach sources.

6 STATEMENT SIMILARITY AND THE TRANSFER ORDERING

  • The comparison uses Qwen3-Embedding and TF–IDF to measure source–target statement similarity through all-pairs averages and nearest-neighbor comparisons.The nearest-neighbor measure is directional.Source: §6 · p. 8
  • Embedding similarity favors ST over SA for all pairs (0.458 versus 0.437) and nearest neighbors (0.739 versus 0.705); TF–IDF nearest-neighbor similarity also favors ST (0.260 versus 0.232).For probability/invariant, the nearest out-of-cell statement is in the ST cell for 63.0% of items and in the SA cell for 35.4%.Source: §6 · pp. 8–9, Figure 7 · p. 9
  • For all five models, the SA–ST ordering holds at each of 150, 300, and 450 source-training examples, and the suite-mean change from base widens with data.Epochs are fixed, so optimizer steps grow with the data.Source: Figure 6 · p. 9

7 DISCUSSION AND LIMITATIONS

Controlled comparisons across two designs find that approach-matched supervision outperforms topic-matched supervision. The authors limit the conclusion to the tested settings and note that final-answer accuracy does not identify the internal transfer mechanism.

7 DISCUSSION AND LIMITATIONS

  • Approach-matched supervision consistently outperforms topic-matched supervision in cross-cell mathematical transfer across five models, three seeds, and all target rotations.Across the two designs, same-topic sources are more similar to targets under lexical and embedding measures, yet transfer less effectively.Source: §7 · p. 9
  • The controlled designs vary topic sharing and approach sharing independently while keeping the comparisons mathematically meaningful.The repeated ordering spans four topics and four approaches.Source: §7 · p. 9
  • The study is not an exhaustive survey of mathematics; broader domains and solution structures would provide additional tests of generality.The synthetic construction does not eliminate every possible source–target interaction, and final-answer accuracy does not identify the internal transfer mechanism.Source: §7 · p. 9

8 CONCLUSION

Across two independently constructed designs, approach-matched supervision beats topic-matched supervision in every seed-pooled model–target comparison. Same-topic sources are more similar by embedding and lexical measures, but the findings support reasoning approach as a stronger organizing principle for transfer-oriented mathematical supervision.

8 CONCLUSION

  • Across two designs, SA outperforms ST in all 40 seed-pooled model–target comparisons, averaging 10.8 points in the first design and 14.3 in the second.The designs vary both mathematical topics and reasoning approaches.Source: §8 · p. 10
  • Across both datasets, same-topic sources are more similar to targets under embedding and lexical measures, yet transfer less effectively.In these experiments, mathematical transfer follows shared reasoning approach more consistently than shared topic.Source: §8 · p. 10
  • The results provide evidence that reasoning approach is a stronger organizing principle than topic for selecting transfer-oriented mathematical supervision in LLMs.The authors suggest that solution-method structure should play a central role in organizing and selecting mathematical fine-tuning data.Source: §8 · p. 10

AI USE STATEMENT

The paper reports AI assistance across problem generation, solution and label checks, and manuscript preparation. The results appendices specify the scope of their tables and describe the second-design replication.

AI USE STATEMENT

  • AI systems supported mathematical problem and solution generation, approach-label judging, answer auditing, manuscript rewriting, literature research, reference checking, and preparation of LaTeX and figure code.Source: AI USE STATEMENT · p. 11

A COMPLETE RESULTS

  • The results tables cover model-level and per-seed SA–ST contrasts, absolute accuracies, and target-level accuracy differences; the reported seeds are 7, 17, and 27.Point estimates and differences were rounded separately, so displayed subtraction may differ by 0.1.Source: Table 1 · p. 13, Table 2 · p. 13, Table 3 · p. 13

B SECOND-DESIGN REPLICATION

  • The second design uses 800 problems across number theory and geometry with complement and pigeonhole reasoning, following the primary dataset’s construction framework.Its tables report model-level, seed-level, and target-level contrasts; all 15 model–seed aggregates and 20 seed-pooled model–target estimates favor SA.Source: B SECOND-DESIGN REPLICATION · p. 15, Table 5 · p. 15, Table 6 · p. 15

C FULL FINE-TUNING ABLATION

Excluding Mistral, full fine-tuning yields a mean SA–ST advantage of +11.1 points, compared with +12.0 for matched LoRA runs. The section also specifies training-set sizes, the primary protocol, statistical uncertainty, and similarity diagnostics.

C FULL FINE-TUNING ABLATION

D TRAINING-SET SIZE ABLATION

  • Training-set sizes are nested at 150, 300, and 450 examples per source cell, with 50 validation items and all 500 target items evaluated.Each size trains for three epochs, so the step budget grows from about 30 to 57 to 87 steps; Table 10 reports accuracy differences averaged over four target cells.Source: D TRAINING-SET SIZE ABLATION · p. 18, Table 10 · p. 18

E CHECKPOINTS AND PROTOCOL

  • The primary protocol uses 450 training and 50 validation problems per source cell, with 500 held-out target problems and seeds 7, 17, and 27.Invariant reasoning uses a preserved quantity or relation; double counting counts the same collection in two ways, and all three approach judges had to agree for retention.Source: E CHECKPOINTS AND PROTOCOL · p. 19

F STATISTICAL ANALYSIS

  • The 95% intervals quantify finite evaluation-set uncertainty conditional on trained checkpoints, not training-seed variability; reported intervals are pointwise and unadjusted for multiple comparisons.Individual seed-level SA–ST contrasts are reported separately to characterize variation across training seeds.Source: F STATISTICAL ANALYSIS · p. 19

G SIMILARITY DIAGNOSTICS

  • The similarity diagnostics compare statement similarity by relation and report how often a nearest out-of-cell neighbor falls in the SA or ST cell.Table 12 distinguishes centered embeddings and notes that its metrics are not interchangeable; Table 13 reports the leftover share matching neither relation.Source: Table 12 · p. 20, Table 13 · p. 20

Related papers

Loading 2610.00331v1…