PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning

Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, Minxin Du, Radha Poovendran, Graham Neubig, Xiang Yue

arXiv:2507.00432v2cs.AIcs.CL

TL;DR

The paper asks whether strong math reasoning transfers to broader LLM capabilities across reasoning and non-reasoning tasks. Evaluating open-weight models and controlled Qwen3-14B variants, it finds that RL preserves transfer while SFT often causes negative transfer and representation and output drift.

  • Problem

    The paper examines whether improved math reasoning transfers to scientific QA, coding, agent planning, and instruction-following tasks.

  • Method

    The study evaluates over 20 open-weight reasoning models and uses controlled Qwen3-14B math-only tuning experiments with latent-space PCA, token-distribution analysis, and RL ablations.

  • Results

    Most math-successful models fail to transfer broadly, while RL-tuned models preserve positive transfer and SFT-tuned models often show negative transfer on non-math tasks.

  • Takeaways & Limitations

    Fine-tuning paradigm is decisive for balancing improved math reasoning with retained performance on reasoning and non-reasoning tasks.

  • Takeaways & Limitations

    The authors identify future work on hybrid recipes, adaptive regularization, and transferability evaluation in multimodal, long-context, and interactive settings.

Abstract

from arXiv · show

Math reasoning has become the poster child of progress in large language models (LLMs), with new models rapidly surpassing human-level performance on benchmarks like MATH and AIME. But as math leaderboards improve week by week, it is worth asking: do these gains reflect broader problem-solving ability or just narrow overfitting? To answer this question, we evaluate over 20 open-weight reasoning-tuned models across a broad suite of tasks, including math, scientific QA, agent planning, coding, and standard instruction-following. We surprisingly find that most models that succeed in math fail to transfer their gains to other domains. To rigorously study this phenomenon, we conduct controlled experiments on Qwen3-14B models using math-only data but different tuning methods. We find that reinforcement learning (RL)-tuned models generalize well across domains, while supervised fine-tuning (SFT)-tuned models often forget general capabilities. Latent-space representation and token-space distribution shift analyses reveal that SFT induces substantial representation and output drift, while RL preserves general-domain structure. Our results suggest a need to rethink standard post-training recipes, particularly the reliance on SFT-distilled data for advancing reasoning models.

1 INTRODUCTION

The paper asks whether rapidly improving mathematical reasoning transfers to broader LLM capabilities. Across model evaluations and controlled Qwen3-14B experiments, transferability depends strongly on the fine-tuning paradigm, with RL outperforming SFT outside mathematics.

  • The study evaluates whether math gains transfer to scientific QA, coding, agent planning, logical deduction, conversational QA, and instruction following.
  • Over 20 open-weight reasoning models are assessed across reasoning and non-reasoning tasks using the proposed Transferability Index.Positive values indicate successful transfer, while negative values indicate performance degradation.
  • Some reasoning models fail to transfer improved mathematical capabilities to other domains, whereas others succeed.
  • Across model families and sizes, RL-tuned models generalize more strongly to non-math tasks than SFT-tuned models, which often show catastrophic forgetting.
  • A controlled Qwen3-14B study trains SFT and RL models on the same math-derived data, using teacher-generated correct responses for SFT and answer correctness as the RL reward.
  • Latent-space PCA and token-distribution KL analyses show substantial representation and output drift under SFT, while RL better preserves internal feature geometry and token-distribution stability.

2 PHENOMENA: PERFORMANCE DISCREPANCIES OF REASONING MODELS

The paper measures transferability across math, other-reasoning, and non-reasoning benchmarks, then compares RL and SFT under controlled data conditions. RL generally preserves or improves broader capabilities, whereas SFT often produces negative transfer on non-reasoning tasks.

  • The evaluation covers math reasoning, other reasoning, and non-reasoning benchmark groups across more than 20 off-the-shelf reasoning models.
  • Transferability Index normalizes benchmark gains within each group, aggregates robust difficulty-weighted gains, and divides each group’s improvement by math improvement.
  • Positive TI values indicate positive transfer, while higher values indicate stronger and more consistent transfer of math capabilities to other domains.
  • RL-tuned models consistently achieve higher indices on other-reasoning and non-reasoning tasks, whereas SFT models often yield negative TInon.
  • Using identical data conditions, the controlled study finds RL-tuned models generally outperform SFT models on both reasoning and non-reasoning tasks.
  • 55.7% on AIME24, 87.8% on MATH500, and 33.8% on OlympiadBench are achieved by UniReason-Qwen3-14B(RL), exceeding corresponding SFT-based models.
  • 17.1% on LiveCodeBench2 and 1.8% on GPQA are the RL model’s gains over SFT on other-reasoning tasks.
  • In non-reasoning evaluations, SFT models stagnate or decline, while RL recovers and exceeds the base model on nearly all benchmarks.

3 LATENT REPRESENTATION SHIFTS: INSIGHTS FROM PCA ANALYSIS

PCA shift analysis compares how SFT and RL alter hidden-state geometry relative to the base model. RL produces the smallest shifts across task types, consistent with more stable latent representations and better preservation of general-domain capabilities.

  • 3.2 INVESTIGATING LATENT SPACE SHIFT: SFT models often impair general-domain performance despite improving mathematical problem-solving and code generation.The controlled comparisons link this transferability gap to larger changes in internal feature geometry.
  • 3.1 EXPERIMENTAL SETUP: PCA shift is used to distinguish changes in internal feature geometry from parameter changes that may leave the underlying feature space intact.The method is motivated as a way to diagnose representation changes relevant to cross-domain generalization.
  • 3.1 EXPERIMENTAL SETUP: PCA shift measures the Euclidean distance between representation centroids before and after training.The analysis applies PCA to hidden states and aggregates projected coordinates across layers before computing the distance.
  • 3.2 INVESTIGATING LATENT SPACE SHIFT: RL models achieve the lowest PCA shift magnitudes across math, other-reasoning, and non-reasoning tasks.Figure 3 shows minimal and tightly clustered latent shifts for RL compared with SFT models.
  • 3.2 INVESTIGATING LATENT SPACE SHIFT: RL’s smaller latent shifts accompany a broader balance between reasoning performance and general-domain capabilities.The paper connects lower shifts with preserving representations across diverse benchmarks and mitigating catastrophic forgetting.

4 TOKEN DISTRIBUTION SHIFTS: INSIGHTS FROM KL DIVERGENCE AND RANK ANALYSES

Token-level analyses show that SFT changes the model’s output distribution more substantially than RL. RL preserves token rankings and selectively shifts task-relevant tokens, whereas SFT shifts many irrelevant tokens.

  • 4.1 ANALYSIS METHOD: KL divergence measures token-distribution change from the original backbone, while rank shift compares fine-tuned and backbone token rankings.The analyses evaluate distributional changes after fine-tuning across reasoning and non-reasoning tasks.
  • 4.2 RESULT: SFT models have significantly larger KL divergence than RL models across reasoning and non-reasoning tasks.Higher KL divergence indicates greater distribution shift from the backbone model.
  • 4.2 RESULT: 0.372 and 0.283 are the SFT-no-think KL divergences on MATH-500 and IFEval, versus 0.084 and 0.019 for RL.Each pair is reported relative to the backbone model on the corresponding task.
  • 4.2 RESULT: 0.98 positions is the RL model’s average token-rank shift, compared with 10.6 positions for the SFT no-think variant.The smaller RL shift indicates closer preservation of the base token distribution.
  • 4.2 RESULT: RL shifts a small set of task-relevant tokens, while SFT shifts 390 reasoning and 158 non-reasoning tokens, including many irrelevant ones.The SFT shifts often inject reasoning markers into non-reasoning prompts.

5 ABLATION: WHICH COMPONENTS OF RL DRIVE GENERALIZATION?

The ablation study separates RL’s transferability advantage into sampling distribution, credit assignment, negative examples, and KL regularization. On-policy training, selective credit assignment, and negative-example learning improve transfer, while KL has a limited additional effect in on-policy RL.

  • 5.1 CONTRASTING OBJECTIVES: The unified surrogate loss contrasts SFT and RL through sampling distribution, credit assignment, KL regularization, and negative gradients.SFT is off-policy and imitates positive examples, whereas RL samples from the live policy and uses advantage-weighted updates.
  • 5.2 ABLATION RESULTS: On-policy methods outperform off-policy methods across both other-reasoning and non-reasoning evaluation categories.The paper attributes this pattern partly to reduced distribution mismatch and more conservative updates.
  • 5.2 ABLATION RESULTS: Credit assignment and negative examples improve transferability and increase response length during training.Credit assignment focuses updates on task-relevant tokens, while negative gradients reduce reinforcement of wrong or irrelevant patterns.
  • 5.2 ABLATION RESULTS: Under on-policy training, RL outperforms SFT on Math and Other Reasoning tasks while showing similar performance on Non-Reasoning tasks.In off-policy settings, RL achieves higher average scores on both Other Reasoning and Non-Reasoning tasks.
  • 5.2 ABLATION RESULTS: On-policy RL performance remains largely unchanged with or without KL regularization.This suggests KL’s additional effect on transferability is limited when updates already follow the on-policy distribution.

6 CONCLUSION

The fine-tuning paradigm is decisive for transferability: RL improves reasoning while preserving broader capabilities, whereas SFT often causes negative transfer. The authors identify sampling distribution as the most important factor within RL and propose broader future extensions.

  • 6 CONCLUSION: RL-tuned models improve math reasoning while preserving positive transfer to reasoning and non-reasoning tasks, whereas SFT often induces negative transfer.RL remains closer to the backbone, while SFT produces larger shifts, especially on non-reasoning data.
  • 6 CONCLUSION: RL selectively adjusts task-relevant tokens, whereas SFT perturbs many irrelevant ones, aligning token-distribution behavior with their different transferability.
  • 6 CONCLUSION: UniReason-Qwen3-14B-RL trained on 47K math examples achieves the strongest balance of reasoning gains and general-domain retention among compared models.
  • 6 CONCLUSION: Future work targets hybrid recipes, adaptive regularization, and broader transferability evaluation across multimodal, long-context, and interactive settings.

A.1 THE USE OF LARGE LANGUAGE MODELS

The authors used large language models for language editing and partial code validation, while stating that the models did not contribute to the research itself.

  • A.1 THE USE OF LARGE LANGUAGE MODELS: Large language models were used only to improve text readability and validate part of the code for edge cases.

A.2 DETAILED EVIDENCE OF OUR OBSERVED PHENOMENA

The appendix provides the complete Transferability Index evaluation for off-the-shelf models on other reasoning and non-reasoning tasks in Table 5.

  • A.2 DETAILED EVIDENCE OF OUR OBSERVED PHENOMENA: Table 5 contains the complete Transferability Index evaluation for off-the-shelf models on other reasoning and non-reasoning tasks.

A.3.1 POST-TRAINING METHODS

The study compares reinforcement learning and supervised fine-tuning as post-training methods, using different optimization objectives and training setups.

  • A.3.1 POST-TRAINING METHODS: RL fine-tunes Qwen-3-14B-Base with GRPO, using answer correctness as the reward signal.The setup uses the Verl framework, a learning rate of 1 × 10−6, batch size 512, and clipping thresholds between 0.22 and 0.28.
  • A.3.1 POST-TRAINING METHODS: SFT adapts the model by minimizing cross-entropy on teacher-generated chain-of-thought traces.The reported learning rate is 5 × 10−5, with batch size 512 and 1.5 training epochs.
  • A.3.1 POST-TRAINING METHODS: SFT and RL use distinct post-training objectives: supervised learning follows high-quality targets, whereas RL optimizes behavior through scalar rewards.

A.3.2 TRAINING DATASETS

The analyses compare how training paradigm, model size, and model family affect representation stability and transfer, alongside case studies of RL- and SFT-tuned outputs. RL generally preserves broader capabilities more effectively, while SFT produces broader token and representation shifts.

  • Representation-Level Analysis: RL-based fine-tuning produces lower PCA shifts than SFT across math, other-reasoning, and non-reasoning tasks.The results associate RL with more concentrated representation shifts and reduced catastrophic forgetting.
  • Effect of Model Size: Increasing model size from 7B to 32B substantially reduces PCA shifts across math, other-reasoning, and non-reasoning tasks.Larger models show more concentrated and stable latent representations in non-reasoning domains.
  • Effect of Model Family: RL-tuned Llama and Qwen models both exhibit small PCA shifts, but Llama models show greater resilience in general domains.Qwen displays higher drift than Llama on other-reasoning and non-reasoning tasks, motivating model-specific RL strategies.
  • Training Dynamics: RL uses gentler early updates, smaller total parameter movement, and more stable later gradients than SFT.SFT instead shows large initial fluctuations followed by rapid decay and oscillation.
  • Training Dynamics: RL increases response length and variation during training, with exploration and credit assignment supporting adaptation beyond narrow SFT supervision.Negative examples discourage uninformative outputs, while task-relevant tokens receive amplified signals.
  • A.6.1 CASE STUDY AND DETAILS: In the treek–squig case, both RL- and SFT-tuned models derive S = 3T, while the RL response presents a complete concise derivation.The RL output concludes that three treeks equal one squig; the SFT output reaches the same answer through repeated checking.
Loading 2507.00432v2…