PaperlayerView exact revision on arXiv

Source-linked AI summary

Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them

Neel Rajani, Aryo Pradipta Gema, Seraphina Goldfarb-Tarrant, Ivan Titov

arXiv:2507.10616v2cs.LGcs.AIcs.CL

TL;DR

Reasoning-training dynamics and their internal effects remain poorly understood. This study compares GRPO and SFT in closely aligned setups, finding modest targeted gains and smaller out-of-domain declines with GRPO than with SFT.

  • Problem

    The dynamics of reasoning-training algorithms and their internal effects on models are poorly understood.

    Source: §1 · p. 1

  • Method

    The authors compare GRPO and SFT on OLMo-2-1124-7B-Instruct, tuning GRPO first and adapting its hyperparameters to SFT for closely aligned training setups.

    Source: §1 · p. 1, §3 · p. 2

  • Results

    GRPO yields modest gains on maths datasets and small out-of-domain degradations, while SFT considerably exacerbates both trends.

    Source: §3.3 · p. 5, §6 · p. 8

  • Takeaways & Limitations

    The observed trade-off motivates further research into reasoning-training gains and out-of-domain capability costs.

    Source: §3.3 · p. 5

    The interpretation is constrained because the parameters may not have been optimally tuned, so the authors refrain from strong claims.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Training large language models (LLMs) for reasoning via maths and code datasets has become a major new focus in LLM post-training. Two particularly popular approaches are reinforcement learning (RL) and supervised fine-tuning (SFT), but their training dynamics are poorly understood. We present a comparative analysis of RL and SFT on the same maths problems with the same model and similar hyperparameters. We find that RL yields minor in-domain gains on maths and slight degradation on knowledge-intensive benchmarks like MMLU, while both trends are more pronounced in SFT. We also analyse model parameters across checkpoints, observing that both algorithms modify query and key weights the most. Meanwhile, SFT exhibits greater updates and also affects mid-layer MLPs more, leading us to hypothesise that this may have caused the out-of-domain degradation. We therefore investigate whether freezing parts of the model during training can mitigate the reduced performance on knowledge-intensive benchmarks. However, our results are inconclusive, with benefits on GPQA:Diamond and degradation on other benchmarks. Taken together, our observations provide a preliminary indication for why RL amplifies existing capabilities, while SFT replaces old skills with new ones.

1. Introduction

The paper compares GRPO and SFT in a controlled setup, finding a trade-off between in-domain gains and out-of-domain capability retention. It also examines checkpoint and parameter changes, then tests whether freezing model components can mitigate degradation, with inconclusive results.

1. Introduction

  • SFT produces more pronounced in-domain gains and out-of-domain degradation than GRPO, whose gains are modest and out-of-domain capabilities stay more intact.The comparison is presented as a trade-off between the two approaches.Source: §1 · p. 1
  • Across 20 checkpoints, SFT affects models more strongly early in training, while both methods adjust query and key weights most and SFT makes greater updates, especially in middle layers.The authors hypothesize that SFT’s larger mid-layer updates may contribute to its greater degradation on knowledge-intensive benchmarks.Source: §1 · pp. 1–2
  • Freezing MLP matrices showed some in-domain and out-of-domain benefits, but results were inconclusive; training only query and key matrices in SFT left performance unchanged or reduced.Source: §1 · p. 2
  • The findings offer preliminary indications of how GRPO and SFT affect models differently, but require further validation.Source: §1 · p. 2

2. Related works

Prior work examines reasoning-focused training through RL and SFT, including claims about capability gains and concerns about how those gains arise. Mechanistic interpretability offers tools for studying model internals, but cross-checkpoint fine-tuning comparisons remain under-researched.

2.1. Reasoning-focused training

  • Reasoning-focused training commonly applies RL with verifiable rewards to maths and code, while SFT can use synthetic reasoning traces; both approaches have been associated with performance gains.The cited account describes GRPO as the algorithm used by DeepSeek’s R1 and its process for collecting traces for SFT.Source: §2.1 · p. 2
  • Prior work questions whether GRPO creates new abilities: reported “aha moments” may already occur in base models, and longer responses may reflect length and difficulty biases.Source: §2.1 · p. 2
  • Other studies suggest GRPO shifts outputs toward correct completions the base model could already produce, while RL can overfit small benchmarks and blending RL with SFT may mitigate limitations (EssentialAI et al., 2025; Gandhi et al., 2025; Hochlehnert et al., 2025; Yan et al., 2025).Source: §2.1 · p. 2

2.2. Mechanistic interpretability

  • Mechanistic interpretability studies circuits and features; prior findings link mid-layer MLPs to factual associations, while cross-checkpoint comparisons during fine-tuning remain under-researched (Geva et al., 2021; Meng et al., 2023).The passage also cites work on induction heads and pre-training dynamics.Source: §2.2 · p. 2

3. Training models to reason

In a controlled comparison of GRPO and SFT on math questions, SFT trained more stably but showed stronger in-domain gains alongside greater benchmark degradation; GRPO’s effects were milder.

3. Training models to reason

  • The study compared GRPO and SFT using OLMo-2-1124-7B-Instruct on NuminaMath 1.5 questions, with model-sampled GRPO completions, DeepSeek R1 SFT completions, and a shared prompt template.The authors saved 20 intermediate checkpoints for each algorithm.Source: §3 · pp. 2–3

3.1. Reasoning training with GRPO

  • On CN-K12-91k, format rewards rose quickly while accuracy rewards rose slowly and plateaued; GRPO remained costly and unstable, requiring extensive tuning.The authors found the easier K-12 subset more reliable than olympiad-level questions that were too challenging for the model.Source: §3.1 · p. 3, Figure 1 · p. 3

3.2. Reasoning training with SFT

  • SFT was more stable and needed less tuning, with smooth loss curves and rare gradient spikes, but its learning rate differed from GRPO’s, a comparison confound.The reported successful learning rates were around 5e-5 for SFT and 1e-6 for GRPO.Source: §3.2 · pp. 3–4

3.3. Evaluating models after reasoning training

  • GRPO brought modest MATH-500 gains but slight GSM8k loss and less knowledge-benchmark degradation; SFT’s larger math gains and drops intensified, though GPQA-Diamond reversed the knowledge pattern.The authors caution that setup limitations, including potentially untuned parameters, constrain the strength of their interpretation.Source: §3.3 · pp. 4–5

4. Cross-checkpoint analysis of reasoning training

The checkpoint analysis finds that GRPO changes the model more gradually and reinforces existing capabilities, while SFT departs more substantially from the base model.

4. Cross-checkpoint analysis of reasoning training

  • Across checkpoints, GRPO makes fewer integral changes and reinforces existing capabilities, whereas SFT departs more substantially from the base model and makes larger parameter changes.These are preliminary indications of different training dynamics.Source: §4 · p. 5

4.1. KL Divergence throughout training

  • On withheld MATH-500, checkpoint KL against the base rose early and sharply under SFT, but grew gradually to a lower plateau under GRPO.The analysis compares average per-token KL divergence; the authors note that learning-rate differences affect the magnitude of internal changes, while GRPO's update controls may help explain its slower KL growth.Source: §4.1 · p. 5, Figure 4 · p. 5

4.2. Norm of differences throughout training

  • The analysis measures normalized Frobenius differences across attention and MLP matrices, but differing learning rates make these internal comparisons approximate.It excludes query and key norm matrices, embeddings, the language-model head, and layer-norm matrices.Source: §4.2 · p. 6

4.2.1. THE PARAMETER-LEVEL IMPACT OF GRPO

  • GRPO's largest updates concentrate in middle-layer query and key matrices, while value and output matrices change less, possibly shifting attended tokens without changing communicated content.The authors present this attention interpretation as a hypothesis requiring corroboration with other models and datasets.Source: §4.2.1 · p. 6, Figure 5 · p. 6

4.2.2. THE PARAMETER-LEVEL IMPACT OF SFT

  • SFT produces much larger updates, including in mid-layer MLPs, which the authors hypothesize may contribute to its greater degradation on knowledge-intensive benchmarks.SFT also shows larger query and key updates, especially in middle layers.Source: Figure 6 · p. 6, §4.2.2 · p. 7

5. Applying insights from cross-checkpoint analyses via freezing

The authors test whether freezing selected parameters during SFT can preserve knowledge or retain reasoning gains, but find mixed and unreliable results.

5. Applying insights from cross-checkpoint analyses via freezing

  • Freezing experiments were limited to SFT to reduce computational cost, with other training details held identical to the original SFT runs.The authors caution that the intervention may be unprincipled: maths-only training may make unfrozen weights less compatible with frozen ones, and causal-tracing results may not transfer to knowledge-intensive tasks.Source: §5 · p. 7

5.1. Training only the query and key matrices

  • Training only query and key matrices leaves MATH-500 below baseline, reduces GSM8k performance considerably less than full SFT, and worsens every other benchmark noticeably.The authors say this suggests training only selected parameters limits the model's ability to fit the training dataset.Source: §5.1 · p. 8

5.2. Freezing the MLP matrices

  • The MLP-freezing test targets the hypothesis that SFT updates may corrupt factual associations stored in mid-layer MLPs.The rationale draws on prior work linking MLPs to key-value memories and mid-layer MLPs to factual associations.Source: §5.2 · p. 8
  • Freezing MLPs leaves MATH-500 performance stable and slightly mitigates GSM8k degradation versus full SFT, while GPQA Diamond rises above baseline but other benchmarks remain poor.The authors conclude that freezing is neither a reliable way to obtain reasoning-training benefits at lower cost nor to mitigate factual-knowledge loss.Source: §5.2 · p. 8

6. Conclusion

The conclusion reports a controlled comparison of GRPO and SFT and finds different trade-offs in training cost, benchmark performance, and model changes.

6. Conclusion

  • In the controlled comparison, GRPO yields modest gains on maths and small out-of-domain degradations, while SFT exacerbates both trends.The study trains the same model on the same problems with mostly the same hyperparameters.Source: §6 · p. 8
  • GRPO is expensive and unstable, whereas SFT is cheap and dependable, motivating further study of their accessibility and performance trade-offs.Source: §6 · p. 8
  • Both methods update query and key matrices most, but SFT produces larger updates and greater mid-layer changes; the authors frame GRPO as amplifying capabilities and SFT as acquiring new ones at old skills' expense.The authors say freezing parameters works poorly and that further research into parameter-level analysis is needed.Source: §6 · p. 8

Limitations

The authors identify limitations that qualify the comparison, including the starting model, data choices, random-seed uncertainty, and matched hyperparameters.

Limitations

  • OLMo-2-1124-7B-Instruct was already trained with RLVR, creating a confound despite being the best available variant for gaining rewards.Source: Limitations · p. 8
  • Data setup choices were made with GRPO's cost in mind, so SFT might have gained more under a different setup.Source: Limitations · p. 8
  • Section 4 results need confirmation beyond random-seed noise, and keeping hyperparameters mostly equal may sacrifice each algorithm's optimal performance.The authors aim to address these issues in future work.Source: Limitations · p. 8

Impact Statement

The statement frames understanding LLM training dynamics as important because models increasingly affect people’s lives and reasoning training is becoming widespread. It presents this investigation as early work that may inform future targeted interventions for bias and safety.

Impact Statement

  • LLMs are entering more areas of life while their internal computations and training dynamics remain comparatively poorly understood.The statement links that knowledge gap to the impact LLMs already have.Source: Impact Statement · p. 9
  • The spread of reasoning training makes it important to investigate the dynamics underlying these approaches.The authors describe their investigation as early results on how popular reasoning-training methods differ and change models internally.Source: Impact Statement · p. 9
  • The authors hope further work could support targeted interventions for bias or safety guardrails as capabilities advance.They seek to better characterize training dynamics and inform practitioner recommendations on alignment and safety.Source: Impact Statement · p. 9

A. Reproducibility

The appendix describes the reproducibility setup and the choices used to obtain workable GRPO runs. Question difficulty was especially consequential: an easier CN-K12 subset produced more reliable accuracy-reward gains than the full, olympiad-heavy dataset.

A. Reproducibility

  • The final runs used OLMo-2-1124-7B-Instruct, and both algorithms used the same preprocessed dataset of examples within the model’s 4,096-token limit.The final model was selected in part because its open data, checkpoints, and training code enable future investigation; comparisons focused on maths questions.Source: A. Reproducibility · p. 12
  • GRPO completions were capped at 1,500 tokens because a 4,096-token cap caused out-of-memory errors, a confound that leaves fewer tokens in backward passes.The authors note that this also means GRPO necessarily learns to generate fewer tokens; they identify mitigation as future work.Source: A. Reproducibility · p. 12

B.1. Question difficulty

  • Filtering to 91k CN-K12 questions yielded more reliable GRPO accuracy-reward gains than training on the full dataset.The full OpenR1-Math-220k training split contained 93k questions, 72.6% from olympiad competitions; the extended split was 69.6% CN-K12 questions, and no other setup adjustment had as pronounced an effect as changing difficulty.Source: §B.1 · pp. 12–13
  • The authors relate the difficulty effect to GRPO reinforcing existing capabilities rather than helping models learn novel tasks.They qualitatively observed that correctly answered OpenR1-Math-220k questions were easier, and leave the broader question of this GRPO behavior for future work.Source: §B.1 · p. 13

B.2. Reward function design

Reward design and training stability were central to obtaining workable GRPO runs. The authors emphasize rewards tied to correct answers, caution that format rewards can still drive updates on incorrect traces, and report stable configurations and completion-length behavior.

B.2. Reward function design

  • Rewarding correctly extracted answers was crucial for updating parameters toward successful reasoning, even when correct responses differed from dataset solutions.The authors considered accuracy rewards the most decisive of the three reward sources.Source: §B.2 · p. 13
  • Format rewards can promote faithful <think> instructions, but incorrect, well-formatted traces may still receive advantages and drive undesirable updates.The authors observed such updates in practice; during a long run, formatting could be learned quickly even as training accuracy struggled to improve.Source: §B.2 · pp. 13–14
  • The authors omitted tag-count rewards and weighted accuracy and format rewards at 1.0 and 0.1, respectively.This compromise aimed to teach prompt-following while placing greater emphasis on rollouts leading to correct solutions; Figure 9 shows format rewards rising quickly as accuracy rewards rose more slowly and plateaued.Source: Figure 9 · p. 13, §B.2 · p. 14

B.3. Training stability

  • A global batch size of 112, 28 completions per prompt, and one gradient update per global step gave the authors their most consistently stable results.They report that this configuration noticeably improved reliability and alleviated issues such as those in Figures 10, 14, and 13.Source: §B.3 · p. 14
  • Completion lengths remained stable during training, though the authors note that excluding harder olympiad questions may explain this.They fixed gradient updates per global step at one to avoid changing how many problems GRPO saw during training.Source: §B.3 · p. 14

C. Unsuccessful experiments with GRPO

The authors report that higher GRPO learning rates and attempts to regularize them did not yield a reliable improvement. The high-rate, tightly clipped setup was less stable and performed worse on every benchmark than the baseline.

C. Unsuccessful experiments with GRPO

  • Learning rates higher than 1e-6 were unreliable, sometimes crashing after hundreds of stable steps or causing huge updates or divergence.The authors also report that accuracy rewards consistently crashed after about 100 steps when they raised the rate from 1e-6 to 1e-5.Source: C. Unsuccessful experiments with GRPO · p. 14
  • Every benchmark evaluated performed worse with the 1e-5 learning rate and stricter regularization than with the 1e-6 baseline.The stricter setup paired 1e-5 with max grad norm 0.05, KL-loss scaling β of 0.04, and clipping scale ϵ of 0.1; Figure 11 compares it with the baseline.Source: C. Unsuccessful experiments with GRPO · p. 14, Figure 11 · p. 15
  • With CN-K12 training, the model began spamming XML tags until the 2,048-token completion limit, alongside exploding gradient norms.The authors present this as an instance of instability after the model learned the format; Figure 10 tracks gradient norms, format rewards, and completion lengths.Source: Figure 10 · p. 15

D. Revisiting Causal Tracing with OLMo-2-1124-7B-Instruct

Causal tracing on OLMo-2-1124-7B-Instruct identifies early and late MLP sites, with the late site motivating a freeze of layers 20–26. The analysis adapts Meng et al. (2023)'s repository and dataset for this model.

D. Revisiting Causal Tracing with OLMo-2-1124-7B-Instruct

Related papers

Loading 2507.10616v2…