On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training

Xueyan Niu; Bo Bai; Wei Han; Weixi Zhang

Source-linked AI summary

On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training

Xueyan Niu, Bo Bai, Wei Han, Weixi Zhang

arXiv:2601.07389v2Jan 12, 2026cs.LGcs.AIcs.IT

TL;DR

The paper asks whether alternating SFT and RL can be theoretically decoupled despite their different objectives. It analyzes both canonical orderings using distributional and landscape-based theory, deriving conditions for degradation, stopping, thresholds, and gradient misalignment. The results show that the second stage degrades performance achieved by the first, with experiments on Qwen3-0.6B confirming the predicted coupling.

Problem
The paper addresses the lack of a theoretical account of whether SFT and RL can be decoupled when used in post-training.
Method
The paper analyzes SFT-then-RL and RL-then-SFT through KL-based distributional analysis, PL-based landscape analysis, gradient-alignment bounds, and experiments.
Results
SFT and RL are inherently coupled: whichever stage comes second degrades the performance achieved by the first.
Takeaways & Limitations
The two post-training stages cannot be separated without loss of prior performance, while quadratic damage versus at-most-linear reward improvement implies an optimal RL stopping point.
Takeaways & Limitations
The theoretical results rely on assumptions including the PL condition and quantitative curvature at an RL maximizer.

Abstract

from arXiv · show

Post-training of large language models routinely interleaves supervised fine-tuning (SFT) with reinforcement learning (RL). These two methods have different objectives: SFT minimizes the cross-entropy loss between model outputs and expert responses, while RL maximizes reward signals derived from human preferences or rule-based verifiers. Modern reasoning models have widely adopted the practice of alternating SFT and RL training. However, there is no theoretical account of whether they can be decoupled. We prove that decoupling is impossible in either order: (1) SFT-then-RL coupling: RL increases SFT loss under both distributional (KL-based) and landscape (PL-based) analyses; and (2) RL-then-SFT coupling: SFT lowers the reward achieved by RL under analogous conditions. Under the PL condition, we further derive the optimal RL duration that balances reward improvement against SFT degradation, identify the non-decoupling threshold governing when RL can improve SFT, and bound the gradient misalignment via spectral concentration. Experiments on Qwen3-0.6B confirm the predicted degradation, verifying that SFT and RL cannot be separated without loss of prior performance in the post-training pipeline.

1 Introduction

The paper studies whether SFT and RL can be decoupled in either training order and shows that the stages remain coupled through losses, rewards, and gradient geometry. It develops theoretical conditions for degradation, stopping, thresholds, and misalignment, then confirms the predicted effects empirically on Qwen3-0.6B.

Motivation: Post-training commonly combines SFT, which teaches task-specific responses, with RL, which reinforces patterns favored by humans or rule-based verifiers.These methods follow pretrained language-model training and refine the base model's behavior.
Research question: The paper asks whether SFT and RL can be decoupled, and when capability degradation should be accepted as the cost of alignment.Any cascade of the two strategies is reduced to SFT-then-RL or RL-then-SFT.
SFT-then-RL: SFT-then-RL coupling: RL inevitably increases SFT loss, so the two stages cannot be decoupled in this order.The result is established in Theorem 3.1, including at an SFT optimum where the first-order term vanishes.
RL-then-SFT: RL-then-SFT coupling: SFT can create a persistent performance gap that lowers the reward achieved by RL.Theorem 4.1 establishes nontrivial interaction when SFT follows RL.
Structural results: Under the PL condition, SFT degradation grows quadratically with RL duration while reward improvement grows at most linearly, yielding an optimal stopping point.The paper also identifies a non-decoupling threshold governing when RL can improve SFT performance.
Structural results: Spectral concentration bounds gradient misalignment and explains why SFT and RL gradients can be nearly orthogonal despite hyperparameter tuning.The paper links this structural mismatch to the observed coupling.
Empirical verification: Experiments on Qwen3-0.6B show increased cross-entropy loss after RL and reward degradation after further SFT, confirming that prior performance cannot be preserved across stages.Both canonical pipelines were evaluated using the CoLA dataset.

2 Problem Setup

The paper formulates SFT and RL as sequential post-training operations with distinct objectives and studies whether their order can be decoupled. SFT minimizes cross-entropy on task-specific input-output data, while RL maximizes reward under boundedness, prompt-distribution, and PL-landscape assumptions.

Post-training operations: SFT and RL are analyzed as two sequential operations whose order may be SFT-then-RL or RL-then-SFT.The setup investigates whether these operations commute in terms of training loss and reward.
Supervised fine-tuning: SFT minimizes next-token negative log-likelihood, equivalently cross-entropy, on task-specific prompt-output pairs.The resulting model is optimized for mappings represented in the SFT dataset.
Reinforcement learning: RL updates the policy to maximize expected reward for responses, using methods such as PPO and GRPO with entropy regularization.When the ground-truth reward is unavailable, a proxy reward model can be trained from preference data.
Assumptions: The analysis assumes SFT and RL prompts share a domain and are sampled from a common prompt distribution.The prompt sets are modeled as uniformly sampled according to q(x).
Assumptions: The RL reward is bounded, and the SFT loss is assumed to satisfy a Polyak–Lojasiewicz condition weaker than strong convexity.The PL condition is cited as applicable to over-parameterized neural networks.

3 SFT-then-RL Coupling

The SFT-then-RL analysis shows that reward improvement necessarily increases SFT loss, even from an SFT-matched reference model. Under the PL condition, the resulting trade-off yields an optimal RL duration, while gradient geometry explains the coupling and identifies when RL can improve SFT.

Non-decoupling result: Any nontrivial RL reward improvement after converged SFT causes a positive increase in SFT cross-entropy loss.Theorem 3.1 establishes this degradation when the SFT reference model matches the SFT data.
Non-decoupling result: Under the PL condition, SFT degradation scales quadratically with RL duration, whereas RL reward improvement scales at most linearly.This quadratic-versus-linear trade-off motivates optimizing the RL stopping time.
Optimal RL duration: The composite-loss lower bound combines an SFT baseline, quadratic coupling damage in T_RL, and linear RL improvement.The optimal duration minimizes this lower bound rather than treating RL duration as independent of SFT degradation.
Optimal RL duration: The optimal RL duration decreases as the learning rate or mean RL gradient norm increases and depends on the SFT landscape constant µ_SFT.Thus, the recommended duration varies across models, tasks, and training progress.
Gradient alignment: When SFT and RL gradients occupy largely disjoint low-dimensional subspaces, their alignment is necessarily small regardless of hyperparameter tuning.This spectral-concentration result provides a mechanism for the observed near-orthogonality and coupling.
When RL improves SFT: RL can improve SFT when the effective curvature is nonpositive or when objective alignment exceeds the curvature-dependent critical threshold.The alignment coefficient may be small or negative for substantially different rewards, but improvement is possible when reward and SFT partially align.

4 RL-then-SFT Coupling

The RL-then-SFT pipeline remains coupled: subsequent SFT degrades the reward achieved by RL under distributional and PL-based analyses.

Weak coupling result: SFT after RL cannot improve RL reward beyond a bound controlled by the distribution-shift budget when the update is small.The weak result assumes bounded average conditional KL shift and covers settings such as early stopping, regularization, or limited steps.
Strong coupling result: Under additional optimality and bounded-shift assumptions, subsequent SFT causes a measurable reward drop relative to the RL checkpoint.The result applies when the first RL phase has converged and further reward gains are negligible.
Strong coupling result: Theorem 4.1 states that the second SFT phase degrades the reward obtained during the first RL phase.The theorem formalizes the non-decoupling of RL followed by SFT.
PL analysis: Under the PL condition on the RL objective, reward degradation admits a quadratic lower bound in the parameter displacement caused by SFT.The proof reverses the roles of SFT and RL from the analogous earlier theorem.
Experimental implication: Figure 3b shows reward collapsing when SFT begins and eventually falling below the base-model level.This experimental pattern supports the theoretical RL-then-SFT coupling result.

5 Empirical Results

Experiments on Qwen3-0.6B with CoLA test both training orders and observe the predicted degradation: RL raises SFT loss, while SFT lowers RL reward.

Setup: The experiments use Qwen3-0.6B and CoLA, whose 10,657 English sentences are labeled grammatical or ungrammatical.A CoLA-style SFT dataset uses the original train/test split with prompts and responses.
SFT-then-RL: The SFT-then-RL pipeline performs two SFT epochs before GRPO reinforcement learning on the resulting checkpoint.This setup tests whether RL preserves the preceding SFT objective.
SFT-then-RL: When RL begins, SFT test loss rises abruptly and eventually exceeds the original base-model value.The result is consistent with Theorem 3.1.
RL-then-SFT: The RL-then-SFT pipeline runs GRPO to convergence, then applies two SFT epochs on the same CoLA-style data.Rewards are evaluated on the RL test set using robust evaluation to reduce format sensitivity.
RL-then-SFT: The final SFT checkpoint reaches mean@1 ≈0.343 versus the base model’s ≈0.385, corresponding to approximately 67.2% versus 69.5% accuracy.SFT steps overwrite RL-tuned behavior and produce a sharp reward decline, consistent with Theorem 4.1.

6 Conclusion

The paper concludes that SFT and RL are inherently coupled: whichever stage comes second degrades the performance achieved by the first, both theoretically and empirically.

Main conclusion: Theoretical analysis shows that either training order incurs degradation in the prior objective, so SFT and RL cannot be decoupled without loss.KL analysis establishes nonnegative degradation, while PL analysis gives explicit quadratic bounds based on parameter displacement.
Implications: Under the PL condition, coupling damage grows quadratically with RL duration while reward improvement grows at most linearly, implying an optimal stopping point.The analysis also identifies a threshold for when RL can improve SFT performance.
Mechanism and evidence: Spectral concentration explains near-orthogonality between the SFT and RL gradients through their concentration in different parameter subspaces.Qwen3-0.6B experiments corroborate abrupt degradation in either cross-entropy or reward after switching stages.