PaperlayerView exact revision on arXiv

Source-linked AI summary

RL's Razor: Why Online Reinforcement Learning Forgets Less

Idan Shenfeld, Jyothish Pari, Pulkit Agrawal

arXiv:2509.04259v1cs.LG

TL;DR

The paper asks why RL preserves prior knowledge better than SFT despite comparable new-task performance. It compares the two fine-tuning approaches and finds that RL retains prior skills better, while on-policy learning favors new-task solutions closer to the original policy in KL.

  • Problem

    The paper asks what mechanism allows RL to improve on new tasks while minimally affecting prior knowledge, unlike SFT.

    Source: §1 · p. 2

  • Method

    The paper compares SFT and RL fine-tuning and analyzes KL divergence from the base policy on new tasks as a predictor of forgetting.

    Source: §1 · pp. 2–3

  • Results

    At similar new-task performance, RL preserves prior skills better than SFT and favors high-reward solutions closer to the original policy in KL.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper concludes that forgetting is governed by new-task KL divergence from the base policy rather than training-algorithm choice.

    Source: §7 · p. 9

    The mechanism linking larger KL shifts to prior-knowledge disruption remains unclear, and behavior at frontier scales, in diverse domains, and with off-policy algorithms is untested.

    Source: §7 · pp. 9–10

Abstract

from arXiv · show

Comparison of fine-tuning models with reinforcement learning (RL) and supervised fine-tuning (SFT) reveals that, despite similar performance at a new task, RL preserves prior knowledge and capabilities significantly better. We find that the degree of forgetting is determined by the distributional shift, measured as the KL-divergence between the fine-tuned and base policy evaluated on the new task. Our analysis reveals that on-policy RL is implicitly biased towards KL-minimal solutions among the many that solve the new task, whereas SFT can converge to distributions arbitrarily far from the base model. We validate these findings through experiments with large language models and robotic foundation models and further provide theoretical justification for why on-policy RL updates lead to a smaller KL change. We term this principle $\textit{RL's Razor}$: among all ways to solve a new task, RL prefers those closest in KL to the original model.

1 INTRODUCTION

Catastrophic forgetting threatens the vision of foundation models that continually adapt as long-lived agents. The introduction finds that RL preserves prior skills better than SFT by favoring solutions closer to the original policy in KL divergence.

1 INTRODUCTION

  • At matched new-task performance, RL preserves prior-task performance substantially better than SFT.Both methods can achieve high new-task performance, but SFT often gains it by erasing prior knowledge.Source: §1 · p. 2
  • Forgetting is accurately predicted by the KL divergence between fine-tuned and base policies, evaluated on the new-task distribution.This measure can be assessed and influenced during fine-tuning without access to past-task data.Source: §1 · pp. 2–3
  • On-policy RL favors high-reward solutions closer to the original policy because it reweights sampled outputs and gradually shifts the policy.A simplified theoretical analysis finds that policy-gradient methods converge to KL-minimal solutions even without explicit regularization.Source: §1 · pp. 2–3
  • Oracle SFT that minimizes KL while achieving perfect accuracy forgets even less than RL, indicating that RL’s advantage stems from implicit KL minimization.The introduction concludes that training biased toward KL-minimal solutions reduces forgetting.Source: §1 · p. 2
  • The findings suggest that post-training algorithms should minimize KL divergence from the base model to support continual adaptation without forgetting.The introduction presents this as a perspective for designing future methods that combine RL’s preservation of prior knowledge with SFT’s efficiency.Source: §1 · p. 3

2 RELATED WORK

The section situates the study within foundation-model post-training and forgetting research, then frames KL divergence as a predictor of forgetting and asks why RL and SFT differ.

Foundation Models and Post-training

  • Foundation models bring broad domain knowledge and some zero-shot abilities, while post-training—including SFT and RL—adapts them to specific downstream tasks.The study focuses on how SFT and RL affect forgetting.Source: §2 · p. 3

Catastrophic Forgetting

  • Prior forgetting-mitigation methods constrain parameter or feature changes or preserve outputs, but the authors say these heuristics address symptoms rather than causes.They seek a simple, predictive metric explaining when and why forgetting occurs across training algorithms.Source: §2 · p. 3
  • The authors identify KL divergence between the fine-tuned and base policy on the new task as a reliable predictor of forgetting across settings.They present it as an empirical forgetting law, not a new training algorithm.Source: §2 · p. 3
  • Elastic Weight Consolidation can approximate KL minimization (Chaudhry et al., 2018), while RL KL regularization has also been observed to reduce forgetting (Ouyang et al., 2022).The authors argue KL divergence is a reliable predictor, not merely a useful heuristic.Source: §2 · p. 3

SFT versus RL

  • Prior SFT–RL comparisons report stronger on-policy performance, generalization, or transfer, but had not examined relative susceptibility to catastrophic forgetting.Lai et al. (2025) report less forgetting under RL but attribute it to negative examples; this work instead identifies RL’s on-policy nature as key.Source: §2 · pp. 3–4

3 REINFORCEMENT LEARNING FORGETS LESS THAN SFT

Across language-model and robotic tasks, RL learned new tasks with minimal forgetting, while SFT’s new-task gains came at the cost of prior knowledge. The results also associate forgetting with KL divergence and show smaller KL shifts for RL than SFT.

3.1 PERFORMANCE TRADE-OFFS

  • The comparison used matched prompts and binary-success-reward GRPO without explicit KL regularization, evaluating held-out new-task performance against retention on unrelated prior-task benchmarks.Models were trained under diverse hyperparameter settings, and Pareto frontiers summarized the best achievable trade-offs between new-task performance and prior-task retention.Source: §3.1 · p. 4
  • Experiments covered three LLM tasks—math reasoning, science Q&A, and tool use—and one robotic pick-and-place task, with models and training setups varying by task.Each LLM task used Qwen 2.5 3B-Instruct; robotics used OpenVLA 7B, trained in SimplerEnv to pick up a can.Source: §3.1 · p. 4
  • Forgetting aligned to a single curve against KL divergence, and RL improved new-task accuracy with smaller KL shifts than SFT.The figure caption describes KL as a strong predictor of forgetting across methods and highlights the conservativeness of on-policy updates.Source: Figure 3 · p. 5
  • As new-task accuracy rose, RL retained prior-benchmark performance nearly unchanged, while SFT gains brought substantial forgetting, especially in Math.In Science Q&A and Tool Use, SFT retained some prior-task ability at lower new-task accuracy, but performance deteriorated rapidly as accuracy increased.Source: §3.1 · p. 5

4 SMALLER KL DIVERGENCES LEAD TO LESS FORGETTING

KL divergence between the fine-tuned and base models on the new task reliably predicts forgetting across RL and SFT, explaining their different retention despite comparable new-task performance.

ParityMNIST experiment

  • The controlled ParityMNIST setup makes full-convergence RL–SFT comparisons tractable by treating any even or odd digit label as correct for the corresponding parity.SFT uses labels from one arbitrary valid distribution, whereas RL rewards parity correctness and can converge to any valid distribution.Source: §4 · pp. 5–6
  • In ParityMNIST, RL achieved high new-task accuracy with slower prior-knowledge degradation than SFT, reproducing the retention gap in a simple MLP.The result shows the effect is not specific to large-scale transformers but also appears in fine-tuning deep generative models.Source: §4 · p. 6

KL as Predictor

  • A quadratic fit to forgetting versus KL achieved R2 = 0.96 on ParityMNIST and R2 = 0.71 in LLM experiments.Across two arbitrary SFT labelings, the ParityMNIST forgetting–KL curves coincided despite different Pareto frontiers; the LLM fit’s residuals were mean-zero and attributed to estimation noise.Source: §4 · pp. 5–6

Optimal SFT Distribution

  • SFT trained on the KL-minimizing oracle distribution retained more prior knowledge than RL, while RL-generated-data SFT matched RL’s accuracy–forgetting trade-off.The oracle distribution achieves 100% accuracy with minimal KL to the base model; these results indicate that the learned distribution, rather than the optimization algorithm, governs forgetting.Source: §4 · p. 6

5 ON-POLICY METHODS LEADS TO SMALLER KL DIVERGENCE

On-policy methods achieve the same new-task performance with smaller KL shifts than offline methods. The paper explains this through updates based on the model’s own distribution and proves that, under stated conditions, policy gradient selects a KL-closest optimal policy.

5.1 EXPERIMENTAL EVIDENCE

  • On-policy GRPO and 1–0 Reinforce reach the same task performance with significantly smaller KL shifts than offline SFT and SimPO, making on-policy data—not negative gradients—the critical factor.The four-objective comparison finds that 1–0 Reinforce behaves like GRPO, while SimPO resembles SFT.Source: Figure 4 · p. 7, §5.1 · p. 8

5.2 THEORETICAL PERSPECTIVE

  • On-policy policy-gradient updates sample from the model’s own distribution and reweight outputs by reward, nudging the policy toward a nearby distribution rather than an external one that may be distant.This mechanism explains why policy-gradient methods tend to remain closer to the base model in KL divergence.Source: §5.2 · p. 8
  • For binary rewards, policy gradient converges under suitable conditions to the KL-closest optimal policy within a convex representable family, measured from its initialization.The paper characterizes rejection sampling as a minimum-KL projection onto optimal policies and extends the result to policy-gradient convergence.Source: §5.2 · p. 8

6 ALTERNATIVE HYPOTHESIS

Across tested alternatives, forward KL between the fine-tuned and base model on the new task was the only consistent, high-fidelity predictor of catastrophic forgetting. Weight shifts, representation drift, update sparsity or rank, and other distributional distances did not explain forgetting as well.

Weight-level changes

  • Parameter-change metrics correlated only weakly with forgetting, which sometimes occurred despite small weight movement and sometimes did not occur despite large shifts.The tested metrics were L1, Fisher-weighted L2, and spectral norm; the Fisher matrix used inputs from the previous task. Prior work explored parameter constraints (Kirkpatrick et al., 2017; Aljundi et al., 2018; Zenke et al., 2017).Source: §6 · p. 9

Representation-level changes

  • Hidden-activation shifts showed representation drift during training, but differed between objectives and were not a good predictor of forgetting.Prior work focused on maintaining previous features (Jung et al., 2018; Dhar et al., 2019).Source: §6 · p. 9

Sparsity and rank of updates

  • Update sparsity arose from bfloat16 precision rather than an inherent RL property, and all tested algorithms produced full-rank weight changes.Float32 training achieved identical performance without sparse updates; bfloat16's limited mantissa can prevent small updates from crossing the representational threshold. This tests the hypothesis from Mukherjee et al. (2025).Source: §6 · p. 9

Distributional distances

  • Forward KL was the only consistent, high-fidelity predictor of catastrophic forgetting; reverse KL and total variation were weaker, and L2 distance did not approach its predictive power.The measures compared included forward KL, reverse KL, total variation, and L2 distance between output distributions.Source: Table 1 · p. 9, §6 · p. 9

7 DISCUSSION AND CONCLUSION

The study attributes forgetting to KL divergence from the base policy on the new task, with on-policy RL biased toward KL-minimal solutions that preserve prior knowledge. It proposes judging post-training algorithms by both new-task performance and how conservatively they move in KL.

7 DISCUSSION AND CONCLUSION

  • Forgetting is governed by KL divergence from the base policy on the new task, rather than training algorithm choice; on-policy RL favors KL-minimal solutions and forgets less than SFT.This bias preserves prior knowledge while acquiring new skills.Source: §7 · p. 9
  • The mechanism behind larger-KL forgetting remains unknown, and evidence is limited to moderate-scale LLMs and toy models, leaving frontier scales, diverse domains, and off-policy RL untested.Potential explanations include representational interference, implicit capacity limits, or other dynamics.Source: §7 · pp. 9–10
  • Post-training algorithms should be judged by both new-task optimization and how conservatively they move in KL relative to the base model.The authors clarify that offline data can still help, but continual learning requires updates to stay close to the KL-minimal path.Source: §7 · p. 10

A THEORY · B TRAINING AND EVALUATION DETAILS

The theory characterizes rejection sampling and policy-gradient updates as complementary KL projections, yielding an EM interpretation of binary-reward RL and conditions for convergence to a minimum-KL solution.

A THEORY

  • Rejection sampling from p with binary reward yields the unique I-projection of p onto distributions assigning reward one.This projection is the conditional distribution p(·|S), where S contains the accepted outcomes.Source: A THEORY · p. 16
  • Policy-gradient optimization is an M-projection of the reward-weighted sampling distribution q onto the feasible policy class, minimizing KL(q∥π).The sampling distribution is treated as fixed when computing the gradient.Source: A THEORY · pp. 16–17
  • Binary-reward policy-gradient RL alternates rejection-sampling I-projections and policy-class M-projections, making it equivalent to EM with information projection.The I-step conditions the policy on reward acceptance; the M-step projects onto the feasible policy class.Source: A THEORY · p. 17
  • With an e-flat, full-support policy class and a nonempty realizable optimal set, exact M-projections converge to the minimum-KL solution.The proposition assumes the optimal set is realizable by the policy class.Source: A THEORY · p. 17
  • Under the same assumptions, inexact M-projections with summable errors converge to the same minimum-KL limit.The proof identifies this case as generalized EM (GEM).Source: A THEORY · p. 17
  • Replacing raw rewards with advantage estimates preserves the expected gradient direction because the substitution acts as a control variate, while reducing variance.The paper says this maintains the projection-based interpretation for algorithms such as GRPO and PPO.Source: A THEORY · p. 18
  • Neural-network policy classes are not generally e-flat, which may prevent exact convergence to the minimum-KL solution but still explains practical RL's observed bias.The optimal policy set defined by the reward-one constraint is m-flat.Source: A THEORY · p. 18

B.1 LLM EXPERIMENTS

The LLM experiments use a multi-step protocol to construct learning–forgetting trade-off curves, with GRPO used for reinforcement-learning runs unless stated otherwise.

B.1 LLM EXPERIMENTS

  • Unless otherwise stated, all reinforcement-learning experiments use GRPO (Shao et al., 2024).Source: §B.1 · p. 18
  • For SFT annotations, Math answers were generated with DeepSeek R1 (Guo et al., 2025a), yielding valid annotations for 96% of the dataset; Science Q&A annotations from GPT-4o were correct throughout.For Math, up to 16 responses per prompt were sampled and one matching the correct final answer was retained.Source: §B.1 · p. 18
  • Trade-off curves use hyperparameter sweeps, task-specific accuracy evaluation, previous-task benchmarks, Pareto filtering within 2 accuracy points, and exponential curve fitting.Figure 6 gives an example of creating Pareto frontier plots.Source: §B.1 · p. 18, Figure 6 · p. 18

B.2 ROBOTIC EXPERIMENTS

The robotic experiments tested whether the KL–forgetting relationship generalizes to embodied policies, using OpenVLA-7B in SimplerEnv for can pick-and-place fine-tuning and drawer opening/closing as the forgetting task.

B.2 ROBOTIC EXPERIMENTS

  • The study fine-tuned OpenVLA-7B (Kim et al., 2024) in SimplerEnv (Li et al., 2024b) on can pick-and-place and measured forgetting on drawer opening/closing.The pareto-frontier followed the same protocol as in the LLM experiments.Source: §B.2 · pp. 18–19
  • Training varied object placement over a 10 × 10 grid, and evaluation sampled 100 random object locations uniformly within the same area.Initial positions covered obj-init-x ∈[−0.35 −0.12] and obj-init-y ∈[−0.02, 0.42].Source: §B.2 · p. 19
  • SFT used 10 successful RT-1 (Brohan et al., 2022) trajectories per grid point, filtered for success, and trained for one epoch with AdamW.The sweep varied batch sizes {16, 32, 64} and learning rates {1×10−6, 3×10−6, 5×10−6, 7×10−6, 9×10−6, 1×10−5, 3× 10−5}.Source: §B.2 · p. 19
  • RL used REINFORCE with a reward-normalization baseline, no explicit KL regularization, and binary task-completion rewards.Each iteration collected five trajectories per grid point, and RL used the same training configuration as SFT.Source: §B.2 · p. 19

B.3 MNIST EXPERIMENTS … C.1 REPRESENTATION PRESERVATION

On the new task, RL-trained models retained greater representational similarity to the base model than SFT-trained models. The section also describes the MNIST fine-tuning strategies and the representation-comparison measures used in the experiments.

B.3 MNIST EXPERIMENTS

  • MNIST experiments used a 3-layer MLP with 512- and 256-unit hidden layers and a 10-unit output, receiving flattened images plus a task indicator.The input indicator was +1 for ParityMNIST and −1 for FashionMNIST.Source: §B.3 · p. 19
  • MNIST fine-tuning compared GRPO, KL-regularized GRPO, two SFT label mappings, and SFT using annotations sampled from an oracle distribution.The two SFT mappings assigned parity classes either to labels 0/1 or randomly to {0, 4}/{1, 5}.Source: §B.3 · pp. 19–20
  • The MNIST oracle distribution achieves perfect task accuracy while minimizing KL divergence from the pretraining distribution, with a closed-form solution computed for each image.The resulting distribution is sampled to produce SFT annotations.Source: §B.3 · p. 20

B.4 CENTERED KERNEL ALIGNMEN

  • CKA compares kernels derived from model representations, while CKNNA masks similarities to mutual k-nearest-neighbor pairs and reduces to CKA when all off-diagonal pairs are included.Source: §B.4 · p. 20

C.1 REPRESENTATION PRESERVATION

  • Because benchmark accuracy may confound capability loss with formatting mismatch, C.1 probes representation geometry on unrelated Wikipedia paragraphs and compares SFT and RL models with similar final accuracy.The analysis compares embeddings on shared data using kernel similarity measures.Source: §C.1 · pp. 20–21
  • RL models retained higher similarity to the base model (CKNNA=0.94) than SFT models (CKNNA=0.56), with less representational drift after fine-tuning.The results suggest RL integrates new abilities while leaving the overall representation space largely intact, whereas SFT alters its geometry more extensively.Source: §C.1 · p. 21

C.2 SCALING AND FORGETTING · C.3 OPTIMIZATION DYNAMICS

Across model sizes, SFT’s gains on Science Q&A come with prior-task forgetting, while optimization steps that cause larger distributional shifts are more aligned with forgetting. The supplied passages also show that RL-trained models remain more representation-aligned with the base model, and that SFT can match an RL teacher when the final distribution is matched.

C.2 SCALING AND FORGETTING

  • With comparable task performance, SFT representations diverge substantially from the base model, whereas RL representations remain more closely aligned.Source: Figure 7 · p. 21
  • To evaluate whether catastrophic forgetting diminishes with model size, the authors repeated Section 3’s SFT experiments on Science Q&A using Qwen 2.5 models with 3B, 7B, and 14B parameters.Source: §C.2 · p. 21
  • Across Qwen 2.5 sizes 3B, 7B, and 14B, SFT’s Science Q&A gains require prior-capability forgetting, with substantial prior-benchmark degradation at high accuracy.Larger models start with better general capabilities, but the new-task performance versus prior-task retention trade-off remains unchanged across sizes.Source: §C.2 · p. 21, Figure 8 · p. 21

C.3 OPTIMIZATION DYNAMICS

  • The per-step analysis measured forgetting direction as the cosine between the FashionMNIST loss gradient and parameter update, and KL shift as the output-distribution change on ParityMNIST.A positive cosine indicates an update that increases FashionMNIST loss; a negative cosine indicates one that reduces it.Source: §C.3 · p. 21
  • Per-step KL changes strongly correlate with alignment to the forgetting gradient: larger distributional shifts tend to accompany updates more aligned with forgetting.The analysis concludes that catastrophic forgetting at the optimization-dynamics level is driven by updates inducing larger distributional shifts on the new task.Source: §C.3 · p. 22, Figure 10 · p. 22
  • SFT distillation can match an RL teacher within noise on both MNIST and FashionMNIST by sweeping student hyperparameters, suggesting the final model distribution matters more than the optimization path.Source: Figure 9 · p. 22

Related papers

Loading 2509.04259v1…