PaperlayerView exact revision on arXiv

Source-linked AI summary

RL's Razor: Why Online Reinforcement Learning Forgets Less

Idan Shenfeld, Jyothish Pari, Pulkit Agrawal

arXiv:2509.04259v1cs.LG

TL;DR

The paper asks why adapting models to new tasks can erase prior capabilities and compares RL with SFT to identify the governing mechanism. It finds that KL divergence predicts forgetting and that on-policy RL implicitly favors KL-minimal solutions, preserving prior knowledge more effectively. The paper supports this account empirically and theoretically while identifying open scope and mechanistic questions.

  • Problem

    Catastrophic forgetting limits foundation models that must continually adapt, and it remains unclear why RL and SFT differ in their effects on prior knowledge.

  • Method

    The paper compares SFT and on-policy RL across language and robotic tasks, relates forgetting to KL divergence from the base policy, and analyzes the mechanism theoretically.

  • Results

    RL forgets less than SFT at comparable new-task performance, while KL divergence reliably predicts forgetting and on-policy updates favor smaller KL shifts.

  • Takeaways & Limitations

    Among high-reward solutions for a new task, favoring solutions closer to the base policy in KL can preserve prior knowledge during adaptation.

  • Takeaways & Limitations

    The KL–forgetting link has not been established at frontier scales, in more diverse generative domains, or for online off-policy algorithms, and its mechanism remains unresolved.

Abstract

from arXiv · show

Comparison of fine-tuning models with reinforcement learning (RL) and supervised fine-tuning (SFT) reveals that, despite similar performance at a new task, RL preserves prior knowledge and capabilities significantly better. We find that the degree of forgetting is determined by the distributional shift, measured as the KL-divergence between the fine-tuned and base policy evaluated on the new task. Our analysis reveals that on-policy RL is implicitly biased towards KL-minimal solutions among the many that solve the new task, whereas SFT can converge to distributions arbitrarily far from the base model. We validate these findings through experiments with large language models and robotic foundation models and further provide theoretical justification for why on-policy RL updates lead to a smaller KL change. We term this principle $\textit{RL's Razor}$: among all ways to solve a new task, RL prefers those closest in KL to the original model.

1 INTRODUCTION

The paper studies catastrophic forgetting during continual adaptation and compares RL with SFT. It identifies KL divergence as a predictor of forgetting and argues that on-policy RL favors KL-minimal solutions that preserve prior knowledge.

  • Motivation: Catastrophic forgetting remains a persistent obstacle to models that must continually acquire new capabilities.The paper frames preserving prior capabilities as essential for long-lived adaptive agents.
  • Core finding: RL fine-tuning forgets less than SFT even when both methods reach the same new-task performance.SFT often gains new-task performance by erasing prior knowledge, whereas RL better preserves old skills.
  • RL’s Razor: On-policy RL is biased toward KL-minimal high-reward solutions, while SFT can converge to solutions much farther from the original policy.RL samples from its own distribution and gradually reweights likely outputs rather than matching an arbitrary external distribution.
  • Validation: An oracle SFT distribution that minimizes KL divergence while achieving perfect accuracy produces less forgetting than RL.This supports the claim that RL’s advantage comes from implicit KL minimization rather than from RL being inherently different.
  • Forgetting law: KL divergence between the fine-tuned and base policy on the new task strongly predicts catastrophic forgetting across objectives and hyperparameters.The law can be measured and influenced during fine-tuning without access to past-task data.

2 RELATED WORK

Related work has addressed forgetting through parameter, feature, or output-distribution constraints, but has not explained its underlying cause or the RL–SFT difference. This paper studies that gap through KL divergence and on-policy learning.

  • Foundation models and post-training: Foundation models are general-purpose backbones adapted to downstream tasks through methods including SFT and RL.The paper focuses its comparison on how these post-training schemes affect forgetting.
  • Catastrophic forgetting: Prior forgetting methods constrain weight updates, preserve learned features, or regularize output-distribution shifts.These approaches can reduce forgetting but primarily target its effects rather than its underlying cause.
  • Paper perspective: The paper identifies KL divergence from the base policy as an empirical forgetting law that explains why RL forgets less than SFT.It also relates some mitigation strategies, such as Elastic Weight Consolidation, to KL minimization.
  • SFT versus RL: Prior comparisons of RL and SFT emphasized new-task performance, while their relative susceptibility to catastrophic forgetting remained unexamined.The paper positions forgetting as the missing comparison between the two post-training methods.

3 REINFORCEMENT LEARNING FORGETS LESS THAN SFT

Across language and robotic tasks, RL achieves comparable new-task performance while retaining prior abilities more effectively than SFT. The trade-off is evaluated through held-out new-task performance and prior-task benchmarks across hyperparameter settings.

  • Experimental setup: The experiments compare SFT with GRPO-based RL across three LLM tasks and one robotic pick-and-place task.The tasks cover math reasoning, science question answering, tool use, and robotics.
  • Evaluation: New-task performance is measured on held-out test sets, while forgetting is measured as performance drops on unrelated prior-task benchmarks.LLM evaluations include Hellaswag, TruthfulQA, MMLU, IFEval, Winogrande, and HumanEval.
  • Evaluation: Dozens of models with diverse hyperparameters define Pareto frontiers trading off new-task performance against prior-task performance.Each frontier represents the best achievable trade-off for a method.
  • Results: As RL accuracy increases on new tasks, prior-benchmark performance remains nearly unchanged, whereas SFT improvements consistently incur substantial forgetting.The contrast is strongest in Math, while Science Q&A and Tool Use deteriorate rapidly near higher new-task accuracy.
  • Results: RL reaches similar new-task performance only by sacrificing less prior knowledge than SFT.The same pattern is summarized across the evaluated language and robotic tasks.

4 SMALLER KL DIVERGENCES LEAD TO LESS FORGETTING

Across RL and SFT, forgetting is reliably predicted by forward KL divergence between the fine-tuned and base models on the new task. Controlled ParityMNIST experiments reproduce the RL–SFT gap and show that explicitly minimizing KL can outperform RL.

  • Forward KL divergence on the new task reliably predicts catastrophic forgetting across RL and SFT.
  • ParityMNIST enables full-convergence experiments and systematic ablations of the RL–SFT forgetting gap.The task reframes MNIST as parity prediction, allowing multiple correct output distributions.
  • RL reaches high new-task accuracy with slower prior-knowledge degradation, whereas SFT exhibits a steeper accuracy–forgetting trade-off.
  • A quadratic fit achieves R2 = 0.96 for the forgetting–KL relationship in ParityMNIST.Different arbitrary SFT labelings produced coincident forgetting–KL curves despite differing Pareto frontiers.
  • Oracle SFT trained on the KL-minimal distribution retained more prior knowledge than RL and achieved the best observed accuracy–forgetting trade-off.SFT distilled from an RL-trained model matched RL’s accuracy–forgetting trade-off.

5 ON-POLICY METHODS LEADS TO SMALLER KL DIVERGENCE

The paper attributes RL’s smaller KL shifts to on-policy sampling rather than negative gradients. On-policy methods favor KL-minimal solutions, and theory formalizes this as projection toward optimal policies while remaining close to the starting policy.

  • 5.1 EXPERIMENTAL EVIDENCE: The key distinction is on-policy sampling: RL draws outputs from the model’s distribution, whereas SFT uses fixed external annotations.
  • 5.1 EXPERIMENTAL EVIDENCE: On-policy methods achieve comparable task performance with smaller KL divergence from the base model than SFT and other offline methods.GRPO and 1–0 Reinforce behave similarly, while SimPO resembles SFT.
  • 5.1 EXPERIMENTAL EVIDENCE: Negative gradients are not the critical factor, because 1–0 Reinforce without negative examples resembles GRPO, while offline SimPO resembles SFT.
  • 5.2 THEORETICAL PERSPECTIVE: Policy-gradient updates conservatively reweight outputs sampled from the current policy toward higher-reward outcomes.This projection perspective explains why policy-gradient methods tend to remain close to the base model in KL divergence.
  • 5.2 THEORETICAL PERSPECTIVE: Figure 5 depicts alternating projections that carry π0 into the optimal-policy set while preferring the closest solution in KL.
  • 5.2 THEORETICAL PERSPECTIVE: In the binary-reward case, policy gradient converges to the KL-minimal optimal policy within the representable family.Among optimal representable policies, it selects the one closest in KL divergence to the initialization.

6 ALTERNATIVE HYPOTHESIS

The paper evaluates weight, representation, sparsity, rank, and distributional alternatives as predictors of forgetting. Forward KL is the only candidate that consistently provides high-fidelity predictive power.

  • Weight-level changes: Weight-level metrics correlate only weakly with forgetting, so large parameter shifts can occur without corresponding forgetting.
  • Representation-level changes: Representation drift occurs during training, but differing curves across objectives make it a poor predictor of forgetting.
  • Sparsity and rank of updates: Observed sparse updates were attributed to bfloat16’s limited mantissa rather than establishing sparsity as the cause of RL’s behavior.
  • Distributional distances: Reverse KL showed a good signal and total variation moderately correlated with forgetting, but neither approached forward KL’s predictive power.
  • Forward KL is the only consistent and high-fidelity predictor of catastrophic forgetting among the evaluated candidates.

7 DISCUSSION AND CONCLUSION

The study identifies KL divergence from the base policy on the new task as the factor governing catastrophic forgetting, explaining why RL preserves prior skills better than SFT. It proposes evaluating post-training by both new-task performance and the conservativeness of KL movement, while noting important unresolved scope and mechanistic questions.

  • Catastrophic forgetting is governed by KL divergence from the base policy on the new task rather than the training algorithm itself.
  • On-policy RL forgets less than SFT because it naturally favors KL-minimal solutions that preserve prior knowledge while acquiring new skills.
  • The mechanism by which larger KL shifts disrupt prior knowledge remains unresolved, with representational interference and capacity limits among possible explanations.
  • The KL–forgetting relationship remains untested at frontier scales, in more diverse generative domains, and for online off-policy algorithms.
  • Post-training algorithms should be judged by both new-task optimization and how conservatively they move in KL relative to the base model.

A THEORY

The theory characterizes rejection sampling as an information projection and policy-gradient updates as a moment projection, yielding an EM interpretation of RL. Under exponential-family and realizability assumptions, exact updates converge to the feasible policy closest to the starting policy in KL, while neural-network parameterizations may prevent exact convergence.

  • Projection view: Rejection sampling from a policy, accepting outputs with binary reward 1, produces the I-projection onto distributions achieving unit expected reward.The feasible distributions place all probability mass on successful outcomes, making the conditional distribution the unique KL minimizer.
  • Projection view: Policy-gradient optimization is equivalent to an M-projection of the reward-weighted distribution onto the admissible policy class.The equivalence follows because maximizing expected log policy probability is equivalent to minimizing KL(q∥π) when the entropy term of q is constant.
  • EM interpretation: Single-step binary-reward RL is equivalent to an EM procedure with information projection.The I-step comes from rejection sampling, while the policy-gradient update supplies the M-step.
  • Convergence: Under an e-flat policy family with full support and a nonempty realizable optimal set, exact projection steps converge to the minimum-KL feasible policy.With inexact M-steps, convergence is supported when the approximation errors are summable.
  • Practical considerations: The exact convergence result may fail for neural-network policy classes because their representable policy sets are not generally e-flat.The theorem nevertheless provides a principled explanation for the bias observed in practical RL algorithms.

B.1 LLM EXPERIMENTS

The LLM experiments construct learning–forgetting trade-off curves by sweeping hyperparameters, evaluating new and previous tasks, and Pareto-filtering trained models. New-task evaluation uses task-specific accuracy or API-call matching, while previous-task performance is measured on unrelated benchmarks.

  • Experimental setup: The LLM experiments use GRPO unless otherwise stated.This establishes the default RL algorithm for the reported experiments.
  • Data preparation: Reasoning annotations were obtained by sampling responses from DeepSeek R1 for Math and GPT-4o for Science Q&A, retaining correct responses.This produced valid annotations for 96% of the Math dataset and the entire Science Q&A dataset.
  • Trade-off curves: Learning–forgetting curves are built from a broad hyperparameter sweep followed by new-task evaluation, previous-task evaluation, and Pareto filtering.Each retained model represents a trade-off between new-task performance and previous-task performance.
  • Evaluation: For Math and Science Q&A, new-task accuracy compares final answers with ground truth while ignoring intermediate reasoning chains.Tool Use instead evaluates extracted API calls against ground-truth calls using regular expressions.

B.2 ROBOTIC EXPERIMENTS

The robotic experiments test the RL–SFT forgetting gap with OpenVLA-7B in SimplerEnv, training on pick-and-place and measuring forgetting on drawer manipulation. RL and SFT use matched configurations, with performance trade-offs constructed as in the LLM experiments.

  • Task setup: The robotic study fine-tunes OpenVLA-7B on can pick-and-place in SimplerEnv and measures forgetting on drawer opening and closing.The distinct evaluation task tests whether the KL–forgetting relationship extends to embodied policies.
  • Data collection: The data vary object placement over a 10 × 10 grid and evaluate 100 uniformly sampled locations.Training positions span obj-init-x ∈[−0.35 −0.12] and obj-init-y ∈[−0.02, 0.42].
  • SFT: SFT uses 10 successful RT-1 trajectories per grid point and sweeps batch sizes {16, 32, 64} and seven learning rates.Training uses one epoch, AdamW, bfloat16 precision, and no weight decay.
  • RL: RL uses REINFORCE with reward normalization, five trajectories per grid point, binary task-success rewards, and no explicit KL regularization.RL uses the same training configuration as SFT apart from its optimization procedure.

B.3 MNIST EXPERIMENTS

The MNIST experiments compare several RL and SFT strategies using a shared MLP and examine whether minimum-KL supervision preserves prior behavior. They also measure representational change with CKNNA on task-unrelated Wikipedia data.

  • Setup: All MNIST experiments use a 3-layer MLP with hidden sizes 512 and 256 and a 10-dimensional output.Inputs are flattened 28 × 28 images concatenated with a task indicator for ParityMNIST or FashionMNIST.
  • Fine-tuning methods: The compared methods include GRPO, GRPO with KL regularization, two arbitrary SFT labelings, and SFT using an oracle distribution.The oracle distribution is designed to achieve correctness while remaining closest to the pretraining distribution.
  • Oracle distribution: The oracle distribution minimizes KL divergence to the pretraining policy subject to perfect task accuracy, and its samples provide SFT annotations.Because the objective is convex with a linear constraint, the solution has a closed form for every image.
  • Representation analysis: CKNNA compares local neighborhood geometry between representations and reduces to standard CKA when all off-diagonal pairs are mutual nearest neighbors.The analysis uses kernel-based comparisons of embedding relationships.
  • Results: RL-trained models achieve CKNNA=0.94 similarity to the base model, whereas SFT-trained models reach CKNNA=0.56.The comparison uses random Wikipedia paragraphs unrelated to fine-tuning tasks and models with similar new-task accuracy.

C.2 SCALING AND FORGETTING

Across model sizes, SFT retains the same trade-off: improving new-task performance comes with prior-task forgetting, while RL and SFT can reach comparable task performance despite different representational change. Optimization analysis links larger new-task distributional shifts to forgetting, and distillation results indicate the final distribution matters more than the optimization path.

  • Representational change: Although SFT and RL achieve comparable task performance, SFT representations diverge substantially from the base model while RL representations remain more closely aligned.This comparison uses CKA similarity to the base model during training.
  • Scaling and forgetting: Across 3B, 7B, and 14B Qwen 2.5 models, higher Science Q&A accuracy still substantially degrades prior-benchmark performance.Larger models begin with better general capabilities, but the new-task versus prior-task retention trade-off remains unchanged.
  • Scaling and forgetting: SFT Pareto frontiers show that gains on the new task require forgetting prior capabilities across all evaluated model sizes.The trade-off is visible on Science Q&A for Qwen 2.5 models of 3B, 7B, and 14B parameters.
  • Optimization dynamics: Per-step updates producing larger KL shifts on ParityMNIST tend to align more with the FashionMNIST forgetting gradient.The analysis measures forgetting-direction cosine similarity and new-task KL shift for individual training steps.
  • Final distribution: SFT can match an RL teacher on both new and prior tasks, suggesting that the final model distribution, rather than the optimization path, determines the accuracy trade-off.The result comes from sweeping student hyperparameters in SFT distillation on MNIST and FashionMNIST.
Loading 2509.04259v1…