PaperlayerView exact revision on arXiv

Source-linked AI summary

Off-Policy Merging Beats On-Policy Self-Distillation for Continual Learning

Chen Henry Wu, Thomas Zhang, Aditi Raghunathan

arXiv:2610.05872v1cs.LGcs.AIcs.CL

TL;DR

Continual learning must add new capabilities without forgetting existing ones, yet useful data is often off-policy and direct SFT can interfere with post-trained capabilities. The paper introduces grafting, which learns scaled updates on earlier donor checkpoints and can mask sensitive directions. Across three settings, grafting Pareto-dominates SFT and OPSD on new- and old-task performance while avoiding expensive on-policy sampling.

  • Problem

    The paper asks whether continual learning requires on-policy training even though new knowledge and capabilities often arrive in off-policy data.

    Source: §1 · p. 1

  • Method

    Grafting applies a scaled update learned on an earlier donor checkpoint to the post-trained model, optionally masking the most Fisher-sensitive directions.

    Source: §1 · p. 2

  • Results

    Across expert-trace distillation, self-improvement, and knowledge injection, grafting Pareto-dominates SFT and OPSD on new-task and old-task performance while avoiding expensive on-policy sampling.

    Source: §1 · p. 3, §9 · p. 11

  • Takeaways & Limitations

    The results suggest that on-policy updates are not necessary for continual learning and motivate further work on making off-policy training generalize.

    Source: §1 · p. 3

    The interference explanation is demonstrated with a minimal model of two linear-regression tasks represented by a mixture of experts.

    Source: §3 · pp. 3–4

Abstract

from arXiv · show

A long-standing goal of AI is a model that can continually learn and improve itself. On post-trained models, supervised finetuning (SFT) on new data often causes poor generalization and catastrophic forgetting. As such, the conventional wisdom is that on-policy training is a prerequisite for continual learning. In practice, however, data containing new knowledge or capabilities are often off-policy. While methods such as on-policy self-distillation (OPSD) try to bridge this gap by converting off-policy data into on-policy signal, they have been shown to cause reasoning collapse. In this paper, we show that off-policy merging beats OPSD for continual learning. We first show that SFT learns a useful signal from new data, but naively applying its update interferes with existing capabilities. We reduce this interference with a simple recipe we term grafting, which changes where the update is learned and how it is applied: (1) learning the update on an earlier donor checkpoint, ideally even before the end of pretraining, and applying the weight update to the post-trained model; (2) scaling the weight update, equivalent to a form of model merging; and (3) optionally, masking the most sensitive update directions when the new data distribution is far from the post-trained model. Across continual learning settings including (1) distilling from expert traces, (2) self-improvement with STaR and Pedagogical RL, and (3) injecting knowledge after pretraining cutoff, grafting Pareto-dominates both SFT and OPSD in new-task and old-task performance, while avoiding expensive on-policy sampling. Therefore, our work challenges on-policy training as a necessity for continual learning on RL-trained models.

1 Introduction

The introduction asks whether continual learning must be on-policy when useful new knowledge and capabilities often arrive in off-policy data. It presents grafting as a way to reduce interference by learning updates on earlier checkpoints, scaling them, and optionally masking sensitive directions; across three settings, it outperforms SFT and OPSD on new and old tasks.

1 Introduction

  • New knowledge and capabilities often arrive in off-policy data, while direct SFT can cause forgetting and fail to generalize.OPSD attempts to convert off-policy data into on-policy updates, but prior work reports reasoning-related failures.Source: §1 · p. 1
  • The paper asks whether on-policy training is necessary for continual learning or whether off-policy updates are being used poorly.Source: §1 · p. 2
  • Model merging shows that SFT learns a useful signal, but applying its update naively interferes with existing capabilities.Merging scales the SFT update without changing its direction.Source: §1 · p. 2
  • Grafting applies a scaled update learned on an earlier donor checkpoint to the post-trained model, optionally masking its most Fisher-sensitive directions.The mask is proposed for cases where the new data distribution is far from the post-trained model.Source: §1 · p. 2
  • Across three off-policy settings, grafting Pareto-dominates SFT and OPSD on new-task and old-task performance while avoiding expensive on-policy sampling.The settings are expert-trace distillation, self-improvement, and knowledge injection.Source: §1 · p. 3
  • The authors conclude that on-policy updates are not a necessary condition for continual learning.They call for more effort on making off-policy training generalize.Source: §1 · p. 3

2 Preliminaries

The preliminaries frame continual learning as jointly improving a target task and retaining or generalizing capabilities outside it. They motivate off-policy data for tasks beyond the model’s current behavior and introduce SFT and OPSD as learning approaches.

2 Preliminaries

  • Continual learning jointly evaluates held-out target-task performance and performance outside the target task, where goals include retention or generalization.Source: §2 · p. 3
  • Off-policy data can supply signal for genuinely new knowledge or hard problems when the model assigns almost no probability to correct outputs and receives no RL training signal.The paper identifies such data as useful in these scenarios.Source: §2 · p. 3
  • Off-policy data consists of input-output pairs whose inputs were not generated under the post-trained policy, including new knowledge, expert demonstrations, environment feedback, and synthetic self-improvement data.Source: §2 · p. 3
  • SFT learns directly from off-policy outputs by maximizing their likelihood, but is associated with memorization and catastrophic forgetting.Source: §2 · p. 3
  • OPSD converts off-policy data into on-policy updates by supervising sampled responses with a solution-conditioned copy of the model.Its teacher is a stop-gradient copy of the student, and the divergence family interpolates between forward and reverse KL.Source: §2 · p. 3

3 Why does SFT interfere with post-trained capabilities?

The section explains SFT interference through implicit task inference: training on narrow distributions can change how a model assigns tasks to inputs. In a two-task mixture-of-experts account, that assignment controls how much new-task learning modifies the expert supporting an existing capability.

3 Why does SFT interfere with post-trained capabilities?

  • In the model, narrow training distributions such as post-training and late pretraining can make models explain new data differently, a phenomenon termed implicit task inference.Source: §3 · p. 3
  • The account models a retained post-trained capability and a capability to learn continually as two tasks in a mixture of linear experts.Source: §3 · p. 4
  • After post-training on task A, the gate is assumed to route more task-B inputs through expert A than before post-training.The routing quantities satisfy 0 ≤ εD < εP < 1, with ε measuring the share routed through expert A.Source: §3 · p. 4
  • Task assignment controls how much fine-tuning on task B modifies the expert supporting task A; under εD < εP, finetuning the pretrained model routes less of that signal through expert A than finetuning the post-trained model.Source: §3 · p. 4

4 Method

Grafting learns an update on an earlier donor checkpoint and applies a scaled version to the post-trained model. It can also project away sensitive directions when the injected data are far from the post-trained distribution.

4 Method

  • Grafting finetunes an earlier donor checkpoint, applies its parameter delta to the post-trained model, and can optionally mask sensitive directions.The recipe forms the donor delta and adds its scaled, optionally projected version to the post-trained parameters.Source: §4 · p. 4
  • The graft scale λ controls update magnitude and can also act as model merging, with moderate values benefiting grafts from hard math data.The merging interpretation draws on interpolating later and earlier checkpoints.Source: §4 · pp. 4–5
  • When injected data are far from the post-trained distribution, grafting can project updates to keep conditional KL divergence over the old domain small.The method uses a quadratic approximation of KL to identify directions to mask.Source: §4 · p. 5
  • The diagonal Fisher approximation ranks update entries by their contribution to model change and masks the top ρ fraction.The full Fisher is infeasible to compute; the paper also proposes a KFAC-based approximation.Source: §4 · p. 5
  • Retention-based Diagonal or KFAC Fisher projections remove interference from donor updates while masking at most 1% of directions.These projections outperform prior Fisher-based merging and improve the old/new-domain performance frontier over unprojected updates.Source: §4 · p. 5

5 Experiments on Learning Hard Math Problems

The experiments evaluate grafting against SFT and OPSD across hard-math learning settings, including teacher-trace distillation and two forms of self-improvement. They find that merging helps, donor choice matters, and grafting compares favorably with the baselines across these settings.

5.1 Experimental setup

  • Experiments use Qwen3-1.7B and 8B thinking and instruct models, with hard math data from teacher traces, rejection sampling, or Pedagogical RL; performance is evaluated on held-out hard problems and three old-task benchmarks.Unless otherwise specified, metrics use 16 samples at temperature 1.0 and top-p 1.0, with up to 32K tokens in thinking mode.Source: §5.1 · pp. 5–6

5.2 Model merging alone improves both old and new tasks

  • Merging improves generalization and erases OPSD’s original edge over SFT; without merging, only grafting with the best hyperparameters outperforms the post-trained model.The study sweeps nine learning rates from 5×10^-7 to 5×10^-5 over five epochs, then sweeps the merging scale.Source: §5.2 · p. 6, Figure 2 · p. 6

5.3 Optimal θdonor is before the end of pretraining

  • Grafting from progressively earlier post-training checkpoints performs better, while earlier pretraining checkpoints clearly outperform the final pretrained checkpoint, with gains that are not strictly monotonic.The donor search moves up to 12k pretraining steps before the final base checkpoint; the authors recommend searching for an early checkpoint that balances the competing factors.Source: §5.3 · pp. 6–7, Figure 3 · p. 7

5.4 Results for distillation with teacher traces

  • Grafting dominates SFT and OPSD in teacher-trace distillation, with the 1.7B result also surpassing the post-trained model and the ordering holding at pass@4 and pass@8.At 8B, grafting and SFT both dominate OPSD; in 8B instruct mode, grafting again dominates SFT and OPSD at pass@1.Source: §5.4 · pp. 7–8

5.5 Self-improvement with rejection sampling (STaR)

  • For STaR, grafting achieves roughly the same new-task performance as SFT while raising old-task generalization by about eight points; it is generally weaker on new-task performance but similar on old-task generalization to teacher-trace distillation.STaR uses up to 32 attempts per problem and keeps one verified trace for each solved problem.Source: §5.5 · p. 8, Figure 5 · p. 8

5.6 Self-improvement with Pedagogical RL

  • With Pedagogical RL, grafting learns the new task better and generalizes to the old task, while SFT loses in both directions.Pedagogical RL trains a copy of the model with access to the reference solution, but not thinking traces, to generate traces for learning.Source: §5.6 · p. 8, Figure 6 · p. 8

6 Experiments on Knowledge Injection

The knowledge-injection experiments use synthetic fictional facts and evaluate learning on unseen paraphrases alongside retention on math benchmarks. Grafting nearly perfectly injects the facts while retaining performance at or above the post-trained model.

6.1 Experimental setup

  • Qwen3-1.7B (thinking) is the post-trained model, and Qwen3-1.7B-Base is the donor.Source: §6.1 · p. 8
  • The knowledge set contains 1,125 atomic facts about 200 invented entities, expressed as question-answer pairs in eight phrasings.Training mixes 9,000 knowledge QA samples with 1,533 AoPS traces; all methods use the same mixture.Source: §6.1 · p. 9
  • New-task accuracy is exact match on held-out paraphrases, while old-task performance averages AIME 2024, AIME 2025, and HMMT 2025.The knowledge examples are decoded greedily for evaluation.Source: Evaluation · p. 9

6.2 Knowledge injection results

  • Grafting nearly perfectly injects fictional knowledge while retaining performance at or above the post-trained model.SFT injects just as well but suffers forgetting, while OPSD retains nearly perfectly but learns less.Source: §6.2 · p. 9

7 Analysis

The analysis finds that protection masking improves the knowledge-injection trade-off, while grafting is faster than OPSD and writes less to sensitive weights than SFT.

Mask ablations

  • The protection-mask ρ sweep Pareto-dominates plain model merging’s λ sweep in the knowledge-injection setting.Fisher merging does not inject knowledge; KFAC masking reaches new-task performance 0.995 and old-task performance 0.390 at ρ=0.001, which diagonal masking needs ρ=0.01 to match.Source: §7 · p. 9, Figure 7 · p. 9

Training latency

  • 10× faster: grafting trains faster than OPSD on Qwen3-8B (thinking), with sampling taking roughly 90% of the OPSD step.The figure compares per-step sampling and weight-update time.Source: §7 · pp. 9–10, Figure 8 · p. 10

Sensitive-weight updates

  • Grafting writes less to the most sensitive weights than SFT for both math and knowledge learning.The top 1% of coordinates by retention Fisher account for nearly 75% of weight changes; the plotted measure is the normalized Fisher-weighted update within those coordinates.Source: §7 · p. 10, Figure 8 · p. 10

8 Related Work

Related work frames grafting against continual-learning methods, OPSD, weight-space merging, and research on declining finetuning plasticity. The paper positions donor selection and sparse Fisher masking as distinctions from prior merging approaches.

Continual learning for post-trained models

  • Continual learning methods address overwriting earlier capabilities, while practical new knowledge, demonstrations, delayed feedback, and synthetic self-improvement data are often off-policy.Prior work links RL’s smaller forgetting to on-policy data, and SFT has been associated with memorization at the cost of generalization (Chen et al., 2025; Chu et al., 2025).Source: §8 · p. 10

On-policy self-distillation and its failure modes

Model merging and weight-space operations

  • Grafting builds on task arithmetic’s use of base-model deltas, adding Fisher masking and donor selection; its retention-only sparse mask is presented as more effective than Fisher merging.The broader literature includes weight interpolation and methods to resolve interference or localize skills in sparse deltas (Wortsman et al., 2022a; Matena and Raffel, 2022; Li et al., 2022; Wortsman et al., 2022b; Ilharco et al., 2022; Yadav et al., 2023; Yu et al., 2024; Ortiz-Jimenez et al., 2023; Panigrahi et al., 2023).Source: §8 · pp. 10–11

Models become harder to finetune

  • Prior work finds that extended pretraining can increase parameter sensitivity and worsen finetuning (Springer et al., 2025); the paper reports earlier pretraining checkpoints are often better grafting donors.The authors distinguish their approach by mitigating harder finetuning through update rules as well as through easier-to-finetune models.Source: §8 · p. 11

9 Conclusion and Future Work

The results suggest continual learning on RL-trained models need not be on-policy: grafting reduces interference from off-policy updates by learning on an earlier donor checkpoint and applying a scaled update, optionally with a sensitivity mask. Across three settings, it Pareto-dominates SFT and OPSD on new- and old-task performance while avoiding expensive on-policy sampling.

Conclusion

  • Across three continual-learning settings, grafting Pareto-dominates SFT and OPSD in new-task and old-task performance while avoiding expensive on-policy sampling.The settings cover expert traces, self-improvement, and knowledge injection.Source: §9 · p. 11

Conclusion and future work

  • Grafting combines an update learned on an earlier donor checkpoint with scaling and optional masking of sensitive directions to reduce interference.Earlier pretraining checkpoints are usually better donors, while selecting the optimal checkpoint and predicting optimal λ and ρ remain future-work directions.Source: §9 · p. 11

A Data

The experiments use held-out math and synthetic-knowledge tasks to evaluate learning from off-policy data. In Pedagogical RL, grafting replaces the SFT stage that trains on synthesized reasoning traces.

A Data

  • The synthetic closed world contains 200 invented entities and 1,125 facts expressed as 9,000 QA pairs, with held-out phrasings and near-zero closed-book exact match.OPSD also receives 200 wiki-style articles for conditioning; all methods train on the QA pairs and 1,533 math traces.Source: A Data · p. 15
  • The math settings mine problems the target model rarely or never solves, then hold out disjoint problems: 111 for Qwen3-1.7B, 100 for Qwen3-8B thinking, and 200 for non-thinking.The corresponding training sets contain 1,533, 985, and 1,163 samples.Source: A Data · p. 15

B Pedagogical RL

  • Pedagogical RL keeps trace synthesis unchanged and replaces its second-stage SFT with grafting on traces synthesized from problems and reference solutions.The 1.7B setting uses 1,533 problem-solution pairs covering 628 unique problems; synthesized traces become the off-policy training data.Source: B Pedagogical RL · p. 16
  • The teacher sees the reference solution and is prompted to solve independently, while the fixed student sees only the problem and scores trace learnability.The teacher is trained to produce correct traces that are plausible under the student, using a learnability score based on the student’s surprise gaps and a correctness reward.Source: B Pedagogical RL · p. 16
  • The teacher’s pedagogical reward combines trace learnability with correctness, and GRPO trains it to maximize that reward.The parameter β controls the balance between mean and largest surprise gaps, while λ sets the penalty strength.Source: B Pedagogical RL · p. 16

C Training

The training section specifies matched SFT and grafting procedures, alongside OPSD’s demonstration-conditioned self-distillation. Evaluation measures held-out new-task performance and performance on three competition-math suites.

C Training

  • Math grafting and direct SFT use identical optimization, differing only in whether the earlier donor checkpoint or post-trained model is updated.The sweep covers nine learning rates and five epochs, with AdamW, global batch 32, and sequence length 18K.Source: C Training · p. 17
  • Knowledge grafting and direct SFT use full fine-tuning at learning rate 2×10−5, with QA samples receiving an empty-think prefix and replay samples standard reasoning format.Both use a cosine schedule, global batch 32, and sequence length 18K.Source: C Training · p. 17
  • OPSD samples a response conditioned on a demonstration and trains against a frozen demonstration-conditioned teacher using divergence Dα.It uses rollout n=1, top-k 100 teacher logits, and reports α ∈{0, 0.5, 1}.Source: C Training · p. 17

D Evaluation

  • New-task performance uses a held-out set, while old-task performance averages AIME 2024, AIME 2025, and HMMT 2025 scores.The old-task measure reflects generalization for math learning and retention for knowledge injection.Source: D Evaluation · p. 17

E OLMo-3 experiments

The OLMo-3 experiments vary the donor and post-training checkpoints and evaluate on mined-hard math problems. The Fisher-estimation section describes approximations for scoring and projecting sensitive update directions, with limitations for coordinate-based projections.

E OLMo-3 experiments

  • OLMo-3 experiments train on mined-hard traces and hold out 100 problems; donor checkpoints range from the released base model to earlier pretraining stages, alongside four post-training checkpoints.Stage-2 checkpoints are adapted to 18,432-token training by copying stage-3’s YaRN block.Source: Data · p. 17, Training · p. 17

F Fisher estimation

  • The diagonal Fisher approximation scores donor-update entries, while an eigenbasis expresses sensitivity scores along Fisher directions and projects away the top-ρ fraction.The projection is exact only when the chosen basis diagonalizes the retention Fisher.Source: F Fisher estimation · pp. 17–18
  • Coordinate-basis scores can miss important off-diagonal interactions between tightly coupled parameters, while exact full-Fisher eigendecomposition is infeasible.The text notes that coordinate basis is often suboptimal for projection.Source: F Fisher estimation · p. 18
  • For matrix-shaped weights, KFAC approximates Fisher blocks using Kronecker factors, yielding scores over paired factor eigenvectors as well as scalar and vector parameters.The stated approximation does not capture dependencies between different parameter blocks.Source: F Fisher estimation · p. 18
  • The retention-Fisher estimate uses 1,858 rollouts of the post-trained Qwen3-1.7B thinking model, with the ρ threshold set from a subsample quantile.Diagonal projection uses a mask; KFAC adds layer-wise eigendecompositions and matrix multiplications.Source: F Fisher estimation · p. 18

Related papers

Loading 2610.05872v1…