PaperlayerView exact revision on arXiv

Source-linked AI summary

Denser $\neq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Training

Meng Wang, Haohan Zhao, Wenzhuo Liu, Lu Yang, Geng Liu, Haiyang Guo, Guo-Sen Xie, Gaofeng Meng, Hongbin Liu, Fei Zhu

arXiv:2607.01763v1cs.LGcs.CL

TL;DR

Continual post-training must add capabilities without forgetting prior ones, but it remains unclear whether on-policy self-distillation preserves the retention benefits attributed to on-policy learning. Studying SDPO, the paper finds that it specializes strongly yet forgets more and is less stable than GRPO in continual learning.

  • Problem

    It remains unclear whether on-policy self-distillation inherits on-policy reinforcement learning’s apparent retention benefits during continual post-training.

  • Method

    The paper evaluates SDPO across single-domain and staged continual post-training while separating on-policy data from the objective used to update rollouts.

  • Results

    SDPO specializes strongly but shows greater forgetting, instability, parameter drift, and response-manifold shifts than GRPO in continual post-training.

  • Takeaways & Limitations

    Dense self-distillation is a rapid specialization signal, not an automatically safe stabilizer for continual learning.

  • Takeaways & Limitations

    Dense token-level supervision improves local adaptation but can accumulate drift and interference across tasks, leading to forgetting.

Abstract

from arXiv · show

Continual post-training enables foundation models to acquire new knowledge while preserving existing capabilities. Recent work suggests that on-policy learning can mitigate forgetting, with on-policy self-distillation emerging as a particularly attractive approach. In this work, we revisit this optimistic view through self-distillation policy optimization (SDPO). Our experiments show that SDPO can accelerate in-domain specialization when teacher signals are stable and well aligned, but it struggles to generalize to out-of-distribution scenarios. In continual post-training, SDPO exhibits stronger forgetting and can even collapse, whereas on-policy reinforcement learning methods such as GRPO adapt more conservatively and better preserve prior capabilities. Further analyses reveal that denser self-distillation induces larger drift in both parameter space and response space, and can amplify high-frequency formatting artifacts through a self-reinforcing teacher--student loop. These findings suggest that on-policy data alone is insufficient for continual learning. Dense self-distillation can accelerate specialization when teacher targets are stable and token-level supervision is reliable, but it should not be treated as a default stabilizer for continual post-training. Our code is available at https://github.com/Moenupa/SDPO-CL.

1 INTRODUCTION

The introduction distinguishes on-policy data from the objective used to update models, showing that SDPO can accelerate specialization but is fragile for continual learning and weaker at retention than GRPO. It attributes this trade-off to supervision density, teacher stability, token reliability, drift, interference, and accumulated artifacts.

  • Motivation: On-policy sampling and the training objective are distinct: data histories come from the policy, while the objective determines which continuation distribution is reinforced.This distinction motivates testing whether self-distillation inherits retention benefits associated with on-policy reinforcement learning.
  • Main findings: Denser supervision strengthens the local learning signal while increasing sensitivity, domain mismatch, drift, interference, and the risk of collapse.The study varies supervision density and evaluates specialization, retention, transfer, parameter drift, response drift, and collapse modes.
  • Main findings: SDPO is a strong specialization method but a fragile continual-learning signal, with benefits depending on teacher stability and token reliability.Chain-of-thought supervision and fast teacher updates can amplify noise and formatting artifacts.
  • Main findings: SDPO exhibits weaker retention than GRPO across single-domain and staged continual post-training, with forgetting concentrated on neighboring-yet-misaligned tasks and accumulating across domains.The comparison evaluates whether self-distillation outperforms sequence-level reward optimization in generalization or continual learning.
  • Analysis: The paper diagnoses SDPO’s forgetting through parameter drift, response drift, collapse analysis, and an excess-KL view.These diagnostics connect training behavior to the mechanisms underlying retention failures.

2 PRELIMINARIES

This section introduces GRPO as a critic-free, sequence-level reinforcement-learning method using group-relative rewards, and SDPO as an on-policy self-distillation method providing dense token-level supervision from an informed version of the same model.

  • Group Relative Policy Optimization: GRPO samples multiple responses per prompt, evaluates them with rewards, and normalizes rewards within each group to compute relative advantages.Its sequence-level rewards support critic-free policy optimization while comparing responses generated for the same prompt.
  • Group Relative Policy Optimization: GRPO uses a reference policy for KL regularization and stabilizes reinforcement learning through relative rewards among sampled responses.The method uses importance ratios between current and old policies in its update.
  • Self-Distillation Policy Optimization: SDPO uses the same model as teacher and student, generating dense token-level rewards for each student response.The teacher receives richer context, including solutions or other auxiliary signals.
  • Self-Distillation Policy Optimization: SDPO trains the student to align with the more-informed teacher by minimizing their generalized Jensen–Shannon divergence.Unlike sequence-level reinforcement signals, this produces supervision at every generated token.

3 TOKEN-LEVEL DENSE SUPERVISION IS POWERFUL BUT SENSITIVE

Token-level dense supervision improves efficiency but is sensitive to teacher stability and target reliability. Stable refresh-and-freeze targets mitigate EMA volatility, while CoT distillation helps short structured traces but can harm long or noisy reasoning.

  • 3.1 Token-level reward stability: SDPO gains depend on dense-signal quality because teacher noise, temporal mismatch, and formatting artifacts are reinforced at every generated token.Every token contributes supervision, increasing sample efficiency while repeatedly amplifying unreliable teacher signals within a response.
  • 3.1 Token-level reward stability: No single EMA update rate is uniformly optimal: α = 1% is best overall in MATH-SDPO, α = 5% is worst, and the frozen teacher leads TOOLUSE.Freshness is therefore not sufficient to predict performance; larger α also creates a more rapidly moving teacher target.
  • 3.1 Token-level reward stability: Large EMA rates, especially α = 2% and α = 5%, produce late-stage entropy and JS-divergence increases, while frozen or smaller-rate teachers risk becoming stale.This exposes a stability–freshness dilemma: fast teachers amplify online noise, whereas frozen teachers provide weakening signals as students evolve.
  • 3.1 Token-level reward stability: StableSDPO substantially repairs α = 5%: on AIME, performance rises from 34.38% to 55.00%, and average score rises from 65.34% to 71.84%.Periodic refresh preserves freshness at refresh points while removing step-to-step EMA volatility; hard restart with γ = 1 also raises average score from 71.88% to 72.94%.
  • 3.2 Supervision density: CoT distillation helps TOOLUSE but hurts MATH and SCIENCE because short schema-constrained traces are reliable, whereas long reasoning trajectories contain weakly grounded or noisy intermediate tokens.Rapid teacher updates make long-CoT supervision especially harmful by copying and reinforcing noisy reasoning patterns through a confirmation-bias loop.

4 SDPO TRADES RETENTION FOR SPECIALIZATION

SDPO’s dense token-level supervision accelerates source-domain specialization but does not provide uniform OOD generalization or retention. Across sequential domain shifts, its accumulated drift and interference can cause severe forgetting or collapse, while GRPO adapts more conservatively and preserves capabilities more reliably.

  • OOD generalization: SDPO0 strengthens source-domain specialization, raising AIME from 32.71% to 56.42%, Math500 from 73.85% to 84.29%, and LCBv6 by 13.39%.These gains show that token-level supervision strongly exploits source-like evaluation distributions.
  • OOD generalization: OOD effects follow a non-monotonic interference pattern: source-like and very distant tasks often improve, while partially aligned tasks are most vulnerable.Distillation shifts the model toward the source-domain response manifold, affecting tasks that are close enough to change but misaligned enough not to benefit.
  • OOD generalization: GPQA consistently degrades after SDPO training from multiple source domains, with additional degradation on MATH-to-SCIENCE, MATH-to-ZLogic, and SCIENCE-to-BFCLv4.GPQA may occupy an intermediate interference region: close enough to be affected by source updates, yet insufficiently aligned to benefit.
  • Continual post-training: SDPO0 initially exceeds GRPO, reaching 56.42% on MATH and 65.70% on SCIENCE versus 44.67% and 56.39%, but later drops to 34.38% on MATH and 9.93% on TOOLUSE.After the full sequence, SDPO0 also falls below the base model on LOGIC and KNOWLEDGE.
  • Continual post-training: SDPOCoT0 degrades earlier, SDPO5% collapses after the third stage and remains near zero, whereas GRPO retains capabilities more reliably and improves over the base model across evaluated categories.The earlier SDPOCoT0 degradation is consistent with instability in long-CoT token targets.
  • Continual post-training: Dense token-level updates accelerate local adaptation but accumulate drift, interference, and artifacts across domain shifts, making SDPO not inherently retention-preserving.GRPO’s more selective sequence-level reward signal better supports long-horizon retention.

5 UNDERSTANDING SDPO IN CONTINUAL POST-TRAINING

SDPO adapts more aggressively than GRPO because denser or less stable token-level supervision drives larger parameter and response drift. In continual post-training, teacher-conditioned token projections can amplify formatting artifacts and induce excess drift rather than reliably mitigate forgetting.

  • Parameter and response drift: SDPO produces larger spectral shifts, subspace rotations, and principal-parameter changes than GRPO, with SDPOCoT0 showing the largest subspace change.GRPO’s principal-mask overlap stays near 99.9%, while the ordering SDPOCoT0 > SDPO5% > SDPO0 > GRPO follows supervision aggressiveness.
  • Parameter and response drift: SDPO shifts response-embedding similarities downward and creates heavier lower tails, indicating stronger reshaping of the response manifold than sequence-level GRPO.Response-embedding similarities remain tightly concentrated under GRPO but change more substantially under SDPO.
  • Artifact amplification: SDPO5% collapses by endlessly repeating “\boxed” tokens after a final-answer formatting change in the third TOOLUSE stage.JS divergence plummets during the third stage, reaching zero within the first few TOOLUSE training steps.
  • Artifact amplification: Token-level self-distillation can reinforce frequent formatting artifacts through confirmation bias because the teacher is derived from the student.The student emits an artifact, the teacher supports it token by token, and subsequent updates make the artifact more likely.
  • Teacher projection and excess drift: On-policy sampling does not guarantee forgetting mitigation because SDPO projects tokens toward an augmented-context teacher rather than a KL-minimal sequence-reward policy.The resulting teacher projection can select particular reasoning traces, formatting patterns, or artifacts and induce excess KL drift when it differs from the sequence-reward Razor kernel.

6 CONCLUSION

SDPO can accelerate specialization when teacher signals are stable and aligned, but it is not uniformly beneficial or automatically safe for continual learning. Its token-level supervision can reinforce both useful signals and harmful artifacts, while GRPO preserves capabilities more reliably in these experiments.

  • SDPO is effective for specialization when teacher signals are stable and aligned with the target domain.Its benefits depend on teacher quality, update schedules, and token structure.
  • SDPO is not uniformly beneficial or automatically safe for continual learning because token supervision amplifies useful supervision and harmful artifacts.Each token can reinforce either useful signals or harmful artifacts.
  • Sequence-level methods such as GRPO learn more slowly but preserve capabilities more reliably in the reported experiments.This contrasts with the faster but potentially dangerous specialization signal provided by token supervision.

A MATH NOTATIONS

This section defines divergence measures, performance-change metrics, and parameter-drift diagnostics used throughout the paper. It also introduces overlap and singular-subspace measures for comparing masks and model representations.

  • Kullback–Leibler divergence compares two discrete distributions P(X) and Q(X).
  • Generalized Jensen–Shannon divergence averages KL divergences from P and Q to their mixture, with β = 0.5 recovering standard symmetric JS divergence.
  • Absolute change ∆ and relative change δ measure each training method’s performance against the base model, indexed by method, training domain, and evaluation benchmark.Gain G and forgetting F are defined as the positive and negative parts of relative change, respectively.
  • Normalized spectral shift measures ℓ2-normalized drift in the singular-value profile, with σi(·) denoting the i-th singular value.

B PROOFS

This section proves closed-form and token-level properties of the matched-success Razor policy, establishes teacher projection and excess reverse-KL results, and connects distributional mismatch to additional forgetting. It also gives a forward-KL counterpart for matched success.

  • Matched-success Razor policy: The matched-success Razor policy reweights the base sequence distribution by α/z on successful responses and (1 − α)/(1 − z) otherwise.Here z = P0(S | x), with α as the target success level.
  • Matched-success Razor policy: The unique minimizer preserves the base policy’s conditional distributions within successful and unsuccessful responses while matching success probability α.The proof decomposes Q into αQ+ + (1 − α)Q− and shows the remaining nonnegative terms vanish only when Q+ = p+ and Q− = p−.
  • Token kernel induced by the Razor policy: The Razor policy induces a token kernel based on the base token probabilities multiplied by a ratio of conditional success probabilities.For α → 1, Equation (B.4) gives π0(a | ht) PrP0(y ∈ Sx | hta) / PrP0(y ∈ Sx | ht).
  • Teacher projection and excess reverse KL: For deterministic c and generalized JSD, the population SDPO target equals the teacher token distribution because generalized JSD is nonnegative and vanishes only for identical arguments.Thus the minimizer is B(· | ht) = Tc(· | ht).
  • Teacher projection and excess reverse KL: The excess reverse-KL term is nonnegative and strictly positive whenever the two token kernels differ on histories with positive PB-probability.This follows by combining the autoregressive KL chain rule with the sequence-level reweighting result.
  • Connection to forgetting: Under the KL-forgetting law, teacher mismatch implies additional prior-task forgetting relative to the matched-success Razor policy.The law assumes a strictly increasing ω with ω(0) = 0 for policies matched in new-task success.

C TRAINING DETAILS · C.1 MODELS AND DATASETS

The study uses Qwen3 and Olmo-3 base models trained on four diverse tasks, evaluating averaged accuracy across in-distribution and out-of-distribution test sets. Data sources, training hyperparameters, domain benchmark relationships, and implementation clarifications are provided in the supplementary training details.

  • C.1 MODELS AND DATASETS: The experiments use Qwen3-4B-Instruct-2507, Qwen3-4B-Thinking-2507, Olmo-3-7B-Instruct-DPO, and Olmo-3-7B-Think-DPO as base models.The training framework follows the official SDPO implementation.
  • C.1 MODELS AND DATASETS: Training covers four diverse tasks: MATH, SCIENCE, TOOLUSE, and CODING.These tasks are evaluated with in-distribution and out-of-distribution test sets.
  • C.1 MODELS AND DATASETS: Model performance is measured using averaged accuracy, denoted Acc@k.The metric is applied across the specified task evaluations.
  • C.1 MODELS AND DATASETS: For Instruct models, SDPO and SDPOCoT are equivalent because think tags are absent.In-distribution and out-of-distribution are pragmatic labels rather than strict measures of inter-dataset distance.
  • C.1 MODELS AND DATASETS: Data sources for training and evaluation span multiple domains and are documented in Table C.1.Table C.1 defines AIME as AIME24 and AIME25 combined, and identifies ZLogic and MMLU-R as ZebraLogicBench and MMLU-Redux 2.0.
  • C.1 MODELS AND DATASETS: Benchmark relationships across domains are visualized through embeddings in Figure C.1.The figure provides an embedding visualization of benchmarks by domain.

C.2 DATA SAMPLES · C.3 MODEL CHAT TEMPLATES · C.4 PROMPT TEMPLATES

The appendix provides representative reasoning, coding, science, chemistry, logic, and tool-use samples, together with model-specific chat formats and task-specific prompt templates. It also documents CoT and teacher-EMA effects, including gains for TOOLUSE but impairments for MATH and SCIENCE, with no globally best EMA value.

  • C.2 DATA SAMPLES: The data samples span quantitative word problems, geometry, physics, algebra, chemistry, coordinate conversion, logic puzzles, coding, and tool-use tasks.Examples include a 9-kilometer walking problem, quantum-state resolution, triangle geometry, Eisenstein irreducibility, assignment retrieval, and axolotl-image tools.
  • C.2 DATA SAMPLES: Many reasoning samples explicitly request step-by-step reasoning and require the final answer within \boxed{}.This instruction appears across mathematical, scientific, and chemistry examples.
  • C.2 DATA SAMPLES: Tool-use samples define callable tools, parameter schemas, successful-response formats, and structured action outputs for triangle area, circle area, and axolotl retrieval.The action format uses Thought, Action, and JSON-formatted Action Input fields.
  • C.3 MODEL CHAT TEMPLATES: The model chat templates wrap user prompts with im_start and im_end markers, optionally append an assistant <think> prefix, and provide system messages for function-calling assistants.The OLMo template identifies the model, its November 2024 date cutoff, and the availability of its weights on Hugging Face.
  • C.4 PROMPT TEMPLATES: The MATH, AIME, Math500, SCIENCE, and GPQA prompt template inserts the question and requests step-by-step reasoning with the final answer inside \boxed{}.A variant also supplies a previously correct solution and asks the model to correctly solve the original question.
  • C.4 PROMPT TEMPLATES: The TOOLUSE and BFCLv4 prompt template supplies available tools, prescribes Thought–Action–Action Input formatting, and begins with the task question.Action Input must be realistic and formatted as JSON.
  • C.4 PROMPT TEMPLATES: CoT enhances TOOLUSE training while impairing MATH and SCIENCE, larger teacher EMA α generally worsens CoT performance, and no α is globally best.These conclusions are reported for SDPO performance measured by Acc@8.

D SUPPLEMENTARY RESULTS

Supplementary results reinforce that SDPO’s behavior depends strongly on teacher EMA, target distance, and data shifts. They also provide broader comparisons with GRPO, continual-training diagnostics, and rollout-embedding analyses.

  • Teacher EMA and confirmation bias: The large degradation at α = 1% and 5% further verifies confirmation bias across additional EMA-teacher settings.Table D.1 covers α ∈ {0, 0.5%, 1%, 2%, 5%} across MATH, TOOLUSE, and SCIENCE; Table D.2 extends SDPO versus SDPOCoT evaluations.
  • Single-domain generalization: GRPO outperforms SDPO at generalization in single-domain training on the Olmo-3 series.Table D.3 reports Acc@8 across training domains and evaluation benchmarks.
  • Target distance and forgetting: Intermediate-distance targets exhibit the clearest forgetting, while source-like and very distant targets are often stable or improve.Table D.4 summarizes representative gain, retention, and forgetting cases.
  • Continual-training collapse: Altering data order only partially mitigates collapse, and SDPO still degrades significantly under data shifts during continual training.Table D.5 evaluates SDPO5% across varied data at the second and third stages.
  • Continual-training configurations: The supplementary continual-training comparisons distinguish SDPO0, SDPO5%, SDPOCoT 5%, and S-SDPO0 by teacher EMA, token masking, and restart-and-freeze strategy.Table D.6 reports stage-wise SDPO versus GRPO Acc@8; S-SDPO0 uses γ = 1.

E GENERATION SAMPLES

The generation sample works through an ambiguous walking-time problem by modeling coffee-shop time explicitly, solving for s = 2.5 km/h and t = 24 minutes, and concluding that the requested total time is 204 minutes.

  • Problem setup: The sample interprets each total duration as walking time plus the same t-minute coffee-shop stop.This yields walking times of (240 - t)/60 hours and (144 - t)/60 hours in the two scenarios.
  • Equation formulation: It formulates 9/s = (240 - t)/60 and 9/(s + 2) = (144 - t)/60.The equations represent the 9-kilometer walk under speeds s and s + 2 km/h.
  • Solving the variables: Eliminating the shared terms gives t = 144 - 48s, which leads to 4s^2 + 8s - 45 = 0.The cancellation produces the intermediate relation before the quadratic is solved.
  • Final computation: The positive root is s = 2.5 km/h, implying t = 24 minutes and a requested speed of 3 km/h.At 3 km/h, the 9-kilometer walk takes 180 minutes; adding the 24-minute stop gives 204 minutes.
Loading 2607.01763v1…