PaperlayerView exact revision on arXiv

Source-linked AI summary

On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamics

Julianna Piskorz, Antonin Berthon, Mihaela van der Schaar

arXiv:2609.35259v1cs.LG

TL;DR

Prior SFT–RL comparisons change several training factors at once, leaving the contribution of rollout policy difficult to isolate. The paper varies rollout policy, token-level KL direction, and learning rate in controlled strong-to-weak distillation. It finds that KL direction and learning rate explain more of the observed dynamics, while rollout-policy effects depend on the objective and evaluation setting.

  • Problem

    SFT–RL comparisons vary multiple training factors, making the contribution of rollout policy difficult to isolate.

    Source: §1 · p. 2

  • Method

    The study independently varies rollout policy, token-level KL direction, and learning rate in strong-to-weak distillation, including experiments along a student–teacher rollout-policy spectrum.

    Source: §1 · p. 2

  • Results

    KL direction more clearly shapes task performance, while learning rate governs forgetting and sparsity; forward KL is robust to rollout policy, whereas reverse KL is more sensitive and favors on-policy rollouts.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The results challenge the view that on-policy rollouts are universally preferable and support including cheaper OffPD as a standard baseline in future on-policy distillation work.

    Source: §7 · p. 9

    The controlled experiments cover students of at most 1.5B parameters and reasoning traces of at most 2,000 tokens, with the teacher held fixed.

    Source: §7 · pp. 9–10

Abstract

from arXiv · show

On-policy learning has been argued to reduce catastrophic forgetting, produce sparser parameter updates, and improve generalisation. However, existing comparisons between supervised fine-tuning and reinforcement learning vary many factors simultaneously, making the contribution of rollout policy difficult to isolate. We study the effect of rollout policy in a controlled strong-to-weak distillation setting, by independently varying rollout policy, token-level KL direction, and learning rate across the Llama3 and Qwen2.5 model families and reasoning tasks spanning scientific, medical, and arithmetic domains. Our analysis reveals a nuanced picture of distillation dynamics in which rollout policy does not necessarily play a central role. Instead, token-level KL direction more clearly shapes task performance and output coverage, while learning rate governs forgetting and update sparsity. Analysis of KL gradients and experiments along a continuous student-teacher rollout-policy spectrum explain this pattern: forward KL is remarkably robust to rollout policy, with its performance stable and strong despite changes to the rollout policy, whereas reverse KL is substantially more sensitive and favours student-generated rollouts. On-policy data nevertheless improves generalisation to harder variants of the Countdown arithmetic task under both KL directions, although this advantage does not reliably persist after subsequent RLVR. Our broader conclusions remain robust to removing gradient clipping, using sampled KL estimators, and training on tasks requiring longer reasoning chains. Overall, our results challenge the view that on-policy rollouts are inherently preferable and show that their value depends critically on the objective, evaluation setting, and optimisation hyperparameters.

1 INTRODUCTION

Prior claims for on-policy training are difficult to isolate because comparisons often change several parts of the training recipe at once. In controlled distillation, the paper finds that KL direction and learning rate explain more observed variation than rollout policy, whose value depends on the objective and evaluation.

1 INTRODUCTION

  • Existing SFT–RL comparisons vary objectives, supervision, and optimisation alongside rollout policy, while on-policy methods require continual generation and off-policy methods can reuse data.This makes isolating the rollout policy’s contribution practically important when weighing the additional cost of on-policy training.Source: §1 · pp. 1–2
  • Controlled strong-to-weak distillation finds no consistent on-policy advantage in final in-distribution accuracy, catastrophic forgetting, or parameter-update sparsity.The study varies rollout policy, KL direction, and learning rate independently across two model families and reasoning tasks.Source: §1 · p. 2
  • Forward KL is robust to rollout-policy changes, whereas reverse KL is substantially more sensitive and favors on-policy rollouts.The account combines gradient analysis with experiments spanning a continuous student–teacher rollout-policy spectrum.Source: §1 · p. 2
  • On-policy data improves generalisation to harder task variants under both KL directions, but this advantage does not reliably persist after subsequent RLVR.On-policy rollouts combined with reverse KL also reduce incidental teacher-style transfer.Source: §1 · p. 2
  • KL direction more clearly shapes task performance and output coverage, while learning rate governs forgetting and update sparsity.The paper reports little to no variation in forgetting or update sparsity caused by rollout policy.Source: §1 · p. 2

2 RELATED WORKS

Related work describes strong-to-weak distillation and reported advantages of on-policy training, while noting that many comparisons change several training factors together. This paper tests those reported advantages in a setting that holds the remaining configuration fixed while varying rollout source, KL direction, and learning rate.

Strong-to-Weak and On-Policy Distillation

  • OffPD uses fixed teacher-generated responses, while OnPD trains on student-generated trajectories with token-level teacher supervision.Hybrid variants can interpolate using mixtures of student- and teacher-generated tokens.Source: §2 · p. 3

Claims About On-Policy Post-Training

  • Prior studies have attributed capability preservation, sparser updates, and improved generalisation to on-policy training.Much of the evidence compares SFT with RLVR, public checkpoints, or recipes that vary multiple training factors simultaneously.Source: §2 · p. 3
  • The study tests these claims in strong-to-weak distillation by explicitly varying rollout policy, KL direction, and learning rate.This design addresses confounds including objective, reward signal, supervision density, optimisation scale, gradient clipping, and learning rate.Source: §2 · p. 3

3 BACKGROUND

The section defines distillation as token-level matching of student and teacher distributions over prefixes sampled under a rollout policy. It treats rollout policy and KL direction as separate choices, despite conventional pairings used in some sequence-level and sampled-estimator settings.

Strong-to-Weak Distillation

  • The distillation objective averages the discrepancy between student and teacher next-token distributions over prefixes visited under rollout policy ρ.OffPD uses teacher rollouts; OnPD uses student rollouts, with gradients stopped through the sampling process for each update.Source: §3 · p. 3

KL Divergence as the Distance Function

  • Full-vocabulary KL uses the complete token supervision and avoids the additional variance of sampled-token estimates.This allows rollout policy and KL direction to be varied independently rather than using their conventional pairings.Source: §3 · p. 4
  • Forward KL emphasizes teacher-probable tokens and penalizes missing them, whereas reverse KL emphasizes student-favored tokens and penalizes tokens unlikely under the teacher.The section characterizes forward KL as mode-covering and reverse KL as mode-seeking.Source: §3 · p. 4

Rollout Policy and KL Direction Coupling

  • Rollout policy determines which prefixes are visited, while KL direction determines how distributions are compared at those prefixes.The conventional teacher-rollout/forward-KL and student-rollout/reverse-KL pairings follow sequence-level decompositions or single-sample estimators, but do not constrain the study’s full-vocabulary KL objectives.Source: §3 · p. 4

4 A CONTROLLED COMPARISON OF ON- AND OFF-POLICY DISTILLATION

The controlled comparison varies rollout policy, token-level KL direction, and learning rate while holding other optimisation settings fixed. It finds little consistent benefit from on-policy rollouts: KL direction more clearly shapes task performance, while learning rate is the stronger determinant of forgetting and update sparsity.

4.1 EXPERIMENTAL SETUP

  • The experiment independently varies rollout policy and KL direction at learning rates of 1 × 10^-5 or 5 × 10^-5, using full-vocabulary KL and fixed remaining optimisation settings.It trains Llama-3.2-1B students from Llama-3.1-8B teachers on medical, scientific, and arithmetic reasoning tasks; evaluation covers target-task accuracy, OOD forgetting, and update sparsity.Source: §4.1 · p. 4

4.2 RESULTS

  • On Countdown-3, increasing learning rate reduces update sparsity approximately linearly and consistently worsens OOD performance under both rollout policies.At matched rates, OffPD has less forgetting in nearly every comparison, but this difference is smaller than the learning-rate effect.Source: Figure 2 · p. 5, §4.2 · p. 5
  • Forward KL reaches 71–73% across rollout policies and learning rates, while reverse KL ranges from 35% to 72% and is more learning-rate-sensitive.Best mean target-task accuracies are nearly identical: 72% for OnPD and 73% for OffPD.Source: §4.2 · p. 5, Figure 1 · p. 5
  • OOD performance drops by 11.2–14.0 points at the higher learning rate, compared with changes of at most 1.3 points at the lower rate.Rollout-policy differences are modest; with forward KL, lowering the rate mitigates most forgetting while maintaining comparable in-distribution accuracy.Source: §4.2 · p. 5, Figure 1 · p. 5
  • Update sparsity is 85.3–89.6% at the lower learning rate versus 51.9–60.0% at the higher rate, and OffPD is at least as sparse in matched comparisons.Reverse KL generally produces greater sparsity than forward KL on on-policy rollouts, but the paper finds no on-policy sparsity benefit.Source: §4.2 · p. 5, Figure 1 · p. 5

5 WHEN DOES ROLLOUT POLICY ACTUALLY MATTER?

The effect of rollout policy depends on the KL objective: forward KL is robust across the rollout spectrum, while reverse KL is more sensitive and favors student-generated rollouts. On-policy data improves generalisation to harder Countdown variants, but does not reliably confer an advantage after RLVR.

5.1 LOGIT GRADIENTS REVEAL DIFFERING SENSITIVITY TO THE ROLLOUT POLICY

  • Forward-KL logit gradients are nonzero wherever teacher and student distributions differ, and bounded coordinates yield updates that change proportionally to rollout-prefix distribution distance.Under bounded student-logit Jacobians, the difference between forward-KL updates is bounded linearly by the total-variation distance between prefix distributions.Source: §5.1 · p. 6
  • Reverse-KL gradients depend more strongly on student-supported prefixes, can change arbitrarily with rollout shifts, and may produce sharp, high-variance updates.The student-probability weighting can hinder recovery of teacher modes the student omits, while large log-ratios can destabilise training.Source: §5.1 · p. 6

5.2 EMPIRICAL VALIDATION USING A ROLLOUT-POLICY SPECTRUM

  • Forward-KL held-out accuracy stayed above 80% across the rollout spectrum, varying only 5.2 percentage points at learning rate 1 × 10−5, while reverse-KL performance varied substantially.The spectrum ranges from teacher-favoured to student-favoured rollouts, with λ = −1 corresponding to OffPD and λ = 1 to OnPD.Source: Figure 3 · p. 7, §5.2 · p. 7
  • Rollout policy has comparatively modest effects on catastrophic forgetting and update sparsity, which are influenced more strongly by learning rate.The spectrum experiments find only modest changes in mean OOD performance and sparsity as rollout policy varies.Source: §5.2 · p. 7

5.3 ARE THERE ANY OTHER DIFFERENCES INDUCED BY THE DIRECTION OF KL DIVERGENCE

  • At comparable Countdown-3 pass@1, forward KL yields higher pass@10 than reverse KL, indicating greater in-distribution output coverage.The pattern is reported across the rollout-policy spectrum and also appears in Qwen2.5 results.Source: §5.3 · pp. 7–8
  • On Countdown-4E, on-policy rollouts yield 10–15% higher pass@k accuracy than off-policy rollouts under both KL directions.This harder-task evaluation uses Countdown-3 checkpoints without further training.Source: Figure 5 · p. 8, §5.3 · p. 8
  • After RLVR on Countdown-4, off-policy checkpoints achieve the strongest sustained performance, despite lower initial accuracy; the on-policy generalisation advantage is not reliable.Some reverse-KL checkpoints improve quickly but later undergo reward collapse.Source: Figure 6 · p. 8, §5.3 · p. 8
  • With a Spanish-instructed teacher, reverse-KL OnPD largely preserves English responses, unlike configurations that show near-complete Spanish transfer.The authors hypothesise that reverse KL can match English continuations on student-generated English prefixes without covering Spanish alternatives.Source: §5.3 · p. 8

6 DO THESE FINDINGS GENERALISE BEYOND OUR EXPERIMENTAL DESIGN?

Robustness checks preserve the central pattern: forward KL is more robust to rollout policy, while learning rate remains the stronger determinant of forgetting and update sparsity. A longer-reasoning experiment suggests a possible on-policy benefit but is based on one run per condition.

Sampled rather than full-vocabulary KL

  • With sampled KL estimation, forward KL remains robust to rollout policy, reverse KL remains more sensitive and brittle, and learning rate continues to govern forgetting and update sparsity.The experiments disentangle rollout policy from the distribution used to sample the token-level KL gradient.Source: §6 · p. 9

Removing gradient clipping

  • Without gradient clipping, reverse-KL performance deteriorates substantially while forward KL remains effective at larger learning rates; no consistent OnPD–OffPD gap appears.Forgetting and sparsity remain primarily determined by learning rate, so clipping does not explain the limited rollout-policy effect.Source: §6 · p. 9

Training on datasets requiring longer rollouts

  • On Numina–MATH, lower-learning-rate reverse-KL OnPD reaches the highest MATH-500 accuracy among tested configurations, but the advantage is suggestive because each condition has one run.Forward KL remains more robust, while learning rate continues to govern forgetting and sparsity.Source: §6 · p. 9

Takeaways

  • Across these checks, forward KL remains more robust to rollout policy, and learning rate remains the strongest predictor of forgetting and parameter-update sparsity.The authors conclude that the main pattern is not explained by full-vocabulary KL, gradient clipping, or short rollouts.Source: §6 · p. 9

7 DISCUSSION, LIMITATIONS AND FUTURE WORK

The discussion argues that rollout policy alone does not explain most observed SFT–RL differences, and recommends including OffPD as a baseline in on-policy distillation research. The study is bounded to relatively small students and reasoning traces, with further work needed on longer rollouts and varied teacher–student pairings.

7 DISCUSSION, LIMITATIONS AND FUTURE WORK

  • Because off-policy methods can reuse existing data while on-policy methods require continual generation, the authors recommend OffPD as a standard baseline in future on-policy distillation work.They also call for closer study of learning objectives, reward signals, supervision density, and optimisation procedures when interpreting SFT–RL differences.Source: §7 · p. 9
  • The experiments are limited to student models of at most 1.5B parameters and reasoning traces of at most 2,000 tokens.The authors identify larger models and longer rollouts as future work.Source: §7 · p. 9
  • The teacher is fixed in this study, leaving open whether OnPD–OffPD differences emerge for particular student–teacher combinations.The authors identify such combinations as a future research question.Source: §7 · p. 10

AI USE STATEMENT

  • The authors report using generative AI tools for demonstrations, mathematical-claim assistance, code, plots, and readability edits, with specified checks by authors.They state that these tools were not used to develop theoretical models or conceptual frameworks, formulate hypotheses, or design the methodology.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The reproducibility statement reports matched comparisons using the same training and evaluation pipeline, with differences limited to the factors under investigation.Results are generally averaged over three seeds with SEM; fewer-run experiments, including the long-rollout ablation, are identified.Source: REPRODUCIBILITY STATEMENT · p. 10

APPENDIX CONTENTS

  • Figure 7 shows pre-clipping gradient norms above the 1.0 threshold at every logged step, with reverse KL producing larger, more variable norms than forward KL.Thin lines denote individual seeds and thick lines their mean.Source: Figure 7 · p. 16

A.1 TRAINING WITHOUT, RATHER THAN WITH, GRADIENT CLIPPING

Without clipping, reverse-KL training becomes substantially less stable than forward KL, while learning rate remains the main determinant of forgetting and update sparsity. Sampled-KL training reproduces the full-vocabulary trends.

A.1 TRAINING WITHOUT, RATHER THAN WITH, GRADIENT CLIPPING

  • Learning rate still largely determines forgetting and update sparsity without clipping, though rollout policy adds some variability.OnPD with forward KL has particularly low OOD performance in this ablation.Source: §A.1 · p. 16
  • Without clipping, forward KL remains effective at larger learning rates, while reverse-KL performance deteriorates substantially.The reverse-KL gradient includes an unbounded log-probability ratio, so unusually large token contributions can increase optimisation variance.Source: §A.1 · p. 16, Figure 8 · p. 17

A.2 TRAINING WITH SAMPLED KL, RATHER THAN FULL-VOCABULARY KL

  • Sampled-KL training shows similar final-performance trends to full-vocabulary KL: rollout-policy differences are limited, while forward KL is more robust than reverse KL.Forgetting and update sparsity remain primarily governed by learning rate.Source: Figure 9 · p. 18, §A.2 · p. 18
  • In the sampled estimator, rollout policy sets the prefix distribution, while KL direction selects a teacher-sampled token for forward KL or a student-sampled token for reverse KL.The sampled token need not be the token appearing in the rollout.Source: §A.2 · p. 18

A.3 TRAINING ON A MATHEMATICAL REASONING DATASET REQUIRING LONGER ROLLOUTS

In longer-horizon mathematical distillation, selected checkpoints improve on the untrained MATH-500 baseline, while learning rate chiefly determines forgetting and update sparsity. At the higher learning rate, Countdown performance generally declines, particularly with reverse KL.

A.3 TRAINING ON A MATHEMATICAL REASONING DATASET REQUIRING LONGER ROLLOUTS

  • The reported MATH-500 result is each condition’s maximum along training, and may be optimistic because checkpoint selection and reporting use the same evaluations.The study also reports full trajectories and final-checkpoint performance.Source: §A.3 · p. 19
  • 53.3%–59.0% MATH-500 accuracy was reached by selected checkpoints, versus 53.1% for the untrained baseline.OnPD outperformed OffPD on average in this regime; forward KL remained more robust to rollout policy than reverse KL.Source: §A.3 · p. 19, Figure 10 · p. 20
  • At 1 × 10−5, methods retained 92–93% update sparsity and approximately preserved baseline OOD performance; at 5 × 10−5, mean OOD accuracy fell from approximately 41.4% to 33.7–35.9% and sparsity to 63–70%.Learning rate primarily determined forgetting and update density; most OOD changes occurred early in training.Source: §A.3 · p. 20

A.4 OUTPUT COVERAGE AND GENERALISATION AT THE HIGHER LEARNING RATE

  • At learning rate 5 × 10−5, Countdown performance generally decreases significantly, particularly with reverse KL, making these higher-rate coverage trends less informative.The comparison uses checkpoint 150 and matched seeds; Countdown-4 has four operands rather than three.Source: §A.4 · p. 21

A.5 ADDITIONAL RESULTS ON QWEN2.5-1.5B-INSTRUCT

Across Qwen2.5 results, KL direction has a clearer association with final performance than rollout policy, while learning rate strongly shapes forgetting and update sparsity. The appendix also examines per-benchmark effects, training dynamics, style transfer, and generation-temperature sensitivity.

A.5 ADDITIONAL RESULTS ON QWEN2.5-1.5B-INSTRUCT

  • Across Qwen2.5 settings, rollout policy shows no consistent performance advantage, while forward KL generally outperforms reverse KL; reverse KL degrades under teacher-favoured rollouts.Across the rollout spectrum, forward KL remains comparatively robust, and reverse KL recovers substantially for λ ≥ 0.Source: Figure 13 · p. 22, §A.5 · pp. 22–23

A.6 PER-BENCHMARK ANALYSIS

  • Across individual OOD benchmarks, learning-rate effects exceed OnPD–OffPD differences, which have no consistent direction; higher learning rates strongly reduce performance on MMLU-Pro, HumanEval-Instruct, IFEval, and EQ-Bench.TruthfulQA and BBQ improve slightly for most configurations.Source: Figure 17 · p. 24, §A.6 · p. 24

A.7 SPARSITY ANALYSIS WITH A SMALLER THRESHOLD VALUE

  • Using the smaller sparsity threshold τ = 10−8 does not substantially change the results or their qualitative interpretation.The main-text threshold was τ = 10−6.Source: §A.7 · p. 24

A.8 LEARNING AND FORGETTING CURVES FOR SCIENCE, MEDREASON AND COUNTDOWN

  • For most main runs, target-task performance stabilizes after 150 steps, while most OOD forgetting occurs within the first 50 steps and then remains relatively stable through step 150.The learning curves report target-task and mean OOD accuracy over training.Source: Figure 19 · p. 25

A.9 COMPARISON OF THE LIKELIHOODS OF STUDENT AND TEACHER TRAJECTORIES DURING ONPD AND OFFPD

  • On teacher-generated trajectories, teacher confidence averages about 90%; on student-generated trajectories, student confidence starts near 80% while teacher confidence is near 20%.During forward-KL OnPD, student confidence initially drops sharply, while teacher likelihood on student traces stays relatively low.Source: §A.9 · p. 26

A.10 A LOW FORWARD KL CAN CONCEAL HIGH-ENTROPY ON-POLICY ROLLOUTS

  • During Science OnPD, forward KL can become small while teacher entropy and cross-entropy approach 12 nats, reflecting near-cancellation rather than confident predictions.As training progresses, the low-entropy region advances along the trajectory; at temperature 1.0, sampled accuracy drops sharply despite high accuracy through temperature 0.7.Source: §A.10 · pp. 27–28

A.11 ROLLOUT SOURCE AFFECTS TEACHER-STYLE ADOPTION

  • Teacher style transfers in most setups, but not with OnPD and reverse KL; OffPD still transfers style under reverse KL.The authors attribute reverse KL’s greater robustness to teacher-favoured style tokens contributing little when the student does not favour them.Source: Figure 24 · p. 29, §A.11 · p. 29

A.12 ROLLOUT-POLICY TEMPERATURE ABLATION

  • Forward-KL results at low learning rate changed little under the generation-temperature swap, while reverse KL was more sensitive, with high variability in some comparisons.The ablation changed rollout sampling temperatures while keeping KL-scoring temperatures fixed; best achievable performance was not affected overall.Source: §A.12 · p. 30

B DERIVATION OF THE TOKEN-LEVEL KL GRADIENTS

The appendix derives token-level KL parameter gradients and defines rollout-conditioned distillation updates with gradients stopped through sampled trajectories. It distinguishes teacher-generated off-policy rollouts from student-generated on-policy rollouts.

B DERIVATION OF THE TOKEN-LEVEL KL GRADIENTS

C.1 SETUP

  • The rollout policy ρ determines sampled completions: teacher-generated rollouts are off-policy, while student-generated rollouts are on-policy.Source: §C.1 · p. 32
  • Training stops gradients through sampled completions and their prefixes; the resulting semi-gradient is an ordinary objective gradient only when ρ is independent of θ.Source: §C.1 · p. 32

C.2 A ROLLOUT-STABILITY BOUND FOR FORWARD KL

Forward-KL updates have a rollout-stability bound under a shared Jacobian condition, while reverse KL has no distribution-free counterpart. The appendix also gives a conditional reverse-KL bound and relates matched rollouts to sequence-level KL.

C.2 A ROLLOUT-STABILITY BOUND FOR FORWARD KL

  • Forward-KL expected updates change in proportion to the total-variation distance between rollout-induced completion distributions, under a shared student-logit Jacobian bound.The theorem’s guarantee can be loose when rollout distributions are far apart or the Jacobian bound is large; it controls one-step expected semi-gradients, not final performance.Source: §C.2 · pp. 33–34
  • Forward KL also bounds individual sampled-completion gradient contributions, even when the student assigns very little probability to a teacher-preferred token.This is consistent with forward KL’s similar final performance across the rollout-policy spectrum.Source: §C.2 · p. 34

C.3 REVERSE KL ADMITS NO DISTRIBUTION-FREE ROLLOUT-STABILITY BOUND

  • Reverse KL has no distribution-free rollout-stability bound: an arbitrarily small change in rollout distribution can produce an arbitrarily large update difference, even with strictly positive teacher probabilities.This does not establish that reverse-KL training is always unstable or that either rollout source is universally preferable.Source: §C.3 · pp. 35–36

C.4 A REVERSE-KL BOUND UNDER CONTROLLED LIKELIHOOD RATIOS

  • Bounding student–teacher log-likelihood ratios restores a conditional reverse-KL stability result; extreme ratios at particular prefixes can amplify small rollout-probability changes.The appendix suggests checking whether such extreme ratios occur at prefixes emphasized by a rollout policy.Source: §C.4 · pp. 36–37

C.5 RELATION TO SEQUENCE-LEVEL KL DIVERGENCES

  • Teacher rollouts with forward KL and student rollouts with reverse KL decompose sequence-level KL, but the crossed pairings do not generally do so.With student rollouts, stopped gradients make the training update a semi-gradient; realized-length normalization also means the implemented objective is not generally a constant multiple of sequence-level KL.Source: §C.5 · p. 38

D.1.1 TRAINING DATASETS.

The appendices specify datasets and evaluation protocols, distillation and sparsity methods, a continuous rollout-policy spectrum, and follow-up transfer experiments. They also describe how Countdown-4 RLVR rewards are reported after distillation.

D.1.1 TRAINING DATASETS.

  • Countdown-3, Science, and MedReason use disjoint training and held-out splits; their training/evaluation counts are 4,163/426, 2,674/507, and 4,748/471, respectively.Countdown-3 and MedReason demonstrations were generated with GPT-5 and filtered; Science examples include worked reasoning traces and answer labels.Source: §D.1.1 · p. 39

D.1.2 OOD EVALUATION DATASETS.

  • OOD retention is measured on seven benchmarks spanning knowledge, instruction following and code, and social reasoning and safety, using 3,935 examples and an unweighted mean of task scores.The protocol uses benchmark-standard metrics and a standardized Evaluation Harness interface.Source: §D.1.2 · p. 40

D.2.2 TEACHER MODEL TRAINING

  • Teachers are task-specific, paired with students from the same model family, and selected from SFT or, where used, GRPO-refined checkpoints.Countdown-3 uses the step-500 SFT teacher; Science and MedReason use selected GRPO-refined teachers, generally at step 500.Source: §D.2.2 · pp. 40–41

D.3 TRAINING DETAILS

  • Distillation fully fine-tunes students for 150 steps, varying learning rate and forward or reverse KL while sampling rollouts from either the student or teacher.The main learning rates are 1 × 10−5 and 5 × 10−5; student and teacher temperatures are 1.0 and 0.5, respectively.Source: §D.3 · pp. 41–42

D.4 PARAMETER-UPDATE SPARSITY

  • Update sparsity is the fraction of all model parameters whose absolute change falls below a threshold; higher values mean more parameters remain close to initialization, not necessarily unchanged.The main threshold is τ = 10−6; analysis at τ = 10−8 preserves the main qualitative conclusions.Source: §D.4 · p. 42

D.5 IMPLEMENTATION OF THE ROLLOUT-POLICY SPECTRUM

  • The rollout-policy spectrum interpolates between teacher and student policies, with λ = −1 and λ = 1 recovering those policies and λ = 0 giving their normalized geometric midpoint.For extrapolation, clipping and plausibility masks limit amplification; across λ ∈ [−2, 2], maximum pairwise-odds amplification is threefold.Source: §D.5 · p. 43

D.6 RLVR AFTER DISTILLATION

  • The Countdown-4 follow-up trains 24 distillation-seeded checkpoints with a common RLVR configuration; Figures 6 and 26 plot logged training rewards, not held-out accuracy.Figure 26 covers distillation at learning rate 5 × 10−5, with each curve representing a distillation seed.Source: §D.6 · p. 44, Figure 26 · p. 44

D.7 SPANISH-STYLE TRANSFER EXPERIMENT

  • The Spanish-style transfer experiment conditions a frozen Qwen2.5 teacher to reason in Spanish while the student receives neither the demonstration nor the style instruction.On-policy rollouts come from the student under the vanilla prompt; off-policy rollouts come from the conditioned teacher. Each condition has one training run and one judge.Source: §D.7 · p. 46

E CHECKPOINT SPARSITY ANALYSIS

This section contextualizes checkpoint-sparsity comparisons by examining learning rates used to produce checkpoints in the main RL and explicit SFT comparisons.

E CHECKPOINT SPARSITY ANALYSIS

  • Table 6 covers learning rates for checkpoints in the main RL and explicit SFT comparisons; “From” identifies the checkpoint used as the sparsity reference.Reported rates are for the language model or policy actor, using the peak rate when a schedule is reported.Source: Table 6 · p. 47, E CHECKPOINT SPARSITY ANALYSIS · p. 47
  • 9 × 10−6 (SFT mean) and 5 × 10−6 (SFT median); RL/preference rates average 1.5 × 10−6 and have a median of 5 × 10−7 across nine disclosed-rate checkpoints.The RL/preference calculation excludes DeepSeek-R1-Zero.Source: E CHECKPOINT SPARSITY ANALYSIS · p. 47
  • PRIME's 1 × 10−6 rate is excluded because it belongs to a separately trained implicit reward model, not the evaluated policy checkpoint.The calculation also excludes DeepSeek-R1-Zero.Source: E CHECKPOINT SPARSITY ANALYSIS · p. 47
Loading 2609.35259v1…