PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Properties of Same-Family On-Policy Distillation

Yuntai Bao, Qinfeng Li, Guoqing Jiang, Liwei Chen, Zhiheng Qin, Xuanping Li, Wenqi Zhang, Xuhong Zhang

arXiv:2609.32722v1cs.LG

TL;DR

The paper asks how much and how quickly RL-acquired capability transfers across model scales, and whether OPD outcomes can be predicted before training. It studies OPD across teacher–student scales and finds a regular early transfer phase, with fitted laws relating outcomes to model scale and teacher score.

  • Problem

    How OPD outcomes depend on teacher and student scale remains poorly understood, leaving open whether a run’s performance can be estimated before training.

    Source: §1 · p. 1

  • Method

    The paper studies OPD across weak-to-strong, same-base, and strong-to-weak setups, fitting laws to transfer dynamics and examining objectives, on-policy supervision, and bootstrapping.

    Source: §1 · p. 2

  • Results

    OPD has an early approximately linear useful-transfer phase followed by a noisy tail, and joint power laws predict peak score and, more approximately, useful-transfer slope.

    Source: §8 · p. 10

  • Takeaways & Limitations

    Peak student error is nearly proportional to teacher remaining error, and at matched score smaller teachers transfer better; bootstrapping does not improve on direct transfer from the smallest expert.

    Source: §8 · p. 10

    Held-out gold-score evaluation provides only partial safeguards and does not establish reliability or safety beyond the evaluated setting.

    Source: Ethics statement · p. 11

Abstract

from arXiv · show

*Reinforcement learning (RL)* can induce substantial reasoning capabilities in large language models (LLMs), but how much of this capability transfers across model scales, and how quickly, remains unclear. We study the scaling properties of *on-policy distillation (OPD)* across *weak-to-strong*, *same-base*, and *strong-to-weak* teacher--student setups. We find that early OPD training dynamics uniformly exhibit a regular *useful-transfer* regime, in which held-out accuracy (the *gold score*, $G$) rises approximately linearly in $d=\sqrt{\mathrm{KL}(\pi_\theta \Vert \pi_{\mathrm{ref}})}$, the square root of token-level reverse KL divergence from the student initialization. In every observed weak-to-strong pair, the student's peak gold score exceeds its teacher's own, so a compact RL expert can transfer capability to a much larger student via OPD. To estimate OPD outcomes, we fit *power laws* for how $G_{\mathrm{peak}}$ and the slope of the useful-transfer regime scale with student and teacher parameter counts and with teacher gold score. These laws show that peak gold score improves with teacher scale only up to roughly the student's scale, and that at a matched gold score smaller teachers transfer better, so a teacher's score alone does not define its supervision value. We also study the scaling effects of two OPD variants, bootstrapping weak-to-strong OPD, and the degree of on-policy supervision.

1 Introduction

The introduction asks whether OPD outcomes can be predicted from teacher and student scale, and studies transfer across weak-to-strong, same-base, and strong-to-weak setups. It reports a regular early transfer regime and scaling laws, alongside findings about OPD objectives, on-policy supervision, and bootstrapping.

1 Introduction

  • Early OPD training has a useful-transfer regime where gold score rises approximately linearly with d, followed by noisy dynamics that include attenuated improvement, saturation, and regression.The paper contrasts these later dynamics with the regular overoptimization regime of PPO and direct alignment.Source: §1 · p. 2
  • The study asks whether OPD has a predictable regime, how scale shapes peak score and transfer rate, and how objective choice, on-policy supervision, and bootstrapping affect transfer.It investigates these questions with controlled math-reasoning experiments on Qwen2.5 models from 0.5B to 14B.Source: §1 · p. 2
  • Joint power laws relate peak remaining error and useful-transfer rate to student size, effective teacher size, and measured teacher score; at matched score, smaller teachers transfer better.The peak law extrapolates to the largest held-out scales within one accuracy point.Source: §1 · p. 2
  • Delta-OPD has a larger local matched-KL gain slope in 15 shared pairs and a larger observed peak gain in 12, with the advantage concentrated in weak-to-strong pairs.The passage also reports that an off-policy SFT phase harms weak-to-strong OPD increasingly with the capability gap.Source: §1 · p. 2
  • Every observed weak-to-strong student peaks above its teacher, while bootstrapped chains peak below direct transfer from the smallest expert at the same student scale.Intermediate teachers in the bootstrapped chains attain higher gold scores.Source: §1 · p. 3

2 Preliminaries

The preliminaries formalize three distillation objectives. Vanilla-OPD uses teacher supervision on student rollouts, Delta-OPD derives rewards from the teacher’s RL-induced policy shift, and OffPD trains on teacher-sampled rollouts.

2 Preliminaries

  • Vanilla-OPD minimizes sequence-level reverse KL between the student policy and a fixed teacher, using student rollouts for token-level teacher supervision.The exact policy gradient uses return-to-go advantages, while modern implementations use a zero-discount update with immediate token rewards [Lu and Thinking Machines Lab, 2025].Source: §2 · p. 3
  • Delta-OPD constructs token rewards from the policy shift the teacher acquired during RL, representing an alternative objective across weak-to-strong, same-base, and strong-to-weak pairs.The paper presents it as a common core of OPD2, Direct-OPD, and W2S-OPD [Yu et al., 2026].Source: §2 · p. 3
  • Off-policy distillation (OffPD) samples rollouts from the teacher and trains the student to maximize their likelihood, as in SFT on teacher demonstrations [Hinton et al., 2015].Source: §2 · p. 3

3 Experimental design

The experiments use Qwen2.5 Base models and a shared training and evaluation setup for math reasoning. Gold score is held-out accuracy, while the square root of token-mean reverse KL serves as a proxy for training progress.

3 Experimental design

  • The study uses Qwen2.5 Base models at 0.5B, 1.5B, 3B, 7B, and 14B parameters, with an initial SFT phase on Dolci-SFT data.The models are base rather than instruction-tuned models.Source: §3 · pp. 3–4
  • The design includes 25 teacher–student combinations across weak-to-strong, same-base, and strong-to-weak setups, with RL and OPD using the same training prompts.Teachers are obtained by GRPO RL on 14.8K mixed GSM8K and MATH training examples; evaluation uses the corresponding 6.3K-example test split.Source: §3 · p. 4
  • Gold score is accuracy on a held-out test set used to evaluate both RL teachers and OPD students.Source: §3 · p. 4
  • The study uses the square root of token-mean reverse KL, estimated with k3 and averaged over response tokens, as a proxy for training progress.The token-mean measure matches the immediate-token objective.Source: §3 · p. 4

4 Characterizing capability-transfer dynamics in OPD

Across Vanilla-OPD runs, gold score initially rises approximately linearly with d, then follows heterogeneous tails. Peak score generally improves with teacher scale for larger students, but not consistently for smaller students.

4 Characterizing capability-transfer dynamics in OPD

  • Across 25 runs, gold score initially rises approximately linearly in d, then tails vary among attenuated improvement, saturation, and regression.Initial linear fits over the first 30 checkpoints have R2 of 0.932–0.988; fitted slopes span 0.175–0.721.Source: §4 · pp. 4–5
  • Local KL geometry explains the initial linearity: gold score changes to first order in the parameter perturbation while KL changes to second order.The slope depends on the Fisher-normalized alignment of the update direction with the gold-score gradient.Source: §4 · p. 5

4.1 How scale shapes transfer and later dynamics

  • For 7B and 14B students, peak gold score rises with teacher scale, while the 0.5B and 1.5B students lose accuracy under their largest teachers.The 0.5B student peaks at 40.8% with a 3B teacher, versus 39.0% with 7B and 37.7% with 14B.Source: §4.1 · p. 5, §4 · p. 5
  • Vanilla-OPD same-base peaks fall within 0.5 percentage points of final direct-RL references at all five scales, exceeding them at three.Peak locations are less monotone than peak values, so the analysis tracks peak value, initial rate, and transfer extent rather than fitting peak location.Source: §4.1 · p. 5
  • Where gold score regresses, the teacher-induced proxy score keeps rising, a pattern the paper identifies as implicit-reward overoptimization.The proxy is logged as token-mean reward on student training rollouts.Source: §4.1 · p. 5
  • The useful-transfer endpoint is the last d before three consecutive checkpoints fall below the initial fit’s 95% predictive band; nine of 25 trajectories depart within observed support.Changing the confidence level and persistence requirement shifts the departure count from eight to twelve.Source: §4.1 · p. 6

5 Fitting power laws for peak gold score and useful-transfer rate

The fitted laws relate peak gold score and initial transfer rate to student scale, capped teacher scale, and teacher competence. They capture peak outcomes more reliably than transfer rates, while transfer extent is treated as an approximately scale-free KL budget.

5 Fitting power laws for peak gold score and useful-transfer rate

  • Teacher scale and competence are collinear among RL-endpoint teachers, but bootstrapped OPD-product teachers reduce that collinearity and identify the teacher exponents.Adding five Vanilla-OPD and three Delta-OPD cells changes the correlation from −0.9999 to −0.95 and −0.97, respectively.Source: §5 · p. 6
  • Every observed weak-to-strong student peaks above its teacher’s score; at matched score, smaller teachers transfer better, while teacher scale is capped at student scale.An out-of-sample 7B-student test found the score-66.0 intermediate 3B checkpoint peaked at 73.8, below the 1.5B teacher’s 77.5; the joint law predicted 74.1 and the correct ordering.Source: §5 · pp. 6–7, Table 3 · p. 7
  • Conditioning peak laws on teacher score halves leave-one-scale-out RMSE versus the scale-only baseline, and predicts held-out largest scales within 0.7 Vanilla-OPD and 0.4 Delta-OPD accuracy points.The joint law extrapolates to the largest held-out student and teacher scales.Source: §5 · p. 7, Table 2 · p. 7
  • The useful-transfer rate law has stronger score dependence than the peak law, but its noisier fits are better read as an interpretable scale summary than a precise predictor.At matched score, the larger teacher transfers more slowly.Source: §5 · p. 7
  • Transfer extent spans d=0.20–0.36 for Vanilla-OPD and d=0.27–0.34 for Delta-OPD, with medians 0.30 and 0.29; it is treated as an approximately scale-free KL budget.Scale dependence is weak, and Delta-OPD exponents are not identifiable.Source: §5 · p. 7

6 Effects of design choices on OPD transfer

The design-choice experiments compare Delta-OPD, different degrees of on-policy supervision, and bootstrapped weak-to-strong chains. Delta-OPD often improves peaks, pure OPD leads the supervision comparison, and bootstrapping does not beat direct transfer from the original expert.

6.1 Effect of OPD variant: Delta-OPD

  • Delta-OPD has steeper slopes than Vanilla-OPD in 15 of 17 shared cells and larger peak gains in 12, with the clearest benefit under the 0.5B teacher.That teacher yields two to four extra absolute peak points; larger teachers stay within about one point. The exceptions in slope are the two smallest students under the 0.5B teacher.Source: §6.1 · p. 8, Figure 7 · p. 8

6.2 Degree of on-policy supervision

  • Pure OPD attains the best peak in every cell; an off-policy cold start increasingly harms weak-to-strong transfer, while pure OffPD also underperforms pure OPD.For 3B, 7B, and 14B students taught by the 0.5B expert, cold-start losses are 6.6, 15.7, and 19.4 points; pure OffPD trails by 28.4 points in the extreme weak-to-strong cell.Source: §6.2 · p. 9, Figure 8 · p. 9

6.3 Bootstrapping weak-to-strong OPD

  • Bootstrapped Vanilla chains peak below direct OPD from the 0.5B expert at 3B, 7B, and 14B; the full Delta-OPD chain is also below direct Delta-OPD at each scale.Vanilla peaks are 60.4 versus 61.4 at 3B, 70.0–71.3 versus 72.7 at 7B, and 76.8 versus 77.7 at 14B; Delta-OPD peaks are 64.5 versus 65.3, 74.0 versus 75.0, and 79.4 versus 79.9. Direct RL remains above every weak-teacher variant, and longer chains end slightly lower than shorter ones.Source: §6.3 · pp. 9–10, Figure 9 · p. 9

7 Related work

Related work covers OPD methods and objectives, proxy-optimization risks, weak-to-strong transfer, and scaling-law approaches. This paper’s cited positioning is to characterize how fixed OPD objectives scale.

On-policy distillation

  • Prior OPD work transfers expertise with student-sampled rollouts and develops variants that reshape token rewards or objectives; related analyses diagnose failure modes and recipes.The paper characterizes how fixed objectives scale, distinguishing its focus from work that designs and diagnoses objectives.Source: §7 · p. 10

Reward model overoptimization

  • Fixed teacher supervision is a proxy-optimization setting: offline teacher data can induce teacher hacking, while noisy-expert theory favors online interaction.The related work connects this risk to analyses of reward-model overoptimization and direct alignment.Source: §7 · p. 10

Weak-to-strong generalization

  • Prior weak-to-strong work reports capability gains from weak supervision, while pretraining distillation finds small teachers can help larger students before gains saturate or reverse.The paper positions its peak scaling law as quantifying this regime through an effective-teacher cap.Source: §7 · p. 10

Predicting model performance with scaling laws

  • Earlier scaling laws predict pretraining loss, fine-tuning, distillation, or RL outcomes; the paper distinguishes its target as KL-indexed trajectories.The cited prior work includes Kaplan et al. (2020), Hoffmann et al. (2022), and Busbridge et al. (2025).Source: §7 · p. 10

8 Conclusion

OPD has a regular early useful-transfer phase followed by a noisy tail, and joint scaling laws predict peak capability from student scale and teacher quality. The conclusion also cautions that held-out gold scores do not establish safety beyond the evaluated setting.

8 Conclusion

  • Joint power laws predict peak gold score and useful-transfer slope; peak student error tracks teacher remaining error, with smaller teachers transferring better at matched score.The laws predict peak gold score more reliably than useful-transfer slope.Source: §8 · p. 10
  • Bootstrapping does not improve on direct transfer from the smallest expert, while OffPD underperforms OPD and an off-policy cold start harms weak-to-strong transfer.Source: §8 · p. 10

Ethics statement

  • Held-out gold-score evaluation provides only partial safeguards and does not establish reliability or safety beyond the evaluated setting.The authors recommend broader bias, safety, robustness, and distribution-shift evaluations before deployment.Source: Ethics statement · p. 11

A Discussion

The discussion argues that OPD's useful-transfer dynamics and peak capability can be estimated from the student–teacher pair, but teacher score alone is insufficient. It also separates predictive scaling claims from variant comparisons and diagnostic interpretations of regression.

A Discussion

  • Every Qwen2.5 trajectory begins with useful transfer, but later checkpoints attenuate, saturate, or regress, and pair-specific reversals make a universal step budget insufficient.Both d_transfer and d_peak are properties of the teacher–student pair.Source: A Discussion · p. 16
  • A higher-scoring teacher can transfer worse: the 3B checkpoint slightly outscoring the 1.5B RL endpoint teaches the 7B student worse, as only the joint law predicts.The 1.5B OPD teacher also outscores the 0.5B RL expert yet teaches the 3B student worse.Source: A Discussion · p. 16
  • Across scale, remaining peak error follows a stable joint power law in student scale and teacher score, while transfer extent stays within a narrow band.The first 30 Vanilla-OPD and first 40 Delta-OPD observations estimate local transfer rate.Source: A Discussion · p. 16
  • Regression can reflect over-imitating a teacher that lacks student capabilities; implicit-reward overoptimization additionally requires fixed-prompt evidence that P rises as G falls.The discussion distinguishes these mechanisms from collapse, an abrupt optimization failure.Source: A Discussion · p. 16
  • Delta-OPD converts token-mean KL into local gold gain faster than Vanilla-OPD in 15 of 17 shared cells and improves peak gain in 12, but more seeds are needed to test stability.The transfer budget is a separate KL allowance; gold measurements indicate whether further optimization brings attenuated gain, saturation, or regression.Source: A Discussion · pp. 16–17
  • If the observed regularity persists at frontier scale, weak-to-strong OPD could elicit capability beyond the supervisor, including with human-level supervision as the fixed weak teacher.The authors connect this possibility to Yang et al. (2026d) and feature-learning theory.Source: A Discussion · p. 17

B Limitations

The paper’s evidence is bounded by its Qwen2.5 math setting, single-run design, and untested transfer beyond the studied family and recipe. Its local analysis explains the leading transfer exponent, while the extent of linearity and later trajectory behavior remain empirical.

B Limitations

  • Evidence covers one Qwen2.5 family, one math mixture, single runs per pair, responses capped at 2K tokens, and intentionally stopped trajectories; transfer to other families, tasks, or recipes is untested.Aggregate evaluation provides neither prompt-bootstrap nor optimization-seed intervals; cell-bootstrap intervals capture variation across scale pairs only.Source: B Limitations · p. 17
  • Several fixed recipe choices may matter, and only one matched-accuracy comparison tests the joint-law ordering; repeating controlled comparisons across the grid remains future work.The listed concerns include rollout length, SFT recipe, teacher checkpoint, KL estimator, validation frequency, and logging convention.Source: B Limitations · p. 17
  • The local expansion explains the leading exponent, but the linear-window extent, later tails, and fitted coefficient power laws remain empirical regularities without a mechanism.The Delta-OPD endpoint law has five observed departures, and its fixed fitting windows are exploratory.Source: B Limitations · p. 17
  • Early termination and finite evaluation frequency can right-censor transfer extent and place the peak at the observation boundary.The paper identifies multiple seeds, denser evaluation, and prompt-level resampling as ways to quantify these uncertainties.Source: B Limitations · p. 17

C Extended related work

  • Unlike compute-coordinate scaling fits, this study tracks dynamics along token-mean KL and compares OPD objectives at matched KL rather than matched compute.The related-work discussion distinguishes on-policy token-level scoring of student rollouts from static labels or demonstration trajectories.Source: C Extended related work · pp. 18–19

D Local geometry of capability transfer

  • For differentiable initial paths, gold score changes to first order while KL changes to second order, yielding a leading linear law in square-root KL.If the initial alignment vanishes, the linear coefficient is zero and the leading change may instead be quadratic; the theorem’s leading-term result does not establish fitted-window or global behavior.Source: §D.1 · p. 19, D Local geometry of capability transfer · p. 19, §D.2 · p. 20

E Hyperparameters

  • All runs use verl v0.8.0, with shared hyperparameter configurations specified for SFT, RL teachers, OPD, and gold-score evaluation.The cited passages identify configuration tables but do not provide their parameter values.Source: E Hyperparameters · p. 21, §E.1 · p. 21, §E.2 · p. 21

F Qwen2.5 math trajectory grid

  • Figures 10 and 11 arrange Qwen2.5 math trajectories with student scales by row and teacher scales by column; their OPD objectives use 30- and 40-observation fit windows, respectively.The grids mark transfer endpoints and gold-score maxima; the Delta-OPD grid omits two strong-to-weak controls.Source: F Qwen2.5 math trajectory grid · p. 22, Figure 10 · p. 23, Figure 11 · p. 24

G Expanded Delta-OPD and RL results

The expanded Delta-OPD results report five observed transfer endpoints, with the remaining endpoints right-censored. Direct-RL local fits closely match early observations, while their later-checkpoint extrapolations become less accurate as improvement diminishes.

G Expanded Delta-OPD and RL results

  • Five Delta-OPD transfer endpoints are observed—four weak-to-strong and one same-base—while the remaining twelve are right-censored.Table 9 reports per-pair results, with peak gains measured relative to each run’s initialization.Source: G Expanded Delta-OPD and RL results · p. 25, Table 9 · p. 25
  • Four-point local fits reach R2 ∈[0.991, 0.998], while later-checkpoint extrapolation RMSE ranges from 0.016 to 0.041 accuracy units and grows with diminishing marginal improvement.Figure 12 shows gold score against token-mean reverse KL and marks the SFT initialization.Source: G Expanded Delta-OPD and RL results · p. 25, Figure 12 · p. 26

H Scaling law estimation and diagnostics

The appendix evaluates scaling laws for peak performance, transfer rate, and transfer extent, including tests at held-out outer scales. It also documents estimator and endpoint sensitivities that qualify those fits.

H Scaling law estimation and diagnostics

I Endpoint and estimator sensitivity

J Extended results for design choices

Extended results examine bootstrapping, on-policy supervision, and cold starts; further analyses identify estimator, optimizer, and modeling choices that affect observed dynamics.

J Extended results for design choices

K Lessons learned and what did not work

  • The k3 estimator gives an accurate reverse-KL value but its policy gradient is forward-KL; the corrected k+3 construction retains the value estimate and backpropagates the k2 gradient.This motivates validating loss-based KL estimators at the gradient level, not only by value agreement.Source: K Lessons learned and what did not work · p. 35
  • Late-stage regression differed between FSDP and Megatron runs with differing optimizer-state precision, prompting fixed-backend numerics and a caution about sensitivity.The authors suspect FP32 follows small late-stage gradients more faithfully while BF16 rounding damps them.Source: K Lessons learned and what did not work · p. 35
  • Quadratic best-of-n and subquadratic RL forms did not fit the useful-transfer regime; the linear law describes that regime, while heterogeneous tails fit neither form.Source: K Lessons learned and what did not work · p. 35
  • Full-vocabulary reverse-KL loss caused earlier and more severe post-peak regression in weak-to-strong Vanilla-OPD than the sampled-token objective.The study therefore uses the sampled-token form throughout.Source: K Lessons learned and what did not work · p. 35
  • Peak location was non-monotone across the grid and resisted identifiable parametric fits, so the maximum’s location is reported as observed.Source: K Lessons learned and what did not work · p. 35

Related papers

Loading 2609.32722v1…