PaperlayerView exact revision on arXiv

Source-linked AI summary

Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals

Lei Zhao, Qichao Zhao, Bowen Zuo, Qishi Zhan

arXiv:2610.11247v1cs.LGcs.CL

TL;DR

The paper asks why OPD loss can plateau before students match their teachers. It analyzes OPD as a continuous-time dynamical system and examines training-log diagnostics. Larger-scale teachers show weaker loss reduction than self-RL teachers, while a local recovery guarantee applies under specific proximity and parameterization conditions.

  • Problem

    Prior explanations do not fully clarify why OPD loss keeps decreasing with some teachers but plateaus early with others.

    Source: §1 · p. 2

  • Method

    The paper models OPD in the small-learning-rate limit as a continuous-time dynamical system, analyzes loss dynamics, and measures a training-log learning-signal proxy.

    Source: §5.1 · pp. 7–8, §5.2 · p. 8

  • Results

    25.1% average final loss reduction with larger-scale teachers compares with 96.2% for self-RL teachers after 200 updates.

    Source: §4.3 · p. 5

  • Takeaways & Limitations

    A local recovery guarantee covers sufficiently close teachers in a shared parameterization, conditionally explaining the experimental success of self-RL teachers.

    Source: §5.3 · p. 10

    The local guarantee applies to sufficiently close teachers in a shared parameterization; larger-scale teachers with different parameter dimensions are not covered.

    Source: §5.3 · p. 10

Abstract

from arXiv · show

On-policy distillation (OPD) enables effective capability transfer between language models, yet the mechanisms underlying its failures are not fully understood. Across code generation and mathematical reasoning, OPD with larger-scale teachers exhibits early loss plateaus, with an average final loss reduction of 25.1% after 200 updates, compared with 96.2% for self-RL teachers, obtained by further reinforcement learning (RL) training of the initial student. To understand this difference, we analyze OPD as an idealized continuous-time dynamical system in the small-learning-rate limit. Our training-log diagnostics associate these plateaus with an early decline in a gradient-based learning-signal proxy while substantial loss remains; these measurements do not establish why the underlying gradient weakens. We further prove a local recovery guarantee for teachers sufficiently close to the initial student in a shared parameterization under regularity conditions, offering a conditional explanation for the success of self-RL teachers in our experiments. Across runs with and without loss plateaus, we observe small relative parameter changes (0.025-0.098%) and high similarity between the student's representations before and after OPD (linear CKA $>0.98$ across layers). These observations suggest that limited representation adaptation may contribute to learning-signal collapse, a hypothesis that remains to be tested. Code is available at https://github.com/leizhao7/opd-learning-signals.

1 Introduction

OPD trains students on their own rollouts, but stronger teachers do not consistently yield effective transfer. The paper examines why loss plateaus occur and presents empirical diagnostics, a conditional recovery guarantee, and evidence motivating a limited-adaptation hypothesis.

1 Introduction

  • OPD matches teacher next-token distributions on student-generated responses, providing dense supervision on prefixes the student visits and reducing train–inference distribution mismatch.Despite this promise, studies report cases where OPD does not improve student performance even when the teacher substantially outperforms it.Source: §1 · p. 1
  • With larger-scale teachers, OPD exhibits early loss plateaus and limited performance gains, while self-RL teachers support sustained loss reduction and substantial student improvements.Self-RL teachers are obtained by further RL training of the initial student.Source: §1 · p. 2
  • Prior explanations include teacher–student incompatibility, unreliable guidance, sampled-token supervision limits, and misalignment with task success, but do not fully explain why loss plateaus with some teachers.The unresolved question is why OPD loss stops falling before the student matches the teacher.Source: §1 · p. 2
  • The paper diagnoses an early decline in a training-log proxy for the learning signal while substantial distillation loss remains, termed premature learning-signal collapse.This is a proxy-based diagnosis, not an established explanation for why the underlying gradient weakens.Source: §1 · p. 2
  • A local recovery guarantee applies to teachers sufficiently close to the initial student in a shared parameterization, while small parameter changes and high representation similarity motivate a limited-adaptation hypothesis.The observations do not establish that limited adaptation causes the plateaus.Source: §1 · p. 2

2 Related Work

Related work describes OPD methods and several proposed explanations for their inconsistent performance. This paper places its analysis alongside work on post-training dynamics and model geometry.

2 Related Work

  • OPD methods train on student rollouts: GKD matches next-token distributions, while MiniLLM minimizes sequence-level reverse KL using policy gradients.The review situates OPD within on-policy imitation learning and language-model post-training [Ross et al., 2011; Lin et al., 2020; Agarwal et al., 2024; Ko et al., 2024].Source: §2 · p. 2
  • Prior work links OPD failures to teacher–student alignment, token-level supervision, task-success misalignment, and small or progressively slower learning signals.The reviewed findings motivate a population-level analysis of loss evolution as both the student and its rollout distribution change.Source: §2 · p. 3
  • Prior studies report limited model changes and preserved representations during post-training, while this paper examines teacher proximity under changing rollout distributions.It compares parameter and representation changes under OPD and RL post-training.Source: §2 · p. 3

3 Preliminaries

The preliminaries define OPD as matching teacher and student distributions on prefixes sampled from the student’s own responses. They also describe a top-K approximation used in practice.

3 Preliminaries

  • A response is a sequence of tokens; at position t, the prefix state contains the prompt and earlier response tokens, and the action is the next token.The student policy factorizes response probability into next-token probabilities.Source: §3.1 · p. 3
  • OPD minimizes reverse KL between student and teacher next-token distributions on prefixes sampled from the student’s own responses.Prefixes are formed by sampling a prompt and response, then choosing a response position uniformly.Source: §3.1 · p. 3
  • In practice, the objective retains the K tokens with highest student probabilities at each prefix and renormalizes the student distribution over them.The full-vocabulary KL requires teacher log-probabilities for every token, whereas the approximation drops the tail.Source: §3.1 · p. 4

4 Early Loss Plateaus in On-Policy Distillation

Across code generation and mathematical reasoning, OPD with larger-scale teachers plateaus while self-RL teachers support sustained loss reduction. The experiments compare seven student–teacher pairs and evaluate performance over 200 updates.

4.1 Experimental Setup

  • The seven main OPD runs cover code generation and mathematical reasoning, using self-RL teachers made by further RL of the initial student and larger-scale teachers with more parameters.The experiments use Eurus-2-RL-Data for code and DAPO-Math-17K for mathematics; the main runs train for 200 steps.Source: §4 · p. 4, §4.1 · pp. 4–5

4.2 Evaluation Metrics

  • Task-level validation accuracy is the unweighted mean of three benchmark accuracies; accuracy gain is the final-minus-start accuracy, and gap recovery is the percentage of the initial teacher–student gap closed.Code uses avg@4 and mathematics uses avg@8; each benchmark contributes equally to task-level accuracy.Source: §4.1 · p. 5, §4.2 · p. 5

4.3 Experimental Results

  • Self-RL runs reached maximum observed loss reductions of 99.5% and 96.3%, closing 77.4% and 70.7% of the accuracy gap, respectively.The values correspond respectively to Qwen3-4B-RL-Code and JustRL-1.5B.Source: §4.3 · p. 5
  • SFT initialization did not resolve limited loss reduction or improve final accuracy over direct OPD in the two tested settings.After SFT followed by 200 OPD updates, final accuracies were 40.2% and 22.6%, compared with 42.7% and 22.7% for direct OPD.Source: §4.3 · pp. 5–6
  • 25.1% average final loss reduction with larger-scale teachers compares with 96.2% for self-RL teachers after 200 updates.With larger-scale teachers, students recovered only 13.0%–47.5% of the teacher–student performance gap; with self-RL teachers, the loss continued to decrease.Source: §4.3 · p. 5, Figure 1 · p. 6
  • Top-16 versus full-logit OPD and one versus eight rollouts per prompt showed similar training behavior and closely matched final results in the tested math setting.The ablations used the Qwen3-1.7B (Non-thinking) student.Source: §4.3 · p. 6

5 Vanishing Learning Signals: Loss Dynamics of OPD

Section 5 analyzes OPD loss dynamics in a continuous-time framework, using training-log diagnostics to examine why plateaus arise and proving a conditional local recovery guarantee. The diagnostics associate early proxy decline with plateaus, while the guarantee applies to sufficiently close teachers sharing the student’s parameterization.

5.1 Mean-Flow Formulation and Loss Dynamics

  • In the small-learning-rate limit, OPD loss decay factors as γ = µ(1 + α), where a weak learning signal or occupancy opposition can yield a small decay rate.Here µ measures learning signal relative to remaining loss; α measures occupancy-gradient alignment, and α ≤ −1 means loss no longer decreases even if g ≠ 0.Source: §5.1 · pp. 7–8

5.2 Premature Learning-Signal Collapse with Larger-Scale Teachers

  • For larger-scale teachers, the logged learning-signal proxy drops by approximately an order of magnitude while over 60% of loss remains; self-RL loss decreases by more than 90%.The proxy uses a noisy, pre-clipping minibatch gradient, so its decline alone does not establish a comparable decline in the underlying gradient.Source: §5.2 · p. 8, Figure 2 · p. 9
  • Occupancy estimates do not consistently indicate cancellation: 1 + α is positive at 19/20 Qwen3-14B checkpoints, 12/20 Skywork-7B checkpoints, and 13/20 R1-Distill-7B checkpoints.The estimates are based on only 20 checkpoints per teacher, while cross-fitted ∥g∥² declines in math but stays roughly stable for Qwen3-14B after step 20.Source: §5.2 · p. 8, Table 3 · p. 9

5.3 Local Convergence with Self-RL Teachers

  • Under regularity conditions, OPD stays within 2ρ of a sufficiently close teacher for Tloc = Ω(1/ρ), with a recovery bound for every cutoff δ.The result covers teachers in the student’s parameter space, not the larger-scale teachers with different parameter dimensions; the experiments do not directly test its spectrum-dependent bound.Source: §5.3 · p. 10
  • With self-RL teachers, the minimum loss reaches 0.5% and 3.7% of its starting value in the two experiments.The guarantee’s residual depends on E∆(δ) and higher-order errors in ρ.Source: §5.3 · p. 10

6 Parameter and Representation Changes in OPD

Across eight OPD runs, parameter and representation changes were small, but the size of the parameter displacement alone did not distinguish successful from stalled runs. The section also compares these changes with those from self-RL teacher training.

6 Parameter and Representation Changes in OPD

  • Relative parameter change measures movement from the starting checkpoint, while linear CKA measures hidden-representation change.The CKA comparison uses starting and final hidden states for prompts from the evaluation benchmarks.Source: §6 · p. 11
  • 0.025%–0.098% relative parameter change and at least 0.985 layerwise CKA were observed across eight OPD runs after 200 updates.These measurements do not distinguish successful from stalled runs by displacement magnitude alone.Source: §6 · p. 11, Table 4 · p. 11
  • 0.030%–0.50% relative distance separates self-RL teachers from their base models, with minimum CKA between 0.967 and 0.9998.These changes are comparable to or moderately larger than OPD’s, up to about 5× for JustRL; distance alone does not verify the recovery analysis assumptions.Source: Table 4 · p. 11, §6 · p. 12

7 Conclusion

The conclusion links early learning-signal decline with remaining loss in OPD runs using larger-scale teachers, while self-RL teachers support sustained reduction. The appendix derives a continuous-time view of OPD and notes that its idealization differs from the experiments.

7 Conclusion

  • Across code generation and mathematical reasoning, training logs show an early decline in a gradient-based learning-signal proxy while substantial loss remains.Self-RL teachers support sustained loss reduction; limited adaptation is proposed as a possible contributor, not established as the cause.Source: §7 · p. 12

A.1 Per-prefix reverse-KL gradient

  • At a fixed prefix, the forward-KL gradient is the negative teacher-policy expectation of the student score.The passage contrasts this fixed-prefix forward-KL gradient with the per-prefix loss discussion.Source: §A.1 · p. 17

A.2 Population gradient split

  • The population-gradient derivation uses the likelihood-ratio identity for student-generated trajectories.The resulting split is substituted into the loss–update alignment.Source: §A.2 · p. 17

A.3 From the OPD update to the ODE

  • As the learning rate tends to zero, the stochastic OPD iteration yields the continuous-time flow under the stated limiting steps.The derivation treats bounded-variance minibatch noise as negligible in the limit.Source: §A.3 · p. 17
  • The analysis idealizes AdamW updates as detached-gradient descent and uses full-vocabulary KL, while the main runs use top-16 KL.Momentum, preconditioning, weight decay, and gradient clipping can alter the update; measured statistics are proxies for full-KL quantities.Source: §A.3 · p. 18

A.4 Proof of Proposition 5.1

Under differentiation and flow-regularity conditions, the proof relates OPD loss decrease to loss–update alignment. It also gives an integrated loss relation and bounds the alignment using gradient and occupancy terms.

A.4 Proof of Proposition 5.1

  • The rate identity follows by the chain rule along the OPD flow, with loss derivative equal to the negative loss–update alignment.The proof assumes differentiation and expectation can be interchanged and that the flow remains in a regularity neighborhood.Source: §A.4 · p. 18
  • Integrating the rate identity expresses positive loss at time t as its initial value multiplied by the exponential of negative twice the integrated learning rate.The integration applies before the loss first vanishes.Source: §A.4 · p. 18
  • The learning rate γ is bounded by µ(1 − ||h||/||g||) ≤ γ ≤ µ(1 + ||h||/||g||).The bounds follow from the factorization and Cauchy–Schwarz.Source: §A.4 · p. 18

B Parameter differences and the initial Fisher

This appendix develops local identities and bounds around a nearby teacher in a shared parameterization. Under regularity conditions, it bounds approximation errors and establishes an interval over which the OPD mean flow remains local.

B Parameter differences and the initial Fisher

  • The teacher and student share architecture, parameter coordinates, tokenizer, vocabulary, temperature, and policy interface.The teacher is set to the self-RL checkpoint, and the teacher Fisher is compared with the initial-student Fisher.Source: B Parameter differences and the… · p. 19
  • The low-curvature energy bound does not require the teacher–student displacement to lie in the range of the initial Fisher.The retained eigenvalues in [0, δ] contribute energy bounded by δ ||Δ||^2 / 2.Source: B Parameter differences and the… · p. 19
  • At the teacher parameter, loss, its gradient, and the detached update vanish; the local Hessian is tied to the teacher Fisher.The proof uses the vanishing conditional KL and its first derivative at the teacher.Source: B Parameter differences and the… · p. 19
  • Under bounded-score and sampling conditions, the detached update differs from the teacher-Fisher linearization by at most Cg ||e||^2.The lemma assumes a bounded number of student-policy draws and fixed stopping and masking rules.Source: B Parameter differences and the… · p. 20
  • The bounded-draw conditions do not cover parameter-dependent top-k or top-p truncation or unbounded retries without separate control.The trajectory law must still obey the stated draw bound.Source: B Parameter differences and the… · p. 21
  • The local approximation proposition bounds loss and update errors on a ball around the teacher, with the Fisher difference controlled by teacher–student distance.The uniform estimates combine the update remainder with the difference between teacher and initial-student Fishers.Source: B Parameter differences and the… · pp. 21–22
  • For positive teacher–student distance, the exact OPD mean flow exists and stays within the ball of radius 2ρ through Tloc.When Cg = 0, the distance is nonincreasing; if ρ = 0, the flow is stationary at the teacher.Source: B Parameter differences and the… · p. 22

C Recovery bound from the parameter difference

The paper proves a conditional local recovery guarantee for the nonlinear OPD mean flow when the teacher is sufficiently close to the initial student. Its supporting local analysis uses Fisher-based expansions and requires control of the teacher–initial Fisher difference over a validity interval.

C Recovery bound from the parameter difference

  • The recovery bound is local around θT = θRL, with radius ρ = ∥θT − θ0∥ and endpoint-dependent error terms that shrink as ρ → 0.The abridged bound gives εL(ρ) = O(ρ^3) and εg(ρ) = O(ρ^2).Source: C Recovery bound from the parameter… · pp. 23–24
  • Under the stated regularity assumptions, the nonlinear OPD mean flow reaches loss at most ε by tε and maintains it through T*, when tε ≤ T*.The guarantee applies over finite times bounded by the local validity horizon Tloc.Source: C Recovery bound from the parameter… · p. 23

D Local expansion and spectral specialization

  • Near the teacher, the loss gradient and detached update share a leading Fisher-linear term, while their difference cancels linear terms.At the same-model root, the full-logit forward KL has the same leading Hessian.Source: D Local expansion and spectral… · p. 25
  • Using initial Fisher H0 instead of teacher Fisher HT requires controlling ∥HT − H0∥ and the Taylor remainder, and estimates hold only within the local validity interval.The interval is supplied for these estimates along the original nonlinear OPD flow.Source: D Local expansion and spectral… · p. 25

D.1 Fixed-Fisher specialization

The fixed-Fisher specialization describes loss decay in an exact quadratic model through the initial Fisher spectrum. It shows that the direction of the teacher–student parameter difference affects relative convergence, even when its magnitude is fixed.

D.1 Fixed-Fisher specialization

  • A decline in loss need not indicate failure to fit the teacher when the remaining loss is already small, because higher-curvature components decay first.This qualification concerns the exact quadratic specialization.Source: §D.1 · p. 25
  • If the spectral residual E∆(δ) ≤ ε/2 and the quadratic model remains valid, loss reaches at most ε after t ≥ (2δ)^−1 log max{1, 2L(θ0)/ε}.The residual also satisfies E∆(δ) ≤ δ∥∆∥²/2.Source: §D.1 · p. 25
  • Equal parameter distances can have different relative decay rates: for H0 = diag(1, η), the two example directions decay as e^−2t and e^−2ηt.Thus distance magnitude alone does not specify a uniform relative convergence rate.Source: §D.1 · p. 26

E Experimental details for the teacher comparisons

The appendix documents the teacher comparisons, training and validation protocols, and diagnostics used to examine OPD loss and learning-signal behavior. It also qualifies what these single-run experiments and finite-sample diagnostics can establish.

E Experimental details for the teacher comparisons

E.1 Additional Training-Dynamics Diagnostics

  • The logged pre-clipping gradient norm is distinct from the loss-normalized learning-signal proxy, and the ratio is an empirical diagnostic rather than a direct population loss-derivative measurement.A small gradient norm alone cannot distinguish successful matching from a plateau; remaining loss must also be considered.Source: §E.1 · p. 32, Figure 4 · p. 33
  • Occupancy-factor estimates fluctuate and can become negative even for the self-RL run, so finite-sample ratios do not establish the population loss-derivative sign at every update.The estimates are not confidence intervals or deterministic predictions of the subsequent AdamW trajectory.Source: §E.1 · p. 33, Figure 5 · p. 34

F Parameter and Representation Measurement Details

The appendices report endpoint parameter and representation measurements alongside instruction-following and SFT-then-OPD experiments. They show contrasting teacher-dependent outcomes, while noting limits on generalization and on interpreting representation similarity.

F Parameter and Representation Measurement Details

  • 0.098% was the JustRL-1.5B OPD student's parameter movement, versus 0.50% teacher distance, even as OPD reduced loss to 3.7% of its starting value.Parameter comparisons use exported checkpoints and count tied embedding/head weights once.Source: F Parameter and Representation… · p. 35
  • High CKA reflects pooled prompt-representation geometry, not equality of individual activations, and does not establish a fixed neural tangent kernel.CKA is invariant to orthogonal transformations and isotropic scaling.Source: F Parameter and Representation… · p. 36
  • 0.9850 was the minimum CKA for the Qwen3-14B Code OPD endpoint, compared with above 0.9978 for Qwen3-4B-RL-Code.JustRL-1.5B's minimum was 0.9674 against its initial student, versus 0.9904 for its OPD endpoint; these are endpoint comparisons, not a time series.Source: F Parameter and Representation… · p. 36, Figure 6 · p. 37

G OPD on Instruction Following

H.1 SFT on Teacher Responses and Subsequent OPD

  • The held-out SFT response-row split can share questions with training, so its cross-entropy is a fitting diagnostic rather than an independent test of mathematical generalization.Mathematical capability is evaluated separately on benchmarks.Source: §H.1 · p. 42
  • Four SFT response datasets had training cross-entropies that fell from their first logged values at update 5 to their last at update 295.The final held-out losses at update 297 were 0.2942, 0.3537, 0.3252, and 0.2332 for JustRL-1.5B, R1-Distill-7B, Qwen3-4B-RL-Math, and Qwen3-4B, respectively; these are not OPD objectives.Source: §H.1 · p. 43, Figure 8 · p. 43
  • 54.1% was reached by SFT with JustRL-1.5B, while SFT→OPD ended at 40.2% with R1-Distill-7B and 22.6% with Qwen3-4B, below direct OPD's 42.7% and 22.7%.The single-run accuracy comparisons do not establish statistical significance.Source: §H.1 · pp. 44–45, Table H.4 · p. 45
  • 37.1% and 23.0% were the maximum OPD loss reductions after SFT for the R1 and Qwen continuations; Qwen's final loss exceeded its starting OPD loss.These findings concern the tested data, recipe, and budgets, not other SFT initializations.Source: §H.1 · p. 45, Table H.5 · p. 45

H.2 Vocabulary support and training rollouts

Vocabulary-support and rollout comparisons largely preserve teacher-dependent outcomes in the tested Qwen and R1 runs. More rollouts do not uniformly improve loss trajectories, and each condition is a single stochastic run.

H.2 Vocabulary support and training rollouts

  • Eight rollouts increased sampled responses per update from 64 to 512 while keeping prompt and optimizer-update counts fixed, so this was not a fixed-generation-cost comparison.Each condition is a single stochastic run without paired- or repeated-seed estimates.Source: §H.2 · p. 47
  • 43.94%, 46.36%, and 45.21% were Qwen3-1.7B's final accuracies with Qwen3-4B-RL-Math across top-16/one-rollout, full-logit/one-rollout, and top-16/eight-rollout settings.The corresponding final loss reductions were 55.37%, 54.85%, and 51.29%; with Qwen3-4B, accuracies were 22.74%, 22.88%, and 24.87%, with reductions of 6.24%, 11.60%, and 9.89%.Source: §H.2 · p. 48, Table H.6 · p. 48, Figure 10 · p. 49
  • Removing the vocabulary restriction or increasing rollouts preserved the broad teacher-dependent behavior in the Qwen runs, but the R1 eight-rollout loss rose late despite its high accuracy.The late increase is established by logs, which do not identify its cause.Source: §H.2 · pp. 48–49
  • 51.88%, 50.46%, and 51.93% were R1-Distill-1.5B's final accuracies with JustRL-1.5B across the same three configurations, exceeding results with R1-Distill-7B.With eight rollouts, JustRL's final loss reduction was 31.09%, versus 92.95% and 91.80% in the one-rollout top-16 and full-logit runs; the R1-Distill-7B full-logit loss ended 7.05% above update 1.Source: §H.2 · p. 48, Table H.6 · p. 48, Figure 11 · p. 50
Loading 2610.11247v1…