PaperlayerView exact revision on arXiv

Source-linked AI summary

Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective

Han Cui, Jianhao Yan, Yun Luo, Hongbo Zhang, Zhizhang Fu, Yue Zhang

arXiv:2610.03185v1cs.AIcs.CL

TL;DR

The paper examines what OPD learns and why it can collapse into long, repetitive generation. It frames OPD as teacher-derived implicit reward on student responses, finding improved sampling without observed capability expansion and reward hacking when preferences misalign with quality. It also reports that masking pathological rollouts and SFT initialization mitigate collapse.

  • Problem

    It remains unclear what OPD learns during training and why it can sometimes collapse, despite reported performance gains and observed unstable or degenerate behaviors.

    Source: §1 · p. 2

  • Method

    The paper views OPD as an RL-like process in which the teacher implicitly rewards student responses, and studies how that feedback relates to successful training and collapse.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    On competition mathematics, OPD improves sampling efficiency without expanding the observed solvable set, while collapsed students increasingly fit teacher preferences as generation quality deteriorates.

    Source: §1 · p. 2, §7 · pp. 9–10

  • Takeaways & Limitations

    Masking truncated responses and initializing from an SFT checkpoint mitigate the observed collapse while keeping the teacher fixed.

    Source: §7 · p. 10

    Experiments use small models and mathematical reasoning tasks; whether the findings extend to larger models or other domains remains to be studied.

    Source: Limitations · p. 10

Abstract

from arXiv · show

On-policy distillation (OPD) has become an important approach to language model post-training. However, despite its performance gains, OPD can also collapse into excessively long and repetitive generation, and the mechanism underlying these divergent outcomes remains poorly understood. We explain these outcomes through a reinforcement learning perspective: the teacher implicitly rewards student behaviors, even those it rarely exhibits itself. From this perspective, our experiments show that OPD improves performance without expanding the student's capabilities. When the implicit reward model is reliable, OPD makes correct responses easier to sample. In contrast, when the preference misaligns with quality, reward hacking happens: the implicit reward model amplifies overlong, repetitive student rollouts, even though it rarely generates such text itself. Guided by this diagnosis, we find that masking unhealthy responses during training and using SFT initialization can each effectively mitigate the collapse. Together, these findings show that OPD amplifies student behaviors favored by the teacher's implicit feedback, shifting the focus from how well the teacher generates to how reliably it evaluates student rollouts. Our code is available at https://github.com/HancCui/opd_hacking.

1 Introduction

The paper frames OPD as teacher feedback that reweights student-generated responses, improving sampling when preferences align with quality but producing reward hacking when they do not. It reports that successful OPD makes existing solutions easier to sample, while masking pathological rollouts or SFT initialization mitigates collapse.

Motivation

  • The paper addresses uncertainty about what OPD learns and why training can become unstable or produce excessive length and repetition.OPD supervises student-generated responses through token-level KL divergence at student-generated prefixes.Source: §1 · p. 2

Core perspective

  • The paper interprets OPD as an RL-like process in which the teacher implicitly rewards the student’s own responses.The teacher’s preferences can therefore amplify student behaviors even when the teacher rarely generates them itself.Source: §1 · p. 2

Successful OPD

  • OPD improves sampling efficiency by making correct responses the initial student can already generate easier to elicit, without expanding its observed solvable set.Gains diminish as the sampling budget increases, and problems solved after OPD were also solvable by the initial student with sufficient sampling.Source: §1 · p. 2

Reward hacking

  • In collapsed runs, the student fits teacher preferences while producing increasingly overlong, repetitive responses and declining in generation quality.Both successful and collapsed runs converge normally, so collapse is not simply an optimization or convergence failure.Source: §1 · p. 2

Mitigating collapse

  • Masking rollouts that reach the generation limit and SFT warmup mitigate collapse by limiting reinforcement of pathological responses or improving initial rollouts.With the teacher fixed, masking improves average evaluation accuracy by 3.08 percentage points.Source: §1 · p. 2

2 On-policy distillation as implicit reward optimization

The paper formulates sampled-token OPD as policy optimization with a teacher-derived reward. Student rollouts determine where the teacher supervises, and the experiments compare three teacher–student settings.

2 On-policy distillation as implicit reward optimization

  • Sampled-token reverse-KL OPD can be formulated as policy optimization with a teacher-derived reward, making the teacher’s scoring role explicit.The formulation motivates checking reward reliability on student-generated trajectories.Source: §2 · p. 3

2.1 On-policy teacher supervision

  • At each rollout iteration, the current student generates a response and the teacher evaluates tokens conditioned on the resulting prefixes.The rollout-prefix distribution is held fixed during each optimization step and refreshed through later student rollouts; the student sets where supervision is evaluated, while the teacher supplies target token probabilities.Source: §2.1 · p. 3

2.2 The teacher as an implicit reward model

  • Expanding the reverse-KL objective yields a maximum-entropy policy objective: expected teacher reward with entropy regularization on sampled prefixes.The token-level teacher reward underlies policy-gradient implementations of OPD.Source: §2.2 · p. 3
  • Across AMC23 and AIME24–26, OPD’s PASS@k gains are largest at small sampling budgets and shrink as k increases.AIME curves extend to k = 256, while AMC23 extends to k = 1024.Source: Figure 2 · p. 4

2.3 Experimental settings

  • The experiments use three fixed-teacher settings, with 100 rollout iterations and mean@4 evaluation on AIME24–26.JustRL and DeepScaleR share student initialization and training data; the Qwen3 setting follows a different teacher, student, and prompt configuration.Source: §2.3 · p. 4

3 Does OPD Expand Student’s Capability Boundary?

Across the two successful settings, OPD improves sampling efficiency on problems already solvable by the initial student, but the audited evidence shows no expansion of the observed solvable set.

Gains diminish as the sampling budget grows

  • SOPD outperforms SINIT at small sampling budgets, but SINIT catches up as k grows, and observed coverage does not improve at higher k.The trend is reported for both teachers; the largest tested sampling budgets erase the positive accuracy gains.Source: §3 · pp. 4–5

No observed expansion in problem coverage after OPD

  • After additional SINIT sampling and manual review, no problem remained solvable only by SOPD in the audited pools.The audited SINIT set contained 73 solvable problems, compared with SOPD coverage of 65 under JustRL-1.5B and 63 under DeepScaleR-1.5B-Preview.Source: Figure 3 · p. 5, §3 · p. 5

Already-solvable problems become easier to solve

  • OPD primarily makes correct answers easier to sample on problems the initial student can already solve.Some problems that SINIT rarely solves become much easier, while high-performing problems do not fall to near-zero accuracy.Source: §3 · pp. 4–5
  • 90.6% of already-solvable problems improved in mean@256 with JustRL-1.5B, versus 84.4% with DeepScaleR-1.5B-Preview.The mean@256 increase averaged 22.6% and 14.7%, respectively.Source: Figure 4 · p. 5, §3 · p. 5

4 Reward Hacking in OPD: Convergence to a Biased Reward Signal

The collapsed setting converges normally while the student concentrates on familiar responses. The student learns the teacher’s preferences, but in the Qwen3-4B setting those preferences favor pathological outputs.

4.1 Collapsed OPD Training Still Converges Normally

  • The Qwen3-4B setting’s optimization converges as smoothly as the successful JustRL setting, so the collapse is not marked by abnormal loss or advantage curves.Both students’ responses remain in a low-NLL region under SINIT, indicating familiarity to the initial student.Source: Figure 5 · p. 6, §4.1 · p. 6
  • Min-10NN distance decreases during training in both settings, much more sharply for Qwen3-4B than for JustRL.The measure averages the ten smallest nearest-neighbor distances among responses to the same problem; lower values indicate greater similarity.Source: §4.1 · p. 6, Figure 5 · p. 6

4.2 The Student Faithfully Fits the Reward Signal, Even When the Teacher Is Hacked

  • In the collapsed Qwen3-4B setting, 99.4% of student responses hit the 8k generation limit and 38% show severe repetition, despite only a modest +5.2-point accuracy gain.The teacher has 2.1% truncation and 0% repetition; the successful JustRL student improves accuracy by 16.2 points with no severe repetition detected.Source: Table 2 · p. 6, §4.2 · p. 6
  • The trained students increasingly discriminate teacher-preferred responses, with the selection gap rising from near zero to 0.07 for JustRL and approximately 26 for Qwen3-4B.The large Qwen3-4B gap, together with its endpoint behavior, indicates overfitting to high-advantage SINIT responses that are pathological.Source: Figure 6 · p. 7, §4.2 · p. 7

5 Diagnosing and Mitigating Reward Hacking

The section diagnoses OPD collapse as reward hacking: teacher preferences can favor pathological student rollouts even when they are rare. It tests masking and SFT initialization as ways to mitigate that failure.

5 Diagnosing and Mitigating Reward Hacking

  • The diagnosis follows a reward-hacking strategy: rank SINIT responses by teacher advantage, then test masking pathological responses or reinitializing from a cleaner distribution.The deciles group responses into ten equally sized sets, from lowest advantage (A1) to highest (A10).Source: §5 · p. 7, §5.1 · p. 7

5.1 The Teacher’s Reward Signal Can Be Misaligned with Response Quality

  • In Qwen3-4B, high-advantage responses have near-zero correctness but increasing length, truncation, and repetition; similar failures appear with Qwen3-8B and Qwen3-30B-A3B teachers.These pathological responses are rare under SINIT, yet the teacher assigns them high advantage, causing OPD to amplify them.Source: §5.1 · p. 8

5.2 Candidate-Pool Interventions Mitigate Reward Hacking

  • Masking generation-limit responses recovers from collapse, while SFT warmup avoids the failure; both interventions keep the teacher, data, and optimization recipe fixed.The SFT-initialized student rarely produces the pathological responses favored by the teacher, leaving fewer for OPD to reinforce.Source: §5.2 · pp. 8–9, Figure 8 · p. 8
  • The 4B warmup setting reaches 16.38% average accuracy, 6.22 points above Base-initialized OPD; masking raises average accuracy by 3.08, 0.51, and 2.58 points for 4B, 8B, and 30B-A3B teachers.These are average-accuracy comparisons over AMC23 and AIME24–26; masking’s gains are not uniform across benchmarks.Source: Table 3 · p. 8, §5.2 · p. 9

6 Related Work

Related work frames OPD among distillation and on-policy training methods that rely on student rollouts. This section distinguishes capability transfer, OPD failure modes, and rollout pathologies.

Distillation paradigms and capability transfer

  • Knowledge distillation transfers teacher distributions or sequences, while OPD supervises student-generated trajectories at student-visited states; the cited work reports that OPD mainly redistributes probability over observed solvable behaviors.Extensions broaden supervision through self-distillation, representation-level supervision, multiple teachers, and cross-tokenizer transfer.Source: §6 · p. 9

Failure modes and stabilization of OPD

  • Prior OPD failure studies identify reasoning-pattern mismatch, unreliable prefix guidance, approximate reverse-KL bias, and rollout-signal mismatches including length exploitation.Proposed stabilization approaches include warm starts, data selection, trust regions, prefix correction, clipping, reference regularization, rollout mixing, and length control.Source: §6 · p. 9

Rollout pathologies in on-policy post-training

  • On-policy training inherits pathologies from its sampled trajectories, and OPD adds a teacher-derived signal that can preferentially retain degenerate student behaviors.Related work describes probability redistribution over accessible solution paths, reduced reasoning diversity, and remedies such as format rewards and overlong filtering.Source: §6 · p. 9

7 Conclusion

The paper interprets OPD as teacher feedback that reweights student-accessible responses. Experiments find improved sampling efficiency without expanding the observed solvable set, while identifying a reward-hacking route to collapse and two mitigations.

7 Conclusion

  • Across competition mathematics experiments, OPD improves sampling efficiency without expanding the observed solvable set.Its gains come from making correct responses already accessible to the initial student easier to sample.Source: §7 · pp. 9–10
  • In collapsed training, the student increasingly follows teacher preferences while producing longer, more repetitive responses that receive high teacher advantage before training.The teacher rarely generates these pathological responses itself.Source: §7 · p. 10
  • Masking loss on truncated responses and initializing from an SFT checkpoint mitigate collapse while keeping the teacher fixed.Source: §7 · p. 10

Limitations

The experiments are limited to small models and mathematical reasoning, and finite sampling cannot rule out additional correct responses from the initial model. The interventions target the observed overlong and repetitive generations rather than general OPD failures.

Limitations

  • Computational constraints limit experiments to small models and mathematical reasoning; whether findings extend to larger models or other domains remains untested.Coverage conclusions use finite samples, so larger budgets or different decoding settings might uncover additional correct responses from the initial model.Source: Limitations · p. 10

AI Use Statement

The authors report using generative AI to polish the manuscript, assist with experimental code, and design cartoon characters in the main figure.

AI Use Statement

  • Generative AI assisted with manuscript polishing, portions of the experimental code, and the main-figure cartoon characters.The authors take responsibility for the final manuscript, experimental implementation, and figures.Source: AI Use Statement · p. 10

Ethics Statement

The paper describes its research scope and training settings, then reports that the DeepScaleR temperature pilots motivate a working choice rather than establish an optimum.

Ethics Statement

  • The work studies language-model reasoning and training dynamics using publicly available models and mathematical reasoning datasets, with no experiment-specific ethical concerns identified.The authors note concerns associated with the underlying models and datasets.Source: Ethics Statement · p. 10

A.1 Training Settings

  • The main experiments use three settings; JustRL and DeepScaleR share student initialization and training data, and each run lasts 100 rollout iterations.The settings are summarized in Table 4; training uses 8 responses for each of 32 prompts per iteration and two optimizer updates.Source: §A.1 · p. 16, Table 4 · p. 16

A.2 DeepScaleR Teacher-Temperature Pilot

  • The temperature pilots support 0.75 as a working choice, not an optimum, because the teacher study uses one sample per problem and training curves are single runs with unequal T = 1.5 coverage.The strongest observed training point was at T = 0.75, but the pilots do not establish optimality.Source: §A.2 · pp. 16–17, Table 5 · p. 16, Figure 9 · p. 17

A.3 Evaluation Protocol

The appendix specifies evaluation and audit procedures, prompt formats, and response-concentration measures, while noting limits on interpreting audited coverage and illustrative examples.

A.3 Evaluation Protocol

  • Routine accuracy uses mean@4, while PASS@k and coverage analyses use 256 responses per problem.For the coverage audit, SINIT receives 768 additional responses on selected problems, forming a 1,024-response pool.Source: §A.3 · p. 17
  • The coverage audit extends SINIT sampling only for problems found in the original pools as SOPD-only, making audited totals asymmetric rather than like-for-like.The audit can raise SINIT coverage and lower SOPD coverage; pooling SINIT recoveries across teacher settings further widens that asymmetry.Source: §A.3 · pp. 17–18

A.4 Prompt Templates

  • JustRL and DeepScaleR prompts leave the reasoning block open after <think>, while Qwen3 disables thinking with an empty closed block and retains the step-by-step instruction.Training and evaluation examples contain one user message and no system message.Source: §A.4 · pp. 18–19

B Observed Solvable Sets

  • Observed solvable sets combine initial 256-response pools with targeted SINIT sampling and manual review, but do not establish equal model support or zero SINIT probability for any response.Review excludes invalid derivations and reinstates valid solutions missed by the verifier.Source: B Observed Solvable Sets · pp. 19–20

C Min-10NN Distance

  • Min-10NN measures surface-form concentration with character-level normalized Levenshtein distance; lower values indicate tighter response clusters, not semantic equivalence or quality.Source: C Min-10NN Distance · p. 20

D Response Case Study

  • Figure 10 contrasts teacher free generation and SINIT responses; the teacher-preferred excerpts show repeated reasoning restarts or mechanical repetition, but do not estimate prevalence.The excerpts predate any OPD update.Source: D Response Case Study · p. 20, Figure 10 · p. 21

E High-Advantage Responses from the Initial Student

E.1–E.3 Repetition Examples

  • Examples include repetition to the 8,192-token cap, 6,465-token repetition below the cap, and repeated fragments reaching the cap; these cases had high relative teacher advantage under SINIT.Source: §E.1 · p. 21, §E.2 · p. 22, §E.3 · p. 22

F Training Details for the Candidate-Pool Interventions

The paper details two Qwen3-4B candidate-pool interventions: masking truncated or failed responses during training, and initializing from a released SFT checkpoint.

F Training Details for the Candidate-Pool Interventions

  • The Qwen3-4B baseline and masked settings start from Qwen3-1.7B-Base, while the warmup setting starts from Qwen3-1.7B-SFT.All three use OpenThoughts3 prompts, temperature 1.0, an 8,192-token generation limit, and the same optimization settings.Source: F Training Details for the… · p. 22
  • The masked setting zeros loss masks for truncated or failed responses, so they contribute no policy, reference-KL, or entropy gradient while completed responses remain in training.Truncated or failed responses still contribute to reward and advantage normalization.Source: F Training Details for the… · p. 22
  • The warmup setting loads the released Qwen3-1.7B-SFT checkpoint without an additional SFT stage and changes only student initialization and fixed reference.Teacher, data, optimizer, sampling, and evaluation schedule remain fixed; the intervention tests whether a better initial candidate pool changes what OPD can amplify.Source: F Training Details for the… · p. 22

Related papers

Loading 2610.03185v1…