PaperlayerView exact revision on arXiv

Source-linked AI summary

Are Full Rollouts Necessary for On-Policy Distillation?

Yaocheng Zhang, Jiajun Chai, Yuqian Fu, Songjun Tu, Xiaohan Wang, Wei Lin, Guojun Yin, Qichao Zhang, Yuanheng Zhu, Dongbin Zhao

arXiv:2605.31490v2cs.CL

TL;DR

Standard OPD generates costly full rollouts, and teacher feedback at late positions may become unreliable, particularly early in training. The paper introduces progressive and permanently truncated horizon strategies; experiments report improved efficiency and comparable training gains with shorter rollouts.

  • Problem

    Standard OPD generates costly full rollouts, and late-position teacher feedback may be unreliable, especially early in training.

    Source: §1 · p. 1

  • Method

    POPD gradually expands the rollout horizon, while TOPD permanently distills from truncated rollout segments.

    Source: §1 · p. 2

  • Results

    POPD improves training efficiency by up to 3×, while TOPD recovers standard OPD training gains with 10% of the rollout horizon and reduces training time by up to 82%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments suggest full rollouts are not always necessary for OPD, and horizon control offers a practical way to improve efficiency in long-horizon policy learning.

    Source: §7 · p. 9

    The work studies two simple horizon-control strategies; adaptive mechanisms based on signals such as teacher confidence or student entropy remain a future direction.

    Source: §8 · p. 10

Abstract

from arXiv · show

On-policy distillation (OPD) provides dense teacher feedback along student-generated rollouts rather than fixed teacher traces and has emerged as a promising post-training paradigm. However, standard OPD typically generates full rollouts during training, which is computationally expensive and may expose the student to unreliable teacher feedback at late rollout positions, especially during early training. We identify the rollout horizon as a key bottleneck in OPD that substantially impacts training efficiency. Unlike Reinforcement Learning with Verifiable Rewards (RLVR), OPD does not require a final answer reward to provide learning signals. Therefore, full rollouts may not always be necessary for OPD. Motivated by this insight, we propose two simple horizon-control strategies: Progressive OPD (POPD), which gradually expands the rollout horizon during training, and Truncated OPD (TOPD), which permanently performs distillation on reliable truncated rollouts. Experiments on mathematical reasoning show that POPD improves the training efficiency of OPD by up to 3$\times$, while TOPD matches OPD performance using only 10\% of the rollout horizon, leading to substantial wall-clock and memory reductions. These results demonstrate that controlling the rollout horizon offers a simple and practical path to more efficient OPD.

1 Introduction

Standard OPD is costly because it generates full rollouts, and late-position teacher feedback may become unreliable, especially early in training. The paper proposes POPD and TOPD to control rollout horizons; experiments report improved efficiency and show that truncated segments can provide useful supervision.

1 Introduction

  • Full rollouts require substantial decoding time, KV-cache memory, and log-probability computation, while late-position feedback may become noisy or misleading as early-prefix deviations accumulate.Sequence-level OPD can propagate unreliable late-position signals backward into gradients for earlier tokens.Source: §1 · p. 1
  • Token-level OPD avoids directly propagating noisy future signals to earlier-token gradients, but still generates full rollouts and distills all positions from the start.This leaves computation spent on late tokens whose supervision may be unreliable, particularly early in training.Source: §1 · p. 2
  • POPD starts with a short rollout horizon and gradually expands it, whereas TOPD permanently distills only truncated rollout segments.The rationale is that OPD can learn from truncated segments because it does not require a complete rollout or final-answer reward to provide token-level learning signals.Source: §1 · p. 2, Figure 2 · p. 3
  • POPD improves OPD training efficiency by up to 3×, while TOPD recovers standard OPD training gains with 10% of the rollout horizon and reduces training time by up to 82%.The authors also report that earlier rollout segments provide more effective supervision signals.Source: §1 · p. 2

2 Preliminaries

The preliminaries define OPD using student-generated responses and token-level signals, then distinguish sequence-level, discounted, and token-level formulations.

2 Preliminaries

  • Given prompt x, the student policy generates response y, and the setup defines a log-ratio signal r_t and student score function s_t at each position.The log-ratio compares teacher and student policies on the student-generated token; the score function gives the gradient direction for increasing that token’s probability.Source: §2 · p. 3
  • Sequence-level OPD expresses the gradient using future log-ratio signals, while past log-ratio terms contribute zero in expectation for the current token.The formulation is rewritten using autoregressive factorization.Source: §2 · p. 3
  • Discounted OPD applies a factor γ to future signals, while token-level OPD is the γ = 0 case that aligns each token only with teacher feedback at the same position.The paper motivates token-level OPD as avoiding direct propagation of future log-ratio signals to earlier tokens; longer future horizons can accumulate noisy or unreliable signals.Source: §2 · p. 3

3 Long-Horizon OPD Is Inefficient

The section argues that teacher feedback can become unreliable later in student rollouts, and that sequence-level OPD can propagate this noise to earlier tokens. It recommends shortening the log-ratio horizon and controlling the rollout horizon.

3.1 Degraded Teacher Reliability

  • In the navigation task, teacher guidance is reliable near its high-probability trajectory but becomes noisier as states move farther away.Source: §3.1 · p. 4, Figure 4 · p. 4

3.2 Accumulation of Future Noise in Sequence-Level OPD

  • When γ = 1 and noise grows with rollout position, sequence-level advantage error grows as O(T^3) at t = 1, with earlier tokens accumulating more future noise.Source: §3.2 · p. 5

3.3 Design Principles

  • Efficient OPD should shorten the log-ratio horizon with token-level feedback and control the rollout horizon, since token-level OPD still generates full rollouts from the beginning.Source: §3.3 · p. 5

4 Methods

The methods control how much of each student-generated rollout is used for distillation. POPD expands the horizon during training, while TOPD keeps it truncated.

4.1 Progressive On-Policy Distillation

  • POPD starts with short rollout segments and gradually expands the horizon, first aligning the student on reliable prefixes before longer trajectories.The schedule increases the horizon by ΔH every Δk training steps, beginning at H0.Source: §4.1 · p. 5

4.2 Truncated On-Policy Distillation

  • TOPD generates and distills only the first H tokens of a rollout, rather than using all positions from 1 to T.OPD can learn from truncated trajectories without a complete rollout or final-answer reward.Source: §4.2 · p. 5
  • When H = ρTmax, TOPD reduces rollout-generation and training cost approximately in proportion to ρ.Figure 5 compares OPD and POPD training efficiency for an R1-Distill-1.5B student and JustRL-R1-1.5B teacher.Source: §4.2 · p. 5, Figure 5 · p. 6

5 Experiments

On mathematical reasoning, TOPD offers a cost–performance trade-off, while POPD reaches strong performance faster. In an autoregressive control task, POPD also speeds learning, but TOPD can miss critical later-state supervision.

5.1 LLM Reasoning Experiments

  • For TOPD, ρ = 25% improves AIME24 accuracy from 49.3 to 51.3 while reducing training time from 29.7h to 10.2h; even ρ = 10% retains most performance gain with 5.3h training.Accuracy is reported as avg@16 averaged over the last 10 evaluations during training.Source: §5.1 · p. 6, Table 1 · p. 6
  • POPD reaches strong AIME24 performance much faster than standard OPD by beginning with shorter rollouts and expanding the horizon during training.This reduces unnecessary long-horizon generation while preserving useful token-level supervision.Source: §5.1 · p. 6

5.2 Analysis on Autoregressive Control Task

  • In the autoregressive control task, POPD reaches a high success rate substantially faster than standard OPD.The task requires decisions based on historical velocity sequences.Source: §5.2 · pp. 6–7
  • TOPD is less effective in the navigation control task because truncation removes critical later states needed to bypass obstacles and reach the target.The authors relate this boundary to control tasks with state-action coupling and phase-dependent behavior.Source: §5.2 · p. 7
  • The authors conclude that POPD can access full rollouts eventually, whereas TOPD suits tasks whose target teacher characteristics are horizon-independent and present in truncated rollouts.Source: §5.2 · p. 7

6 Why Training on Truncated Rollouts Is Effective?

The analyses indicate that TOPD’s gains come from distillation updates, not simply from teacher-generated prefixes. They also find that early and middle rollout segments provide stronger supervision than later segments, whose teacher feedback becomes less reliable.

6.1 Prefix-continuation Analysis

  • Teacher-generated prefixes alone yield only modest improvement, far less than TOPD training on rollouts of the same ratio.After TOPD training, performance is the same with or without teacher-generated prefixes, supporting the conclusion that prefix-triggered sampling paths do not explain TOPD’s effectiveness.Source: §6.1 · pp. 7–8

6.2 Analysis of Reverse Distillation

  • Reverse distillation on truncated rollouts pulls the stronger JustRL-R1-1.5B student toward the weaker R1-Distill-1.5B teacher, showing that truncated-rollout updates can reshape the student policy.The analysis attributes the transfer to early segments containing problem decomposition, intermediate setup, and initial solution direction.Source: §6.2 · pp. 8–9, Figure 11 · p. 8

6.3 Where Do Useful Distillation Signals Come From?

  • Early and middle rollout segments provide more effective supervision than later segments, whose distillation performance can fall below the initial policy.The segment study distills from one 10% rollout segment at a time.Source: §6.3 · p. 9, Figure 12 · p. 9
  • Teacher–student KL divergence increases deeper in the rollout, consistent with late segments providing less reliable feedback.The analysis argues that full-rollout OPD aggregates gradients from late positions, while TOPD focuses on reliable prefixes to reduce cost and improve distillation quality.Source: Figure 13 · p. 9, §6.3 · p. 9

7 Conclusion

The paper studies whether OPD needs full rollouts and proposes horizon control through progressively expanded or permanently truncated rollouts. Experiments report efficiency gains, while the discussion identifies evaluation of incomplete RLVR rollouts as an open challenge.

7 Conclusion

  • POPD and TOPD control rollout horizons; experiments on LLM reasoning and autoregressive control report improved training efficiency and a favorable cost–performance trade-off.The conclusion states that full rollouts are not always necessary for OPD.Source: §7 · p. 9
  • Truncated RLVR rollouts may not reach a verifiable final answer, making evaluation a central challenge.The discussion identifies reliable credit assignment or proxy evaluation as possible needs for evaluating such segments.Source: §7 · p. 9
  • Horizon control may also improve RLVR efficiency, where long-horizon rollouts are often expensive and noisy.Source: §7 · p. 10

8 Limitations

The paper presents two simple horizon-control strategies and frames adaptive control as future work. It also reports related work, implementation settings, and analysis of noise accumulation in sequence-level OPD.

8 Limitations

  • The study validates two simple horizon-control strategies; adaptive horizons based on KL divergence, entropy, teacher confidence, or signal variance are future directions.Source: §8 · p. 10

A Related Work

  • Unlike prior OPD efficiency work on parameter dynamics, this study examines rollout-horizon control as an orthogonal approach.The related-work passage also describes OPD as training on student-generated prefixes, unlike offline distillation on fixed traces.Source: A Related Work · p. 12

B Implementation

  • LLM reasoning uses ΔH=9 and Δk=1, while autoregressive navigation uses ΔH=1 and Δk=2; the reasoning runs use a 15360-token maximum response length.Source: B Implementation · p. 12
  • Implementation tables report OPD hyperparameters and the MLP and Transformer policy-network architectures for the navigation tasks.The simple and autoregressive navigation tasks use maximum rollout horizons of 50.Source: Table 3 · p. 12, Table 4 · p. 12, B Implementation · p. 12

C.1 Noise Accumulation

  • Early-token gradients in sequence-level OPD can accumulate substantial future noise in long-horizon rollouts.Source: §C.1 · p. 13

D Autoregressive Navigation Task

The autoregressive navigation task uses a teacher and an independently initialized student with distinct targets. Its state includes task, velocity-history, and temporal information, and its action specifies two-dimensional acceleration.

D Autoregressive Navigation Task

E.1 Sequence-Level OPD Under Optimal Teacher Policy

Sequence-level OPD can be sensitive to long horizons when student–teacher mismatch makes future log-ratio signals unreliable. With an optimal teacher, that failure mode is largely mitigated, although accumulated signals can still slow convergence relative to token-level OPD.

E.1 Sequence-Level OPD Under Optimal Teacher Policy

  • Under large policy mismatch, unreliable future signals can make sequence-level OPD substantially worse than token-level OPD.With small mismatch, future signals remain useful and both variants work well.Source: §E.1 · p. 13
  • An optimal teacher largely removes the systematic bias from accumulated future signals, but their variance remains and can make sequence-level convergence slightly slower than token-level convergence.Source: §E.1 · pp. 13–14
  • The TOPD memory estimate uses (P + ρT)/(P + T), with P=1024 and T=15360; unchanged prompt tokens make total length-dependent memory exceed ρ.Source: Table 6 · p. 14

F Memory Analysis of TOPD

TOPD reduces response-length-dependent memory by truncating rollouts, while static memory remains unchanged. Consequently, peak-memory savings are smaller than the ideal length-dependent reduction.

F Memory Analysis of TOPD

  • Peak memory includes unchanged model, gradient, and optimizer memory plus length-dependent KV cache, log-probability buffers, and activations.The length-dependent terms scale with effective sequence length.Source: F Memory Analysis of TOPD · p. 14
  • TOPD uses an effective sequence length of P + ρT instead of OPD’s P + T, distilling only the first H response tokens.H = ρT; P is prompt length and T is standard OPD’s maximum response length.Source: F Memory Analysis of TOPD · p. 14
  • TOPD reduces response-part memory approximately in proportion to truncation ratio ρ, but unchanged static memory makes peak-memory reduction smaller.The length-independent memory term remains the same between OPD and TOPD.Source: F Memory Analysis of TOPD · pp. 14–15
Loading 2605.31490v2…