PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation

Yuchen Cai, Ding Cao, Liang Lin, Chunxi Luo, Xin Xu, Kai Yang, Weijie Liu, Saiyong Yang, Tianxiang Zhao, Guangzhong Sun, Guiquan Liu, Junfeng Fang

arXiv:2605.11739v3cs.CL

TL;DR

The paper examines why OPD reaches strong performance early, focusing on unresolved questions about update directions and magnitudes. It analyzes OPD’s parameter dynamics and proposes EffOPD, achieving average 3× training acceleration while maintaining comparable final performance.

  • Problem

    The source of OPD’s remaining early-checkpoint performance gap—direction formation or update magnitude—requires clarification.

  • Method

    The paper analyzes OPD’s module allocation and update-direction dynamics, then introduces EffOPD, which extrapolates along current update directions.

  • Results

    3× average training acceleration over multiple baselines is achieved by EffOPD while maintaining comparable final performance.

  • Takeaways & Limitations

    OPD’s efficiency is associated with early directional stabilization and compact parameter allocation, offering a perspective for understanding and accelerating post-training.

  • Takeaways & Limitations

    The findings’ applicability to multi-turn agent tasks and multimodal reasoning remains unexamined, while the theoretical analysis is local around the base model.

Abstract

from arXiv · show

On-policy distillation (OPD) has emerged as an efficient post-training paradigm for large language models. However, existing studies largely attribute this advantage to denser and more stable supervision, while the parameter-level mechanisms underlying OPD's efficiency remain poorly understood. In this work, we argue that OPD's efficiency stems from a form of ``foresight'': it establishes a stable update trajectory toward the final model early in training. This foresight manifests in two aspects. First, at the \textbf{Module-Allocation Level}, OPD identifies regions with low marginal utility and concentrates updates on modules that are more critical to reasoning. Second, at the \textbf{Update-Direction Level}, OPD exhibits stronger low-rank concentration, with its dominant subspaces aligning closely with the final update subspace early in training. Building on these findings, we propose \textbf{EffOPD}, a plug-and-play acceleration method that speeds up OPD by adaptively selecting an extrapolation step size and moving along the current update direction. EffOPD requires no additional trainable modules or complex hyperparameter tuning, and achieves an average training acceleration of $3\times$ while maintaining comparable final performance. Overall, our findings provide a parameter-dynamics perspective for understanding the efficiency of OPD and offer practical insights for designing more efficient post-training methods for large language models.

1 Introduction

The paper argues that OPD’s efficiency arises from “foresight”: stable, aligned update trajectories established early in training. This foresight operates through module allocation and update-direction alignment, motivating EffOPD, a simple acceleration method.

  • Foresight Mechanism: OPD establishes stable and highly aligned update directions early in training, enabling rapid convergence with limited exploration and correction.The paper identifies this early trajectory stability as the core parameter-level source of OPD’s efficiency.
  • Module-Allocation Level: Under the same update norm constraint, OPD achieves larger performance gains than RL by concentrating updates on modules more critical to performance improvement.RL instead accumulates substantially larger update norms in modules with limited contribution, despite similar sensitivity patterns across layers and modules.
  • Update-Direction Level: OPD concentrates updates on a few stable dominant directions early, whose subspaces align closely with the final update subspace and remain stable thereafter.This early alignment distinguishes OPD’s update-direction foresight from the contrasting behavior observed for RL.
  • EffOPD: EffOPD performs linear extrapolation along the current update direction to match vanilla OPD’s final performance with fewer training iterations and samples.It is designed as a simple and intuitive acceleration framework that leverages OPD’s inherent foresight.
  • EffOPD: 3× average training acceleration is achieved by EffOPD across model scales from 1.5B to 32B parameters, without additional modules or complex hyperparameter tuning.The method remains orthogonal to existing acceleration techniques.

2 Functional Redundancy Avoidance

OPD exhibits modular-level foresight by concentrating updates in reasoning-critical, high-marginal-utility regions while suppressing changes in low-utility regions. Compared with RL, this produces more compact, task-relevant updates and higher reasoning gains throughout training.

  • Functional Redundancy Avoidance: OPD achieves substantially higher reasoning gains than RL when final updates are scaled to the same norm, indicating fewer weakly task-correlated components.The comparison evaluates WBase + α∆W_RL/OPD with α ∈ [0, 1].
  • Functional Redundancy Avoidance: OPD consistently reaches the same reasoning accuracy as RL with smaller parameter updates and improves accuracy faster under comparable update magnitudes.This pattern holds across intermediate checkpoints throughout training.
  • Functional Redundancy Avoidance: OPD and RL rely on similar functional pathways, but RL makes substantially larger changes in low-sensitivity peripheral layers and middle-layer modules with higher functional contributions.Intervention sensitivity distributions are highly consistent across module types and layer positions.
  • Functional Redundancy Avoidance: OPD improves update efficiency by distinguishing high-benefit from low-benefit parameter regions rather than learning an entirely new update mechanism.Its updates reduce ineffective changes in peripheral layers while preserving stronger reasoning-related parameter changes.
  • Functional Redundancy Avoidance: Property 1, Functional Redundancy Avoidance, formalizes OPD’s early formation of compact, task-relevant updates concentrated in reasoning-critical modules.The property links modular foresight with stronger reasoning performance gains and higher update efficiency than RL.

3 Early Low-Rank Lock-in

OPD exhibits stronger low-rank concentration and stabilizes its dominant update subspaces early, with later training mainly increasing magnitude along those directions. This Early Low-Rank Lock-in helps explain OPD’s efficient, less redundant update trajectory.

  • Property Definition: OPD’s Early Low-Rank Lock-in is characterized by stronger low-rank concentration, early stabilization of dominant subspaces, and subsequent progress mainly along those subspaces.The section evaluates this property through spectral structure, subspace contributions, and temporal stabilization.
  • Top-k% Subspace: 95% of full-model reasoning performance is recovered using 10% of the rank, with OPD outperforming RL across evaluated rank levels, model scales, and thresholds.The Top-k% subspace is therefore the primary carrier of reasoning improvements, and OPD concentrates updates on more effective directions.
  • Bottom-k% Subspace: 1.6 to 2.5 times: RL’s Bottom-50% tail-subspace norm exceeds OPD’s, while RL’s slight performance advantage remains limited.This indicates that RL allocates substantial update magnitude to tail directions with low marginal utility.
  • Subspace Evolution: OPD’s dominant subspaces align more strongly with their final counterparts than RL’s across evaluated ranks, with smaller fluctuations, especially during 0%–30% of training.The t-SNE trajectories likewise show OPD evolving more compactly and smoothly than RL.
  • Magnitude Scaling: 80% of final-model performance is recovered by a checkpoint at 10% training progress after norm scaling, while KL divergence to the teacher also decreases.These results indicate that early OPD checkpoints already contain effective directions, with later training primarily accumulating update magnitude.

4 Accelerating OPD via Directional Extrapolation

EffOPD accelerates OPD by extrapolating along locally estimated, early-stable update directions and adaptively accepting only candidates that improve lightweight validation performance. Across model scales and tasks, it converges substantially faster than vanilla OPD while improving stability and often reaching a higher performance upper bound.

  • Acceleration procedure: EffOPD performs extrapolation at exponentially spaced checkpoints using parameter displacements as local approximations of subsequent OPD update directions.The first checkpoint uses the displacement from the initial parameters to W1; later checkpoints use displacement between adjacent exponential checkpoints.
  • Acceleration procedure: EffOPD evaluates five increasingly aggressive candidate parameters using 50 randomly sampled training examples as a lightweight validation set.Candidates are evaluated sequentially from the current checkpoint parameters, with the coefficient 2^k controlling extrapolation scale.
  • Adaptive acceptance: EffOPD terminates extrapolation when a candidate fails to improve validation performance, reverting to vanilla OPD if the first candidate fails.This immediate validation mechanism limits performance degradation from excessive extrapolation while exploiting stable update directions.
  • Experimental results: More than 3× speedup is achieved on mathematical reasoning tasks, with EffOPD typically converging in about 10 training steps versus 30–40 for vanilla OPD.The evaluation covers 1.5B, 4B, 14B, and 32B models trained on code-generation and mathematical-reasoning datasets.
  • Ablation studies: EffOPD improves training stability by filtering overly aggressive extrapolation steps through lightweight validation, unlike larger-learning-rate vanilla OPD, which exhibits oscillations and instability.The ablation study reports that larger learning rates accelerate early convergence but can reduce stability.

5 Conclusion · A Impact Statement · B Related Work

The paper explains OPD’s efficiency through modular redundancy avoidance and early low-rank lock-in, and introduces EffOPD for up to 3× faster training. It also situates OPD among efficient post-training methods and discusses its potential benefits and misuse risks.

  • 5 Conclusion: OPD exhibits Functional Redundancy Avoidance and Early Low-Rank Lock-in, revealing foresight at modular and update-direction levels.These properties were identified through parameter-level analyses across model scales, RL algorithms, and task domains.
  • 5 Conclusion: OPD concentrates optimization early on high-utility modules and directions while achieving RL-comparable reasoning gains with more compact, structured updates.The conclusion attributes this pattern to concentration on high-utility regions from the early training stage.
  • 5 Conclusion: up to 3× training speedup is achieved by EffOPD, a plug-and-play method leveraging early directional stability while maintaining final performance.EffOPD is presented as an acceleration method built on the paper’s parameter-dynamics findings.
  • A Impact Statement: The analysis and EffOPD may reduce post-training computation and make efficient model improvement more accessible, but the techniques may also be misused for harmful applications.The impact statement says the work conforms with the NeurIPS Code of Ethics.
  • B Related Work: OPD has students generate their own samples while receiving dense teacher supervision, and Qwen3 is reported to achieve substantially higher training efficiency than RLVR.MiMo-V2-Flash and Deepseek-V4 are described as integrating multiple teacher skills into small models through multi-task OPD.
  • B Related Work: Prior work finds that RL improves pass@1 sampling efficiency without directly enhancing reasoning ability, while entropy collapse can prematurely degrade exploratory capacity.The cited studies connect rapid early convergence with excessive confidence during reinforcement learning.

C Limitations and Future Work · D Preliminaries and Experimental Setup · D.1 Preliminaries

The paper identifies limitations in extending OPD’s findings to complex settings and in its local theoretical analysis, while outlining future theory and diagnostic directions. It also introduces RL and OPD preliminaries, emphasizing OPD’s dense token-level supervision versus RLVR’s sparse rewards.

  • C Limitations and Future Work: OPD’s identified properties remain unexamined in multi-turn agent tasks and multimodal reasoning, where stronger distributional shifts and complex teacher-student residuals may arise.The study validates the properties from multiple perspectives but does not establish their applicability to these settings.
  • C Limitations and Future Work: The theoretical analysis is inherently local, limiting its characterization of OPD dynamics.The supplied passage identifies locality as a limitation without specifying the omitted global phenomena.
  • C Limitations and Future Work: Future theory should model coupling among the distillation objective, evolving on-policy distribution, and spectral evolution of parameter updates.This direction targets interactions not fully captured by the current analysis.
  • C Limitations and Future Work: Early directional lock-in in OPD may help monitor post-training dynamics through directional alignment, spectral concentration, and update compactness.These metrics are proposed as diagnostic signals for assessing training progress.
  • D Preliminaries and Experimental Setup: The experiments focus on Reinforcement Learning and On-Policy Distillation, with πθ denoting the policy model being optimized.These are the two training paradigms studied in the experimental setup.
  • D.1 Preliminaries: RL samples trajectories from the current policy, uses response quality as reward, and applies a KL constraint to limit deviation from a reference policy.In RLVR, the reward is deterministic and verifiable, such as answer correctness or unit-test passing, without a learned reward model.
  • D.1 Preliminaries: RLVR commonly provides sparse reward only after the full response, whereas OPD supplies dense token-level supervision and therefore achieves higher training efficiency.The RL gradient uses token advantages relative to a baseline, while the practical RLVR signal is often delayed until completion.
  • D.1 Preliminaries: OPD preserves on-policy training by using student-generated trajectories while minimizing reverse KL divergence to a fixed teacher model.A common zero-discount approximation focuses optimization on immediate token-level supervision, producing dense signals at every token position.

D.2 Experimental Setup · E Property 1 Additional Experiment

The experiments evaluate OPD and reinforcement-learning methods across model scales from 1.5B to 32B parameters under shared training configurations. OPD uses a distinct rollout and optimization setup, while GRPO and DAPO adopt method-specific hyperparameters.

  • D.2 Experimental Setup: Experiments span models from 1.5B to 32B parameters, including public pretrained checkpoints and locally trained Verl models.RL experiments cover PPO, GRPO, and DAPO across varying model scales.
  • D.2 Experimental Setup: The study evaluates PPO, GRPO, and DAPO as representative reinforcement-learning algorithms across models of varying scales.These methods are applied to the experimental model suite to assess the generality of the findings.
  • D.2 Experimental Setup: Shared RL configurations use a 2,048-token maximum prompt, 20,480-token maximum response, 22,528-token total budget, 32-sample mini-batches, and 16-step gradient accumulation.The shared setup applies one optimization step after gradients are accumulated for 16 iterations.
  • D.2 Experimental Setup: GRPO uses 0.2 high and low clipping ratios with KL-loss coefficient 0.001, while DAPO employs clip-higher, dynamic sampling, token-level policy gradients, and overlong reward shaping.DAPO follows recommended hyperparameters from Yu et al. (2025), while GRPO follows DeepSeek-AI et al. (2025).
  • D.2 Experimental Setup: The experimental prompt template requests step-by-step reasoning and places the final answer within \boxed{} tags.The template includes explicit <think> </think> markers around the assistant's chain of thought.
  • D.2 Experimental Setup: OPD uses a 2,048-token prompt limit, 16,384-token response limit, 18,432-token total budget, prompt batch size 1,024, and n = 1 rollout output.Its learning rate is 1 × 10^-6 without warmup for 3 training epochs, with runs conducted on 8× or 32× H20 96GB GPUs.
  • D.2 Experimental Setup: The OPD training command is implemented through the Verl framework following the stated OPD experimental setting.The passage identifies the command as the next-page implementation of the OPD configuration.
  • D.2 Experimental Setup: The Verl implementation configures GRPO token-level rollout correction with a 5.0 importance-sampling threshold and a 1,024-example training batch.The command also sets maximum prompt and response lengths to 2,048 and 16,384 tokens, respectively.

E.1 Additional Experiment

Additional experiments show that OPD’s parameter-update efficiency generalizes across model scales and reinforcement-learning algorithms, while component-level analyses support functional redundancy avoidance as an intrinsic and stable property.

  • Scaling Behavior: Across 1.5B–32B models, OPD matches RL reasoning performance with substantially smaller parameter update norms.This indicates that OPD’s functional efficiency is not scale-specific.
  • Algorithmic Robustness: Across examined reinforcement-learning algorithms, OPD consistently achieves superior parameter-update efficiency throughout training.The advantage holds regardless of the teacher RL method’s learning dynamics or convergence behavior.
  • Embedding Analysis: PCA, t-SNE, and cosine similarity analyses quantify embedding shifts between base-model token representations and reasoning-sequence embeddings.The analysis addresses the distributional shift of embedding-layer updates relative to the base model.
  • Component-Level Validation: Across broader datasets and algorithmic settings, results support negligible embedding-layer contributions, middle-layer MLP dominance, and consistent redundancy suppression across architectural components.Figure 10 reinforces these component-level properties as intrinsic and stable.

E.2 Detailed Setup of Sliding-Window Intervention Analysis · F Property 2 Additional Experiment

The sliding-window intervention analysis localizes the functional contribution of parameter updates across Transformer layers and modules by injecting localized updates and measuring performance changes. Additional comparisons show that OPD achieves greater reasoning gains per update norm than RL and reaches comparable reasoning accuracy with smaller update norms.

  • E.2 Detailed Setup of Sliding-Window Intervention Analysis: Localized parameter-update interventions isolate marginal functional contributions across network depths and reveal how update location affects performance.The method injects updates into selected regions and measures the resulting performance change instead of replacing the full model.
  • E.2 Detailed Setup of Sliding-Window Intervention Analysis: A sliding window centered at layer l spans up to 17 layers, extending 8 layers on each side and truncating near model boundaries.The window balances locality and stability by reducing single-layer intervention variance while preserving spatial resolution.
  • E.2 Detailed Setup of Sliding-Window Intervention Analysis: Separate intervened models replace only Attention or MLP parameters inside each window while fixing all other parameters to the base model.This design isolates the independent contributions of the two core Transformer modules.
  • E.2 Detailed Setup of Sliding-Window Intervention Analysis: The analysis evaluates every valid window center on MATH500 using four independent forward passes per intervened model, averaging results to reduce evaluation noise.The valid centers are l = 1, 2, . . . , L −8.
  • F Property 2 Additional Experiment: OPD achieves substantially higher reasoning gains than RL under an identical update norm budget.This comparison comes from scaling analysis of the final checkpoints.
  • F Property 2 Additional Experiment: OPD achieves the same reasoning accuracy as RL with substantially smaller parameter update norms at intermediate checkpoints.The comparison tracks checkpoints throughout training.
  • F Property 2 Additional Experiment: Layer-wise update norms and sliding-window interventions connect update distributions across architectural components with RL reasoning accuracy on MATH500.The analysis uses RL/OPD-trained Qwen3-8B-Base models and separately examines embedding-layer replacement effects.
  • F Property 2 Additional Experiment: Token-embedding comparisons quantify cosine similarity with the Base model and visualize embedding shifts among Base, RL, and OPD models using t-SNE.The visualizations distinguish Base-to-RL and Base-to-OPD shifts.

F.1 Geometric Metrics for Parameter Update Matrix

This section defines four complementary metrics for characterizing the geometric structure of parameter update matrices through their singular-value decompositions. The metrics quantify update magnitude, concentration in dominant directions, spectral entropy, and energy captured by the Top 1% subspace.

  • SVD foundation: The update matrix is decomposed as ΔW = UΣVᵀ, with singular values σ1 ≥ σ2 ≥ ··· ≥ σr > 0 sorted by magnitude.Here, r = rank(ΔW).
  • Spectral concentration: The spectral norm, σ1, measures update magnitude along the dominant parameter-space direction.It corresponds to the maximum amplification induced by ΔW on any input vector.
  • Spectral concentration: The spectral-to-Frobenius norm ratio measures dominance of the leading singular direction, with values near 1 indicating concentration along one direction.Smaller values indicate update energy distributed across multiple directions.
  • Spectral entropy: Effective rank measures singular-spectrum entropy, where smaller values indicate rapid decay and concentration in a low-dimensional subspace.Larger effective rank indicates a more diffuse distribution of update energy.
  • Top-1% concentration: The Top-1% subspace norm ratio measures the fraction of update energy captured by the leading 1% of singular directions.Values close to 1 indicate confinement to an extremely low-dimensional subspace; reported metrics are averaged across all MLP and attention matrices for each model.

F.2 Cosine Similarity Analysis of Subspaces

Cosine-similarity analysis shows that OPD locks dominant update subspaces into stable, final-aligned directions early in training, unlike RL’s continuously changing subspaces. This early alignment is especially strong in intermediate layers and supports parameter-efficient optimization focused on task-relevant directions.

  • Subspace Analysis: SVD-based cosine similarity between early-stage and final-stage Rank-1 subspaces measures directional lock-in during training.Rank-1 denotes the strongest singular direction and captures the primary low-dimensional structure of update energy.
  • RL: RL maintains consistently low early-to-final subspace similarity across modules, indicating unstable dominant update directions.Its subspaces undergo substantial changes throughout training, requiring continued exploration and correction before convergence.
  • OPD: OPD achieves substantially higher subspace consistency across most modules, with intermediate-layer cosine similarity reaching up to 0.9.This indicates that OPD identifies stable dominant directions early, after which optimization mainly amplifies their magnitude.
  • Mechanistic Interpretation: Early stabilization of dominant subspaces in reasoning-critical intermediate modules explains how OPD concentrates updates into compact, parameter-efficient changes.The analysis supports Property 1 by linking suppression of redundant updates with stable task-relevant directions.

F.3 Trajectory Evolution of Subspaces · F.4 Scaling Effects on Accuracy and Distribution Alignment

OPD follows a smoother, more concentrated, low-dimensional trajectory than RL, reflecting stable update directions. Scaling early updates improves performance and teacher-distribution alignment, with optimal results around 0.8 ≤β ≤1.2.

  • F.3 Trajectory Evolution of Subspaces: OPD’s Rank-1 subspace trajectories occupy a narrower region and follow a smoother, near-linear path than RL’s dispersed, irregular trajectories.This indicates stronger directional stability and a more structured, predictable optimization trajectory.
  • F.3 Trajectory Evolution of Subspaces: OPD consistently achieves higher EVR0:2 than RL, showing that its representations remain concentrated in a compact, low-dimensional subspace during training.RL distributes variation across more directions, indicating less structured trajectory evolution.
  • F.3 Trajectory Evolution of Subspaces: OPD preferentially updates along a small number of stable, effective directions, imposing an implicit low-rank bias and compact parameter evolution.The concentration constrains both update direction and support under dense teacher supervision.
  • F.4 Trajectory Evolution of Subspaces: Increasing β from 0 improves performance, gains plateau near β ≈0.8, and performance degrades when β exceeds approximately 1.2.Figure 14 identifies the optimal performance range as 0.8 ≤β ≤1.2.
  • F.4 Scaling Effects on Accuracy and Distribution Alignment: KL divergence decreases monotonically as β increases, stabilizes across the performance plateau, and rises again for β > 1.2.The alignment trend mirrors accuracy changes and links scaling with closer student-teacher distribution matching.
  • F.4 Scaling Effects on Accuracy and Distribution Alignment: Scaling reveals that early update directions already capture dominant teacher-distribution structure, as KL reduction coincides with improving task accuracy.This interpretation connects distribution alignment with the geometry of the early update subspace.
  • F.4 Scaling Effects on Accuracy and Distribution Alignment: Matching the early checkpoint’s update norm to the final model significantly improves generated-response quality, producing more reasoning steps with finer-grained progression.The example compares scaled and unscaled early-checkpoint responses.

F.5 A Local Geometric View of OPD Dynamics

A local geometric analysis models OPD as a convex quadratic near the base model, explaining its early low-rank and directionally stable updates. Concentration of the driving signal in dominant eigenspaces suppresses weakly coupled modules and confines optimization to critical parameter directions.

  • Local Quadratic Approximation: Linearizing logits around the base model yields a quadratic OPD objective weighted by the Fisher geometry of the output distribution.The approximation is valid for small parameter displacements and uses the base-model Fisher matrix when teacher and student logits are sufficiently close.
  • Spectral Decomposition and Directional Dynamics: High-curvature eigen-directions saturate earlier, so updates remain low-dimensional when the driving-term projections vanish across many directions.Each eigen-direction approaches its asymptotic contribution at a rate determined by its curvature, while zero projections exclude directions from the effective update.
  • Early Low-Rank Lock-in: When the driving term concentrates in a top-k eigenspace and a spectral gap exists, optimization stays approximately confined there from early training.Later optimization primarily increases update magnitude within the identified subspace rather than exploring substantially new directions.
  • Why is b low-rank in practice?: Teacher-base residuals concentrated on functionally important token positions produce parameter updates focused on directions affecting critical predictions.The Fisher matrix reweights residual directions, and averaging Jacobian-transformed signals over contexts concentrates the driving term in parameter space.
  • Module-Level Dynamics: Modules with negligible coupling to the teacher residual receive negligible updates, explaining suppressed updates in embedding and bottom/top transformer layers.This provides a mechanism for Functional Redundancy Avoidance when cross-module coupling terms are not dominant.
  • OPD versus RL: OPD uses dense per-token residual supervision, whereas sparse-reward RL has noisier gradients and higher-variance credit assignment, yielding a smoother OPD trajectory.The analysis expects Tr(ΣRL) > Tr(ΣOPD) in practice.
Loading 2605.11739v3…