PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Geometry of On-Policy Distillation

Zhennan Shen, Yanshu Li, Qingyu Yin, Chak Tou Leong, Zhilin Wang, Yanxu Chen, Rongduo Han, Sunbowen Lee, Yi R. Fung

arXiv:2606.07082v3cs.LGcs.AI

TL;DR

OPD’s parameter-space training dynamics remain unclear because it combines features of supervised fine-tuning and reinforcement learning. This paper uses parameter-space diagnostics and trajectory analyses to compare OPD with SFT and RLVR, finding that OPD occupies a distinct relaxed off-principal regime with early subspace locking.

  • Problem

    OPD’s parameter-space dynamics remain poorly understood because its distillation and on-policy optimization components connect it to both SFT and RLVR.

  • Method

    The paper compares OPD, SFT, and RLVR using parameter-space diagnostics and tracks cumulative-update subspaces across training checkpoints.

  • Results

    OPD occupies a relaxed off-principal regime and rapidly locks cumulative updates into a narrow low-dimensional channel that preserves OPD performance when enforced early.

  • Takeaways & Limitations

    OPD induces its own update geometry, suggesting that future algorithms should monitor its locked channel and treat objective composition as a key control lever.

  • Takeaways & Limitations

    The analysis uses controlled Qwen3-family reasoning settings, so the observed geometry may vary across model families, modalities, and task distributions.

Abstract

from arXiv · show

On-policy distillation (OPD) is increasingly used to improve large language model reasoning, but its training dynamics remain poorly understood. We characterize the trajectory of OPD updates in parameter space and compare it with supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). A suite of parameter-space diagnostics consistently places OPD in a relaxed off-principal regime: compared with SFT, its updates affect fewer weights and avoid principal directions more strongly, while compared with RLVR, they remain less tightly constrained. Beyond this static localization, OPD exhibits subspace locking: its cumulative updates rapidly enter a narrow low-dimensional channel. Constraining training to the update subspace formed early in training preserves OPD performance but substantially degrades SFT, indicating that the locked subspace is functionally sufficient for OPD. Control experiments further show that sparsifying the update tokens and shifting rollout generation off-policy preserve the rank dynamics, whereas mixing the OPD objective with RLVR changes them. Overall, these results suggest that OPD is not merely an intermediate point between SFT and RLVR, but induces its own update geometry in parameter space.

1 Introduction

The introduction frames OPD as a poorly understood post-training paradigm and positions it as a distinct update geometry rather than a simple interpolation between SFT and RLVR. It highlights relaxed off-principal localization, early subspace locking, and sensitivity to objective composition.

  • Motivation: OPD combines dense token-level distillation with on-policy sampling, leaving its parameter-space dynamics distinct from the established SFT and RLVR geometries.SFT is described as dense and principal-aligned, whereas RLVR is sparse and off-principal.
  • Relaxed off-principal regime: Across update sparsity, spectral drift, principal-subspace rotation, and update localization, OPD occupies a relaxed off-principal regime between SFT and RLVR, biased toward RLVR.OPD is more selective and geometry-preserving than SFT, yet less constrained than RLVR.
  • Subspace locking: OPD rapidly enters a narrow low-dimensional update band during training, while SFT expands and RLVR contracts.The analysis tracks cumulative updates across checkpoints using effective dimension, update scale, and spectral shape diagnostics.
  • Control experiments: Token sparsification and off-policy rollouts preserve OPD’s rank trajectory, changing update scale at most.These controls test token supervision density and rollout policy as factors distinguishing OPD from standard SFT or RLVR.
  • Control experiments: Objective-level interpolation changes OPD’s rank dynamics, showing that subspace locking is robust to runtime perturbations but sensitive to objective composition.The introduction identifies objective composition as the factor that controls the locked trajectory.

2 Related Work

Prior work frames post-training around SFT, RL-based methods, and OPD, while establishing a geometric split between dense, principal-direction SFT updates and localized, off-principal RL updates. OPD has mainly been studied behaviorally or for efficiency, leaving its position in the SFT–RLVR parameter-space spectrum and its training trajectory open.

  • Post-training for large language models: Post-training reasoning research centers on SFT, RLHF/RLVR, and OPD as three broad paradigms.SFT uses offline demonstrations and cross-entropy objectives, while RL methods optimize reward-based objectives; recent work also studies OPD.
  • Post-training for large language models: Recent OPD studies examine empirical failure modes, scaling recipes, multi-teacher distillation, and training efficiency, but not its parameter-space position.These accounts characterize when OPD works behaviorally or efficiently while leaving its location within the broader SFT–RLVR spectrum largely unexamined.
  • Post-training weight geometry: Prior weight-geometry studies contrast dense, principal-direction SFT updates with highly localized, off-principal online-RL updates.The RL bias is described as a conservative projection that limits policy-level KL drift.
  • Post-training weight geometry: Zhu et al. (2025) provide the closest methodological precedent by explaining RLVR’s off-principal updates through a model-conditioned optimization bias and Three-Gate account.Their account includes a KL anchor constraining update magnitude and pretrained model geometry steering bounded steps toward low-…
  • Post-training weight geometry: This paper extends the established SFT–RLVR geometric split to ask where OPD lies in that spectrum, how its updates evolve, and which OPD-specific factors control the trajectory.The authors explicitly build on the prior geometric lens while studying a different regime.

3 Locating OPD in Parameter Space

OPD occupies a relaxed off-principal regime between SFT and RLVR: it is more selective and geometry-preserving than SFT, yet less constrained than RLVR. Its dense token-level supervision broadens directional support while preserving geometry-steered updates.

  • Overall localization: OPD is more selective and geometry-preserving than SFT, yet less constrained than RLVR, defining a relaxed off-principal regime.Across diagnostics, OPD follows RLVR’s off-principal direction with weaker selectivity and larger visible support.
  • Update sparsity: 51.6% of weights remain unchanged for OPD at bf16 precision, between SFT’s 8.1% and RLVR’s 77.2%.OPD variants remain within 48.6%–57.1% sparsity across teacher scale, student scale, code data, and a MoE teacher.
  • Subspace rotation and spectral drift: Representative-layer principal angles are around 1° for OPD, versus above 10° for SFT and below 0.5° for RLVR.Spectral drift follows the same ordering, with RLVR at the 10−5 level.
  • Update localization: 46.28% global update density for OPD lies between SFT’s 92.73% and RLVR’s 27.76%.Low-magnitude overlap is 53.59% for OPD, compared with 31.88% for SFT and 73.48% for RLVR; principal-mask overlap places OPD below the 30% random baseline.
  • Signal granularity: OPD replaces RLVR’s shared sequence-level reward coefficient with heterogeneous token coefficients based on local teacher–student disagreement.This broadens accessible directional support while keeping updates geometry-steered, matching OPD’s intermediate preservation between SFT and RLVR.

4 Subspace Locking in OPD

OPD rapidly enters a persistent, narrow low-dimensional update channel rather than interpolating temporally between SFT and RLVR. This channel emerges early and is functionally sufficient for OPD training, unlike for SFT.

  • Subspace emergence: OPD rapidly enters a persistent low-dimensional update channel, while SFT expands its subspace and RLVR contracts toward a low-dimensional endpoint.These dynamics show that OPD is not a temporal interpolation between SFT and RLVR.
  • Update geometry: OPD accumulates a substantially larger update than RLVR while ending with comparable stable rank, ruling out a trivial small-update explanation.Hill tail estimates additionally show mild OPD evolution, versus sharp SFT increases and RLVR decreases.
  • Early subspace emergence: OPD aligns with its final V16 subspace from the first measured checkpoint, whereas SFT and RLVR converge more gradually.This indicates that OPD’s low-dimensional channel emerges early rather than being assembled late.
  • Functional sufficiency: Rank-16 projection leaves OPD essentially unchanged but degrades SFT, showing that the early low-dimensional channel is sufficient for OPD training.The same qualitative pattern holds across additional reasoning benchmarks.
  • Implication: OPD subspace locking identifies an early-emerging update channel that is both persistent and sufficient for training.The paper suggests monitoring and exploiting this channel rather than treating low-dimensionality only as a post-hoc spectral statistic.

5 What Controls Subspace Locking?

Subspace locking remains stable under token sparsification and off-policy rollouts, but changes when the objective is mixed with RLVR. This indicates that locking is sensitive to the gradient source rather than runtime sampling perturbations.

  • Runtime perturbations: Token sparsification at 25% and 50% density closely tracks the OPD stable-rank trajectory, including random 25% retention.Random retention changes update scale more than spectral shape, so the low-rank profile is not localized to a small set of high-KL tokens.
  • Runtime perturbations: Off-policy rollouts leave the stable-rank trajectory nearly unchanged while producing a modestly larger update norm.Stable rank remains matched to on-policy OPD despite the rollout-policy change.
  • Mechanistic interpretation: Objective composition changes rank dynamics in a way that token sparsification and rollout-policy changes do not.Objective mixing changes the gradient source, whereas runtime perturbations alter sampled data or token coverage while preserving the teacher-token gradient source.
  • Objective composition: OPD-dominant OPD–RLVR mixtures retain the OPD-like stable-rank trajectory, but weakening the OPD component produces a distinct spectral regime.The trajectory no longer follows the baseline when the OPD component becomes weak.
  • Implication: Subspace locking is objective-sensitive rather than runtime-induced.The implication is to regulate objective-induced update geometry, not only token coverage or rollout generation.

6 Discussion & Conclusion

The discussion characterizes OPD as a distinct parameter-space regime with a relaxed off-principal trajectory and rapid subspace locking into a persistent low-dimensional channel. It argues that future OPD methods should treat geometry control—especially objective composition—as central to preserving effective updates while limiting drift.

  • Discussion & Conclusion: OPD is a distinct parameter-space regime, not a simple interpolation between SFT and RLVR.It occupies a relaxed off-principal region while exhibiting subspace locking.
  • Discussion & Conclusion: Cumulative OPD updates rapidly enter a small, persistent low-dimensional channel sufficient to preserve training progress.This locked channel is presented as functionally sufficient for OPD learning.
  • Discussion & Conclusion: Future OPD algorithms should treat training as geometry control rather than merely denser token supervision.The proposed recipe is to monitor the locked update channel and use objective composition when the geometry drifts.
  • Discussion & Conclusion: Runtime perturbations preserve the OPD stable-rank trajectory, whereas objective-level interpolation changes it.This identifies objective composition as the sensitive control axis for subspace locking.
  • Discussion & Conclusion: Token selection, rollout policy, and teacher scale should be tuned through their effects on update geometry.Geometry-aware control may improve OPD stability, interpretability, and transferability by preserving the learning-supporting channel while avoiding unnecessary drift.

Limitations … C Experimental Details

The paper’s limitations are restricted to controlled Qwen3-family reasoning settings, and its parameter-space diagnostics provide mechanistic explanations rather than complete causal theories. The authors also describe their use of ChatGPT for writing and scripting assistance, rely on publicly available research artifacts, and report author verification of all scientific content.

  • Limitations: The analysis is limited to controlled Qwen3-family reasoning settings, so its observed geometry may vary across model families, modalities, and task distributions.The controlled design isolates training-paradigm effects but may not generalize broadly.
  • Limitations: The Three-Gate account and covariance analysis are mechanistic explanations consistent with checkpoint evidence, not complete causal or formal theories of OPD optimization.The diagnostics characterize parameter-space trajectories from stored checkpoints.
  • A AI Usage: ChatGPT assisted with writing, language polishing, LaTeX formatting, organization suggestions, and presentation refinement.The authors also used ChatGPT to assist with drafting plotting and analysis scripts.
  • A AI Usage: The authors reviewed, verified, and edited all scientific claims, experimental designs, code, figures, numerical results, and final manuscript content.No generative AI system was used as an author, to produce experimental results, or to make autonomous scientific decisions.
  • B Artifact Use: The study uses publicly available model checkpoints, datasets, and benchmarks only for research evaluation and analysis.The corresponding creators are cited, and the artifacts are used consistently with their intended research use.
  • B Artifact Use: The authors checked public licenses or usage terms where available for major artifacts including Qwen3, DAPO-Math-17k, Dolci-Think SFT, DeepCoder, and LiveCodeBench.The original artifacts are not redistributed.

C.1 Training Setup · C.2 OPD Variant Setup · C.3 Parameter-Space Diagnostic Implementation

The paper compares SFT, OPD, and RLVR from explicitly defined initialization points, supplements OPD with robustness variants, and computes offline parameter-space diagnostics on checkpoint pairs. The analyses standardize matrix-level averaging while specifying procedures for sparsity, principal-angle rotation, spectral drift, and update-mask overlap.

  • C.1 Training Setup: The shared training setup uses Qwen3 checkpoints, with math-domain OPD/RLVR runs on dapo-math-17k and the SFT anchor trained on Dolci-Think SFT data.OPD and RLVR share the SFT anchor, while the SFT trajectory begins at the pretrained base checkpoint.
  • C.1 Training Setup: The main comparison uses approximately 5k, 300, and 1k training steps for SFT, OPD, and RLVR, respectively, based on evaluation-curve saturation.These rounded horizons were selected for interpretability and reproducibility rather than diagnostic optimization.
  • C.1 Training Setup: OPD and RLVR start from the shared SFT anchor iter_0005375, whereas SFT is measured from the pretrained Qwen3-8B-Base checkpoint.All updates use ∆W = W+ − W0 relative to the corresponding stage initialization.
  • C.2 OPD Variant Setup: OPD variants follow the default Table 4 configuration unless an explicitly listed factor changes, with variants covering code-domain data, teacher choice, and random seeds.The code-domain variant uses DeepCoder data and LiveCodeBench v5 evaluation; the MoE-teacher variant uses Qwen3-30B-A3B.
  • C.3 Parameter-Space Diagnostic Implementation: All diagnostics run offline on saved checkpoint pairs and compute ∆W = W+ − W0, averaging metrics across matrices unless otherwise specified.For OPD and RLVR, W0 is the shared SFT anchor; for SFT, it is the pretrained Qwen3-8B-Base model.
  • C.3 Parameter-Space Diagnostic Implementation: bf16-aware update sparsity casts W0 and W+ to bf16 before converting them back to fp32, while overall sparsity is parameter-count weighted across analyzed weights.Metrics also include per-type summaries, with matrices analyzed across layers and QKV projections split into Q, K, and V when needed.
  • C.3 Parameter-Space Diagnostic Implementation: Principal-angle rotation compares top-k left and right singular subspaces using CPU float64 SVD to avoid unstable angles for near-identical matrices.Spectral drift uses descending singular-value vectors, while update-mask overlap compares principal and low-magnitude masks against a random baseline of α.

D Objectives and Three-Gate Extension · E Trajectory Metric Definitions

The paper extends the Three-Gate account by framing OPD as an anchored, geometry-constrained update process that uses dense token-level supervision, enabling broader directional support and more visible updates than RLVR. It defines trajectory diagnostics that quantify cumulative update dimensionality, magnitude, and spectral shape.

  • D Objectives and Three-Gate Extension: The Three-Gate account attributes RLVR’s visible update sparsity to policy anchoring, geometry-steered location, and bf16 precision filtering.The KL anchor limits update size, model geometry steers updates off principal directions, and precision determines which coordinates are visible.
  • D Objectives and Three-Gate Extension: OPD preserves RLVR’s on-policy structure but replaces sequence-level scalar credit with dense token-level coefficients induced by teacher–student disagreement.This heterogeneous token supervision expands accessible directional support while remaining within the same geometry-steered update family.
  • D Objectives and Three-Gate Extension: OPD is relaxed rather than unanchored: it remains constrained by a local weight bound while using denser token-level information inside the anchored region.The distinction from RLVR is the information granularity, not removal of the anchor.
  • D Objectives and Three-Gate Extension: OPD’s dense token-level signal permits larger subspace rotation and spectral drift than RLVR while remaining in a bounded-update regime that tends to preserve pretrained spectral structure.Small operator and Frobenius norms preserve spectral structure under Wedin-type perturbation bounds.
  • D Objectives and Three-Gate Extension: OPD’s larger-than-RLVR updates cause more coordinates to cross the bf16 realization threshold, producing intermediate bf16-aware update sparsity.Precision determines which underlying coordinate changes become visible in stored weights.
  • E Trajectory Metric Definitions: Trajectory analysis evaluates the cumulative update ∆W_t = W_t − W_0 at each checkpoint, averaging metrics across analyzed weight matrices.This establishes the common object used for the trajectory-level diagnostics.
  • E Trajectory Metric Definitions: Stable rank measures effective update dimensionality, Frobenius norm measures cumulative update magnitude, and the Hill tail estimator provides an auxiliary measure of spectral shape.Stable rank is scale-aware and insensitive to arbitrarily small singular values; Frobenius norm helps rule out negligible movement as the cause of low stable rank.

F Additional Evaluation of Rank-Constrained Training … G.2 Perturbation Protocols

Additional benchmarks confirm that OPD remains robust under an early rank-16 subspace constraint, unlike SFT. The control setup then isolates token sparsification, rollout policy, and objective composition while preserving the stated baseline conditions.

  • F Additional Evaluation of Rank-Constrained Training: OPD remains broadly robust under the early rank-16 subspace constraint, whereas SFT is substantially more sensitive.This extends the functional-sufficiency conclusion beyond the primary AIME 2024 evaluation.
  • F Additional Evaluation of Rank-Constrained Training: Across five additional reasoning benchmarks, OPD is consistently less affected by rank-16 projected training than SFT.The projection subspace, projection start, and matched training windows match the main experiment.
  • G.1 Shared OPD Control Setup: All control experiments use the same OPD baseline configuration, changing only the stated intervention while keeping the remaining training pipeline fixed.Full training hyperparameters are provided in Appendix C.
  • G.1 Shared OPD Control Setup: Spectral diagnostics analyze 144 matrices from 36 layers and four attention weight types per layer, averaging metrics across matrices.Stable rank is the primary diagnostic; Frobenius norm and Hill tail are auxiliary diagnostics.
  • G.2 Perturbation Protocols: Token sparsification independently masks sampled response tokens according to student–teacher log-probability deviation while keeping rollout policy and the OPD objective fixed.Teacher model, on-policy generation, optimizer, batch size, learning rate, and training data remain identical to baseline.
  • G.2 Perturbation Protocols: Objective interpolation mixes the OPD teacher-correction signal with the RLVR reward signal while keeping the student rollout policy fixed.The OPD term encourages correction toward the teacher, while the RLVR advantage comes from math accuracy reward.
  • G.2 Perturbation Protocols: The interpolation evaluates α ∈{0.75, 0.50, 0.25, 0.05, 0.01}, with α = 1 as pure OPD and α = 0 as the RLVR endpoint.Runs use raw signal mixing without per-batch standard deviation normalization and otherwise match the baseline evaluation setup.

G.3 Auxiliary Metrics

Auxiliary diagnostics show that runtime perturbations preserve OPD-like spectral trajectories up to scale shifts, while objective interpolation changes the update geometry more substantially.

  • Auxiliary metrics: Frobenius norm and Hill tail estimates corroborate that runtime perturbations preserve OPD-like spectral profiles up to modest scale changes.These diagnostics complement stable rank by tracking update scale and spectral shape.
  • Auxiliary metrics: Objective interpolation induces a distinct spectral trajectory and changes update geometry more substantially than runtime perturbations.The comparison is made while varying one target factor and keeping the remaining OPD setup fixed where possible.
Loading 2606.07082v3…