PaperlayerView exact revision on arXiv

Source-linked AI summary

Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation

Guo Yu, Wenlin Liu, Yulan Hu, Hao-Xuan Ma, Jun-Peng Jiang, Han-Jia Ye

arXiv:2606.13657v3cs.LG

TL;DR

How on-policy distillation changes model parameters remains poorly understood. This paper analyzes OPD updates across language and vision-language model pairs and finds small, sparse, spectrally concentrated changes that nevertheless nearly preserve full-training reasoning performance when restricted to the discovered subnetwork.

  • Problem

    Despite OPD’s empirical success, evidence remains limited about how its on-policy dense-supervision regime changes models in weight space.

  • Method

    The paper analyzes OPD checkpoint updates across 12 model pairs and tests their sparsity, geometry, source-spectrum changes, and masked-training recoverability.

  • Results

    OPD produces small, coordinate-sparse, spectrally concentrated yet numerically full-rank updates whose visible supports avoid source-principal coordinates, while discovered-mask training nearly recovers full-training reasoning performance.

  • Takeaways & Limitations

    Despite dense teacher supervision, OPD resembles sparse on-policy editing rather than dense supervised rewriting, with on-policy data implicated as a major determinant of update geometry.

  • Takeaways & Limitations

    The analyses mainly use static checkpoint diffing, leaving the learning dynamics that produce final updates unresolved.

Abstract

from arXiv · show

On-policy distillation (OPD) has recently become a prominent post-training recipe by combining two desirable ingredients: on-policy student-generated trajectories and dense token-level teacher supervision. Yet how this hybrid training regime shapes a model remains poorly understood. We characterize the sparsity and geometry of OPD parameter updates across several language and vision-language model pairs and application settings. OPD updates are small and coordinate-sparse at checkpoint precision, while remaining distributed across layers and modules. This sparse support is operationally meaningful: masked training on the discovered subnetwork nearly recovers full-training performance. At the matrix level, the updates are numerically full-rank but spectrally concentrated. Their visible supports avoid coordinates emphasized by the source's principal structure and favor low-magnitude source coordinates, while the source singular-value spectra change little. Together, these findings show that OPD exhibits important weight-space signatures of on-policy post-training despite using dense teacher supervision.

1 Introduction

OPD combines on-policy student samples with dense teacher feedback, yet its parameter updates are small, sparse, and closer to sparse on-policy editing than dense supervised rewriting. Masked retraining further shows that the discovered update support nearly preserves reasoning performance, unlike density-matched random masks.

  • Motivation: OPD combines on-policy student samples with dense teacher feedback, positioning it between supervised fine-tuning and sparse-reward RLVR in training design.This combination addresses distribution shift from fixed demonstrations while avoiding the sparsity of outcome-based rewards.
  • Contributions: OPD updates are small and coordinate-sparse at checkpoint precision, while remaining distributed across layers and modules.The paper measures update norms, coordinate sparsity, and layer- and module-level structure across several model pairs and settings.
  • Operational significance: The discovered nonzero checkpoint-delta mask nearly recovers full-training reasoning performance, whereas density-matched random masks generally perform worse.The comparison uses OPD restarted from the source model with training restricted to selected coordinates.
  • Interpretation: Dense teacher supervision does not make OPD an ordinary dense parameter-rewriting process; its updates are closer to sparse on-policy editing than dense supervised rewriting.The paper also connects these observations, under explicit assumptions, to minimum-norm corrections and on-policy curvature.

2 Background

OPD combines student-generated, on-policy trajectories with dense token-level teacher supervision, distinguishing it from fixed-dataset distillation and sequence-level reward-based RLVR. This formulation motivates studying whether on-policy training preserves sparse, geometry-related update patterns despite dense feedback.

  • Distinction: Unlike ordinary supervised fine-tuning and offline distillation on fixed target sequences, OPD conditions on prefixes generated by the student rather than a fixed teacher trace.RLVR also samples from the current policy, but commonly uses environment- or verifier-derived sequence-level rewards.
  • Formulation: OPD samples responses from the student’s current policy and applies token-level teacher–student divergence along those student-generated trajectories.The divergence may be forward or reverse KL, JS, or another token-level f-divergence.
  • Motivation: OPD serves as a diagnostic for whether sparse post-training updates arise from sparse rewards or from training near the model’s own policy distribution.Dense teacher feedback provides a contrast with RLVR’s sparse environment rewards while changing both the source and granularity of the learning signal.
  • Applications: OPD applications mainly comprise cross-scale capability transfer, multi-capability consolidation, and on-policy self-distillation.These groups include larger external teachers supervising smaller students, specialized policies supervising one student, and asymmetric teacher–student roles within the same model.
  • Related work: Recent work analyzes post-training through parameter updates, reporting small RLVR subnetworks and geometry biased away from source-weight principal directions and toward low-curvature, spectrum-preserving regions.Subnetwork-only fine-tuning can recover the full RLVR run.

3 Parameter-Update Analysis Overview

This section analyzes OPD checkpoint deltas through scale, coordinate-support, spectral, and source-geometry metrics. The study covers 12 model pairs, including six OPD-style pairs.

  • Analysis framework: Checkpoint deltas ΔW = Wtrained − Wsrc define the analysis of what OPD changes in weight space.The framework asks complementary questions about update scale, coordinate support, spectral structure, and source geometry.
  • Overview findings: Figure 1 characterizes OPD-style updates as small, coordinate-sparse, spectrally concentrated, and biased away from source-principal coordinates.Gray bars provide offline SFT/SEQKD and RLVR references.
  • Analysis framework: The metrics test whether updates are globally small, coordinate-sparse, strictly low-rank or full-rank with concentrated singular-value energy, and geometrically source-aligned.Source-geometry metrics examine preservation of the source singular-value spectrum and whether visible supports favor source-principal or low-magnitude coordinates.
  • Experimental scope: 12 model pairs are analyzed, including six OPD-style pairs spanning language and vision-language models and multiple distillation settings.The listed OPD-style pairs include R1-Qwen1, MiniCPM5-1B, Qwen2.5-VL-3B, Qwen3-1.7B, and Qwen3-4B configurations.

4 Sparsity of OPD Updates

OPD produces small, coordinate-sparse updates whose visible supports are distributed across layers and reused nonrandomly across training variations. The discovered support is functionally meaningful, allowing masked retraining to nearly recover full-training performance.

  • Global sparsity: Across six pairs, OPD deltas are small and coordinate-sparse at checkpoint precision, with 0.0488% relative norm and 83.04% visible sparsity for R1-Qwen.Fixed-data SFT reaches 0.8280% relative norm and 26.63% visible sparsity in the matched comparison.
  • Global sparsity: Even an atypically large and dense JustRL teacher produces a small, sparse, and off-principal OPD student delta.A sparse teacher update is therefore not necessary for OPD to produce a sparse student update.
  • Layer and module distribution: R1-Qwen OPD updates span layers and major projection matrices, while every module’s relative delta norm stays below 0.14% and no module consistently dominates.Most coordinates remain unchanged at the 10−5 threshold, contrasting with the module-localized pattern reported for reasoning SFT.
  • Support reuse: Across algorithm, teacher, and data variations, one-sided support overlap reaches 2.26–3.06× its independent-support baseline, indicating nonrandom support reuse.For R1-Qwen, DeepScaleR RLVR and OPD show 51.80% and 67.54% directional overlap versus 16.96% and 22.11% baselines.
  • Functional sufficiency: Learned OPD masks nearly recover full-training performance and outperform density-matched random masks in both R1-Qwen and Qwen2.5-VL settings.R1-Qwen peaks are 35.10%, 35.52%, and 32.92%; Qwen2.5-VL peaks are 54.24%, 55.72%, and 52.30% for learned-mask, full-training, and random-mask runs, respectively.

5 Geometry of OPD Updates

OPD updates concentrate energy in a small leading singular subspace without numerical rank deficiency, and their leading 1% subspaces nearly recover full performance. At checkpoint precision, OPD supports avoid source-principal coordinates and favor low-magnitude source coordinates.

  • Spectral concentration: 28.69% top-16 energy and 9.15 stable rank under OPD versus 15.34% and 46.48 under matched R1-Qwen SFT show stronger energy concentration.Across all six pairs, OPD has higher top-16 energy and lower stable rank than fixed-data SFT and offline distillation references.
  • Spectral concentration: Every OPD pair has median numerical rankτ = 1 at τ = 10−5, indicating full numerical rank despite concentrated leading singular directions.Smaller singular directions remain present even though leading directions dominate update energy.
  • Functional recovery: Top-1% OPD reconstructions recover 120.41% and 115.38% of the respective gains over Base, matching or slightly exceeding the full checkpoint on both benchmarks.Because evaluation is sampling-based, values above 100% indicate full recovery within evaluation variability rather than improvement from truncation.
  • Source-relative geometry: 4.98% principal-mask overlap for R1-Qwen OPD falls below the 10% independent-support baseline, while low-magnitude-mask overlap reaches 36.68%.Fixed-data SFT and offline distillation remain closer to independent-support overlap, while DeepScaleR RLVR shows the same qualitative bias as OPD.

6 A Local View of Small, Off-Principal Updates

The local analysis explains OPD’s small, off-principal updates through selective on-policy gradient support and behaviorally unnecessary movement. It further links visibility to update norm and source-relative curvature, while emphasizing that these explanations are conditional and provide bounds rather than guarantees.

  • On-policy gradient support: Absent or negligibly probable prefixes contribute no or negligible local OPD gradient, whereas fixed offline data can keep generating gradients for them.This distributional difference makes on-policy updates selective with respect to current-policy support.
  • Behaviorally unnecessary movement: The extra fixed-trajectory component has no first-order effect on current-policy prefixes, formalizing how on-policy training can stop behaviorally unnecessary movement.The comparison assumes both procedures fit the same relevant on-policy corrections, while fixed-trajectory training also fits absent or negligibly weighted prefixes.
  • Norm and spectrum consequences: Small update norm limits how many coordinates cross a fixed visibility threshold and directly bounds source-spectrum drift, but guarantees neither high coordinate sparsity nor concentrated ∆W spectrum.These are upper bounds, not guarantees of the observed checkpoint-level sparsity or spectral concentration.
  • Source-relative support: A coordinate is visible at threshold ε only if |gi| > ε(hi + λ); higher curvature in source-principal coordinates therefore lowers their threshold-crossing probability when gradients are comparable.This mechanism predicts lower visible-update rates inside the principal mask than outside it.
  • Source-relative support: The curvature-based explanation accounts for the below-baseline principal-mask overlap observed in Section 5, but the local explanations remain conditional on their assumptions.Appendix C contains the stated assumptions and derivations.

7 Discussion and Future Work

OPD produces small, sparse, spectrally concentrated updates whose visible supports avoid source-principal coordinates, rather than linearly interpolating between SFT and RLVR. Future work should examine OPD’s learning dynamics, behavioral effects, scale generality, and specialized adaptation or optimization methods.

  • Core message: OPD produces small, sparse, spectrally concentrated updates whose visible supports avoid source-principal coordinates.The paper characterizes OPD as remaining closer to sparse on-policy post-training than to linear interpolation between SFT and RLVR in parameter space.
  • Limitations and future work: Static checkpoint diffing leaves the learning dynamics behind OPD’s final updates unresolved.Complementary behavioral analyses of output-distribution changes and forgetting alleviation also remain important directions.
  • Scale and scope: Larger models and additional domains, including agentic or embodied tasks, are needed to test whether OPD’s observed geometry is stable or recipe-specific.Current interventions are limited to R1-Qwen and Qwen2.5-VL math reasoning, despite experiments spanning relatively small-scale LLMs and VLMs.
  • OPD-native methods: OPD’s spectral concentration motivates low-rank adaptation, while source-principal-coordinate avoidance motivates orthogonal fine-tuning.The discussion specifically points to LoRA and its variants for low-rank adaptation and orthogonal fine-tuning methods for coordinate avoidance.

8 Conclusion · A Related Work

The paper characterizes OPD as sparse, spectrally concentrated, numerically full-rank, off-principal, and spectrum-preserving, with subnetworks nearly recovering full-checkpoint reasoning performance. It relates these findings to on-policy distillation, parameter geometry in post-training, and parameter-efficient adaptation and optimizer design.

  • 8 Conclusion: OPD updates are small, coordinate-sparse, spectrally concentrated yet numerically full-rank, off-principal, and spectrum-preserving.These properties characterize OPD despite dense teacher supervision.
  • 8 Conclusion: Subnetwork-only training and leading-subspace reconstruction nearly recover full-checkpoint reasoning performance.The conclusion identifies both sparse-subnetwork training and leading-subspace reconstruction as effective approximations.
  • 8 Conclusion: Under explicit assumptions, local analysis connects OPD observations to minimum-norm corrections and on-policy curvature.The conclusion frames OPD as resembling sparse on-policy editing despite dense teacher supervision.
  • A Related Work: Early OPD formulations address exposure mismatch in offline sequence-level distillation by training students on sampled trajectories with dense teacher feedback.The related work notes that OPD has become a practical component of large language model post-training, including large-to-small distillation and capability consolidation with multiple experts.
  • A Related Work: Related parameter-geometry work finds that RLVR fine-tunes small subnetworks, moves away from source-weight principal directions, and exhibits dominant low-rank directions predictive of later reasoning improvements.These studies motivate asking whether similar geometry appears in OPD.
  • A Related Work: The paper’s spectral and source-relative coordinate results also relate to parameter-efficient adaptation and optimizer design.The supplied passage specifically introduces low-rank adaptation and orthogonal finetuning as related approaches.
  • A Related Work: LoRA exploits concentration of fine-tuning updates in a low-dimensional additive subspace, whereas OFT/BOFT parameterize adaptation through orthogonal transformations.OFT/BOFT aim to preserve hyperspherical energy and pretrained representations, according to the supplied passage.

B Post-Training Objectives for LLMs

Post-training objectives differ along two axes: whether trajectories are fixed or student-generated, and whether supervision is dense token-level feedback or sparse sequence-level rewards. OPD combines student-generated rollouts with dense teacher-derived feedback through GKD-style or PG-style formulations.

  • SFT and Sequence-Level KD: SFT and SEQKD optimize maximum-likelihood objectives on fixed offline trajectories, with every target token supplying a supervised signal.SEQKD differs from SFT because its target response is generated by a teacher rather than drawn from human annotations.
  • RLVR and GRPO: RLVR samples responses from the current or recent policy and typically optimizes sparse sequence-level rewards, often with a KL penalty to πref.GRPO removes the learned critic by sampling response groups for the same prompt and normalizing rewards within each group.
  • OPD: OPD combines on-policy student rollouts with dense teacher feedback through GKD-style token-level f-divergence or PG-style policy optimization on student-generated samples.The PG-style formulation can be optimized with policy-gradient estimators because sampled trajectories depend on the student policy.
  • OPD: Both OPD styles differ from SEQKD through student-generated trajectories and from RLVR through dense teacher-derived feedback replacing sparse scalar rewards.The PG view is connected to reverse-KL distillation through a score-function identity.

C Local Analysis of On-Policy Updates · D Additional Details

The local analysis identifies two sufficient on-policy mechanisms: minimum-norm updates need not move more than fixed-prefix updates, and curvature can make visible support avoid source-principal coordinates. Deterministic norm bounds further connect smaller movement to threshold-visible sparsity and limited source-spectrum drift, subject to explicit assumptions and failure cases.

  • C Local Analysis of On-Policy Updates: On-policy conditioning is analyzed locally with held rollouts from d0 = dθ0, separating prefix weighting from higher-order sampling-distribution changes.The framework covers next-token likelihood, teacher–student divergence, and policy-gradient local surrogate losses near source parameters θ0.
  • C Local Analysis of On-Policy Updates: Additional fixed-prefix constraints cannot reduce the Euclidean minimum movement under matched feasible constraints and minimum-norm optimization.The result relies on linearized logits, nested constraints, and gradient descent selecting the minimum-norm interpolating solution from δ = 0.
  • C Local Analysis of On-Policy Updates: The extra fixed-prefix component lies in Null(Jon), so it leaves current-policy-prefix logits unchanged to first order while increasing absolute and same-source relative update norms.The ordering does not extend across unrelated source models.
  • C Local Analysis of On-Policy Updates: A smaller update norm yields a no-looser bound on threshold-visible density, but does not determine the realized support because coordinates may distribute differently.The bound applies to visible support Aε = {i : |δi| > ε} and can be loose.
  • D Additional Details: Small relative matrix movement bounds singular-value drift of each trained matrix, explaining low source-spectrum drift without implying concentrated singular values for the update matrix.The perturbation argument applies separately to each matrix.
  • C Local Analysis of On-Policy Updates: Under diagonal curvature, a coordinate becomes visible when |gi| > ε(hi + λ), so higher curvature on source-principal coordinates raises their inclusion threshold.The below-baseline principal-mask-overlap result additionally assumes comparable gradient-magnitude distributions and groupwise law-of-large-numbers behavior.
  • C Local Analysis of On-Policy Updates: The curvature-separation mechanism is specifically plausible on-policy because d0 weights source-produced prefixes, whereas fixed human- or teacher-prefix distributions need not align with source-policy-sensitive coordinates.These propositions are sufficient local mechanisms, not a complete nonlinear training theory; Proposition 1 can fail when objectives, distributions, or optimization solutions are not comparable.

D.1 Model Pairs and Training Details

The study analyzes checkpoint pairs spanning LLM PG-style and VLM GKD-style OPD across three use cases, using public implementations and specified training datasets. Experiments use official-implementation configurations without hyperparameter search, with NVIDIA A800 hardware and fixed evaluation settings.

  • Model pairs: Checkpoint pairs span LLMs with PG-style OPD updates and VLMs with GKD-style OPD updates across three OPD use cases.These pairs are listed in Table 5 for the delta analysis.
  • Implementations and data: Training and evaluation use public HybridFlow in verl v0.8.0 plus public OPD and OPSD implementations where applicable.The R1-Qwen runs use DAPO-Math-17K or OpenThoughts3; VLM GRPO and OPD use Geo3K; the Qwen3-1.7B pair is distilled from Qwen3-4B-Base-GRPO; OPSD uses OpenThoughts-Math-30K subsets.
  • Configuration and reproducibility: Each controlled experiment uses a single configuration based on the corresponding official implementation, with no hyperparameter search or manual seed overrides.For verl-based training, the default seed configuration of verl v0.8.0 is retained.
  • Configuration and reproducibility: All training runs use NVIDIA A800 GPUs, while AIME24 and AIME25 evaluations use temperature 0.7, top-p = 0.95, and 16 responses per problem.These settings are reported as the common training and evaluation conditions.
  • Offline SFT references: Offline SFT references are built from the same R1-Qwen source checkpoint using DeepScaleR-1.5B-Preview and JustRL-DeepSeek-1.5B to generate one response per DAPO-Math-17K prompt.Rollouts use temperature 1.0, top-p = 0.95, top-k = −1, and maximum response length 7,168 tokens, with rejection of missing boxed answers or degenerate repetition.
  • Random-mask baselines: Density-matched random-mask baselines match only the reference mask’s global active-parameter fraction and sample coordinates uniformly within fixed-size chunks using seed 42.This procedure is applied to both R1-Qwen and Qwen2.5-VL, with allocation approximately proportional to floating-point tensor size.

D.2 Delta Metrics and Their Interpretation … D.7 Benchmark-Wise Breakdown of the Subnetwork Intervention

Across OPD checkpoint analyses, updates are measured in loaded bfloat16 representations and exhibit sparse, spectrally structured, source-relative geometry. Training trajectories and interventions show that the discovered OPD subnetwork closely preserves full-training benchmark performance, unlike random masks.

  • D.2 Delta Metrics and Their Interpretation: Checkpoint deltas are computed as Wtrained − Wsrc after bfloat16 casting, with float32 norms, SVDs, and energy statistics measuring deployed-representation changes.Visible sparsity uses the absolute 10^-5 convention, while relative-threshold sparsity scales cutoffs by each source tensor’s RMS.
  • D.2 Delta Metrics and Their Interpretation: Source-mask overlap compares visible update coordinates with principal and low-magnitude source masks, using realized density as the independent-support baseline.Values below baseline indicate avoidance, whereas values above baseline indicate enrichment; low principal overlap signals avoidance of source-leading coordinates.
  • D.3 Update-Support Overlap Metrics: Directional support-overlap scores are asymmetric because each normalizes by a different subnetwork size, while union sparsity reflects union density rather than overlap strength.Independent-support baselines are defined from marginal densities pA and pB.
  • D.4 Additional OPD Update Statistics: Additional statistics cover global norms, exact-zero and visible sparsity, relative-threshold sparsity, coordinate-energy concentration, spectral structure, and source-relative mask overlap across 12 checkpoint pairs.Tables 9–12 use final checkpoint deltas, loaded bfloat16 subtraction, and unweighted medians for eligible spectral and mask-overlap quantities.
  • D.5 Evolution of Sparsity and Stable Rank Along Training: Both runs become less sparse, but SFT stable rank rises from 14.86 to 27.60 while OPD stable rank remains between 6.17 and 6.80.SFT sparsity decreases from 43.09% to 29.02%, whereas OPD sparsity decreases from 76.02% to 70.00%.
  • D.6 Top-1% SVD Reconstruction Protocol: The intervention retains each matrix’s top 1% update singular components, keeps embeddings, heads, biases, and normalization parameters at source values, and controls per-matrix update magnitude.For 196 two-dimensional matrices, qp equals 16 for query, output, and MLP projections; the construction isolates directional quality rather than raw retained energy.
  • D.6 Top-1% SVD Reconstruction Protocol: Reconstructed, full, and base checkpoints are evaluated benchmark-wise on AIME24 and AIME25 using temperature 0.7, top-p = 0.95, 16 responses, and 31,744-token maximum generation.Gain recovery is computed separately against the same Base checkpoint.
  • D.7 Benchmark-Wise Breakdown of the Subnetwork Intervention: On both AIME24 and AIME25, the OPD mask closely tracks full training, whereas the random-mask run generally performs worse.This benchmark-wise pattern matches the averaged result reported for the subnetwork-masked OPD experiment.
Loading 2606.13657v3…