PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation

Yuchen Cai, Ding Cao, Liang Lin, Chunxi Luo, Xin Xu, Kai Yang, Weijie Liu, Saiyong Yang, Tianxiang Zhao, Guangzhong Sun, Guiquan Liu, Junfeng Fang

arXiv:2605.11739v3cs.CL

TL;DR

The parameter-level mechanisms behind OPD’s training efficiency remain poorly understood. This paper characterizes OPD’s module allocation and update-direction structure and proposes EffOPD; across models from 1.5B to 32B parameters, it reports 3× average training acceleration over multiple baselines with comparable final performance.

  • Problem

    The parameter-level mechanisms behind OPD’s training efficiency remain poorly understood.

    Source: §1 · p. 2

  • Method

    The paper characterizes OPD’s foresight through module allocation and early update-direction alignment, then proposes EffOPD to extrapolate along the current update direction.

    Source: §1 · p. 2

  • Results

    3× average training acceleration over multiple baselines across models from 1.5B to 32B parameters, while maintaining comparable final performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Early directional lock-in may serve as a diagnostic signal for monitoring post-training dynamics.

    Source: C Limitations and Future Work · p. 15

    The properties’ applicability to multi-turn agent tasks and multimodal reasoning remains to be examined.

    Source: C Limitations and Future Work · p. 15

Abstract

from arXiv · show

On-policy distillation (OPD) has emerged as an efficient post-training paradigm for large language models. However, existing studies largely attribute this advantage to denser and more stable supervision, while the parameter-level mechanisms underlying OPD's efficiency remain poorly understood. In this work, we argue that OPD's efficiency stems from a form of ``foresight'': it establishes a stable update trajectory toward the final model early in training. This foresight manifests in two aspects. First, at the \textbf{Module-Allocation Level}, OPD identifies regions with low marginal utility and concentrates updates on modules that are more critical to reasoning. Second, at the \textbf{Update-Direction Level}, OPD exhibits stronger low-rank concentration, with its dominant subspaces aligning closely with the final update subspace early in training. Building on these findings, we propose \textbf{EffOPD}, a plug-and-play acceleration method that speeds up OPD by adaptively selecting an extrapolation step size and moving along the current update direction. EffOPD requires no additional trainable modules or complex hyperparameter tuning, and achieves an average training acceleration of $3\times$ while maintaining comparable final performance. Overall, our findings provide a parameter-dynamics perspective for understanding the efficiency of OPD and offer practical insights for designing more efficient post-training methods for large language models.

1 Introduction

The introduction argues that OPD’s efficiency reflects early “foresight”: it suppresses low-utility updates and aligns with final update directions, motivating EffOPD’s extrapolation-based acceleration.

Foresight at the Module-Allocation Level

  • Under equal update-norm constraints, OPD gains more performance than RL by suppressing updates to low-utility modules and concentrating them on reasoning-critical intermediate layers.RL instead accumulates larger update norms in modules that contribute little to performance improvement.Source: §1 · p. 2

Foresight at the Update-Direction Level

  • OPD’s early dominant directions stably align with the final subspace, unlike RL’s dispersed, delayed alignment; after module-wise norm scaling, its 10%-training checkpoint recovers approximately 80% of final reasoning performance.The early alignment suggests OPD captures the final solution’s main structure with limited exploration and correction.Source: §1 · p. 2

EffOPD

  • Across 1.5B–32B models, EffOPD delivers 3× average training acceleration over multiple baselines while maintaining comparable final performance.It linearly extrapolates along the current update direction and requires no additional modules or complex hyperparameter tuning.Source: §1 · pp. 2–3

2 Functional Redundancy Avoidance

OPD’s modular-level “foresight” means forming compact, task-relevant updates early, suppressing changes in low-utility regions while concentrating them in reasoning-critical modules.

2.1 Experimental Setting

  • The study compares RL and OPD updates from a shared initialization across 1.5B–32B models, including pretrained, SFT-tuned, and Thinking-series models.RL methods include PPO, GRPO, and DAPO; OPD trains a student with a pattern-aligned teacher, typically a stronger model from the same family (Li et al., 2026).Source: §2.1 · p. 3

2.2 Parameter Updates & Reasoning Gains

  • OPD achieves higher reasoning gains than RL at equal update norms and reaches the same accuracy with smaller updates, forming a compact, task-relevant pattern early in training.The results indicate that OPD’s efficiency is not merely due to avoiding late-stage redundancy.Source: §2.2 · pp. 3–5
  • Replacing embeddings with base-model embeddings has negligible effect on reasoning, indicating that embedding updates contribute little to reasoning gains.The analysis decomposes updates into embedding, MLP, and attention components.Source: §2.2 · p. 4, Figure 3 · p. 4
  • Both methods rely on similar functional pathways, but OPD suppresses updates in low-sensitivity peripheral layers and concentrates them in middle-layer modules, especially MLPs.Interventions show greater reasoning sensitivity in MLPs than attention modules and an inverted U-shaped pattern across layer positions, with middle layers yielding the largest gains.Source: §2.2 · pp. 4–5, Figure 3 · p. 4

3 Early Low-Rank Lock-in

OPD’s updates concentrate more strongly in dominant low-rank subspaces than RL’s, and those directions stabilize early in training. Experiments show that principal directions carry stronger reasoning gains than tail directions, while early OPD checkpoints benefit from added update magnitude.

3.1 Spectral Concentration of Update Matrix

  • Across model scales, OPD exhibits stronger low-rank structure than RL, with a higher spectral-to-Frobenius norm ratio, lower effective rank, and higher Top-1% subspace norm ratio.For the 8B model, the respective values are 36.8% vs. 32.7%, 2341 vs. 2754, and 94.7% vs. 88.5%; OPD concentrates more update energy in dominant directions despite a smaller overall update norm.Source: §3.1 · p. 5

3.2 Functional Partition of the Update Spectrum: Principal vs. Tail Subspaces

  • With equal norm budgets, both methods recover over 95% of full-model reasoning performance at 10% rank, while OPD outperforms RL across evaluated ranks and model scales.This supports the principal subspace as the primary carrier of reasoning gains and indicates OPD’s principal directions provide more effective signals under the same norm budget.Source: Figure 4 · p. 6, §3.2 · p. 6
  • Tail directions yield limited reasoning recovery: on Qwen2.5-1.5B-DeepSeek, 10% of principal rank raises accuracy from 23.33% to 40.3%, while 50% of tail rank reaches only around 30%.RL has a slight tail-direction advantage, but its tail-subspace norm is approximately 1.6 to 2.5 times OPD’s for limited performance gain, consistent with lower marginal utility.Source: §3.2 · p. 6, Figure 4 · p. 6

3.3 Directional Stabilization and Magnitude Development

  • OPD’s dominant update subspaces align with their final counterparts earlier than RL’s, and scaling a 10%-trained OPD checkpoint recovers approximately 80% of final performance.The scaled checkpoint also has lower KL divergence from the teacher; results indicate early directions are task-relevant and limited update magnitude constrains performance.Source: Figure 5 · p. 7, §3.3 · p. 7
  • OPD’s training primarily accumulates magnitude along early-stabilized dominant directions rather than making large directional adjustments, a pattern termed Early Low-Rank Lock-in.The paper links this property to Functional Redundancy Avoidance: locking into efficient directions early reduces redundant exploration and correction and avoids overlearning redundant information.Source: §3.3 · pp. 7–8

4 Accelerating OPD via Directional Extrapolation

EffOPD accelerates OPD by extrapolating along update directions that stabilize early in training, using lightweight validation to adapt extrapolation and limit performance degradation.

4.1 Method

  • EffOPD searches at exponentially spaced checkpoints, beginning at t = 1, and estimates directions from the initial-to-W1 displacement and then between adjacent checkpoints.These accumulated displacements approximate subsequent update directions because OPD directions remain relatively stable during training.Source: §4.1 · p. 8
  • At each checkpoint, EffOPD tests five increasing extrapolation magnitudes on 50 sampled training examples, accepts candidates that do not lower validation score, and stops at the first failure.If the first candidate fails, it falls back to vanilla OPD; this progressive validation is intended to avoid degradation from excessive extrapolation.Source: §4.1 · pp. 8–9

4.2 Main Results

  • About 10 training steps versus 30–40 for vanilla OPD, EffOPD converges on mathematical reasoning tasks with >3× speedup and a higher performance upper bound.On Qwen3-4B-Non-Thinking, it reaches strong reasoning performance by the 4th step; unlike fixed extrapolation strategies, it adapts the magnitude using validation feedback.Source: §4.2 · p. 9
  • Ablations show lightweight validation filters aggressive steps for stability, validation-set difficulty is not critical, and faster convergence outweighs validation overhead under the same time budget.Different validation-set difficulty levels provide consistent directional signals, indicating validation checks direction effectiveness rather than supplying precise supervision.Source: §4.2 · p. 9, Figure 7 · p. 9

5 Conclusion · A Impact Statement · B Related Work

The conclusion attributes OPD’s efficiency to compact allocation toward high-utility modules and early stabilization of update directions, and presents EffOPD as an acceleration method. The accompanying sections discuss potential benefits and misuse, then situate the work among research on OPD and on-policy training.

5 Conclusion

  • OPD exhibits Functional Redundancy Avoidance and Early Low-Rank Lock-in, concentrating updates on high-utility modules and directions early in training.The paper identifies these properties as the modular- and update-direction-level manifestations of OPD’s “foresight.”Source: §5 · p. 9
  • Across model scales, RL algorithms, and task domains, OPD achieves RL-comparable reasoning gains with compact, structured updates; EffOPD provides up to 3× training speedup while maintaining final performance.EffOPD is a plug-and-play method that leverages early directional stability.Source: §5 · p. 10

A Impact Statement

  • The work may reduce post-training computational costs, but efficient techniques could also enable harmful applications, warranting responsible use, safety evaluation, and deployment safeguards.The authors state that they believe the work conforms with the NeurIPS Code of Ethics.Source: A Impact Statement · p. 15

B Related Work

  • Related work describes OPD as student-generated sampling with dense teacher supervision and surveys its efficiency, applications, theoretical properties, and training dynamics.Agarwal et al. (2024a) describe the paradigm; Qwen3 (Yang et al., 2025) reports higher training efficiency than RLVR; MiMo-V2-Flash (Xiao et al., 2026) and Deepseek-V4 (DeepSeek-AI, 2026) use multi-task OPD. Song & Zheng (2026) survey OPD; Fu et al. (2026) report token-level bias but tighter variance, O(T 2) versus O(T 4); Yang et al. (2026b) establish theoretical equivalence between token-level distillation and RLVR; Li et al. (2026) identify compatible thinking patterns and genuinely novel teacher capabilities as necessary conditions for success.Source: B Related Work · p. 15
  • Prior work on on-policy training examines sampling efficiency, entropy collapse, RL scaling, and rank-1 update dynamics, including AlphaRL’s 3× training acceleration.Yue et al. (2025) find RL improves pass@1 sampling efficiency but not reasoning ability; Cui et al. (2025b) describe entropy collapse; Tan et al. (2026) report a power-law relationship between test loss, compute, and data volume, with larger models showing higher learning efficiency; Cai et al. (2025) identify Rank-1 dominance and Rank-1 linear dynamics and introduce AlphaRL.Source: B Related Work · p. 15
  • Unlike prior work on RL’s low-rank trajectories, this study attributes OPD’s efficiency advantage to the synergy of modular redundancy suppression and early directional stabilization.Chen et al. (2026) train a predictor using the early Rank-1 subspace to forecast subsequent optimization-subspace directions.Source: B Related Work · p. 15

C Limitations and Future Work · D Preliminaries and Experimental Setup · D.1 Preliminaries

The paper identifies limits in validating OPD’s properties and in its local theoretical analysis, while the preliminaries contrast sparse-reward reinforcement learning with on-policy distillation’s dense teacher supervision.

C Limitations and Future Work

  • OPD’s applicability to multi-turn agent tasks and multimodal reasoning remains unexamined, and the local theory does not capture global non-convex post-training behavior.These settings may involve stronger distribution shifts and more complex teacher-student residuals.Source: C Limitations and Future Work · p. 15
  • Future work should model the coupling of the distillation objective, evolving on-policy distribution, and update spectra, and test directional alignment, spectral concentration, and update compactness as diagnostics.The authors suggest these metrics could monitor post-training progress and stability and support more adaptive, efficient OPD.Source: C Limitations and Future Work · p. 15

D.1 Preliminaries

  • Experiments focus on Reinforcement Learning (Zhang et al., 2025a) and On-Policy Distillation.The preliminaries denote the policy being optimized by πθ.Source: §D.1 · p. 16
  • In RL, trajectories come from the current policy; RLVR uses deterministic verifiable rewards without a learned reward model, while a KL constraint limits deviation from a reference policy.The KL constraint strength is controlled by β.Source: §D.1 · p. 16
  • Policy-gradient methods optimize the RL objective using token advantages relative to a baseline, while RLVR rewards are often sparse until the full response is complete.The passages describe the policy-gradient estimate and the practical sparsity of RLVR feedback.Source: §D.1 · p. 16
  • OPD trains on student-generated trajectories by minimizing reverse KL to a fixed teacher, preserving on-policy training (Yang et al., 2026a).A common zero-discount approximation focuses on immediate token-level optimization, yielding dense signals at every token position and greater training efficiency than sparse-reward RLVR.Source: §D.1 · p. 16

D.2 Experimental Setup · E Property 1 Additional Experiment

The experimental setup spans multiple model scales, pretrained checkpoints, and RL algorithms, using capability-aligned teachers for OPD. The appendix specifies shared and method-specific training configurations, datasets, and hardware.

D.2 Experimental Setup

  • Experiments span 1.5B–32B models, including public pretrained checkpoints and locally trained models, and compare PPO, GRPO, and DAPO across scales.For OPD students, teachers are the RL-tuned versions of their own base models; Qwen3-8B-Base also uses Qwen3-14B-Base-DAPO as a teacher.Source: §D.2 · p. 17
  • RL methods share 2,048-token prompts, 20,480-token responses, 32-sample mini-batches, 16 gradient-accumulation iterations, and an effective batch size of 512.Training uses Float16, n = 16 rollout outputs per prompt, a 1 × 10−6 learning rate with warmup, and gradient clipping of 1.0.Source: §D.2 · p. 17
  • RL training monitors average batch reward and stops when it fails to improve for five consecutive steps.Locally trained RL models use a Verl-adapted codebase and their corresponding training setups.Source: §D.2 · p. 17
  • GRPO uses 0.2 high and low clipping ratios and a 0.001 KL coefficient, while DAPO uses ϵlow = 0.2 and ϵhigh = 0.28, with KL terms removed.DAPO also uses clip-higher, dynamic sampling, token-level policy-gradient loss, and overlong reward shaping; settings follow DeepSeek-AI et al. (2025) and Yu et al. (2025), respectively.Source: §D.2 · p. 17
  • RLVR on Qwen3-14B-Base uses DeepMath-103K and MATH-12K, while Qwen3-14B rollout and training use non-thinking mode and the built-in chat template.The specified template prompts step-by-step reasoning and requests a final answer inside \boxed{} brackets.Source: §D.2 · p. 17
  • OPD uses 2,048-token prompts, 16,384-token responses, a prompt batch size of 1,024, n = 1 rollout output, a 1 × 10−6 learning rate without warmup, and three epochs.OPD follows Yang et al. (2026a); runs use 8× or 32× H20 96GB GPUs.Source: §D.2 · p. 17

E.1 Additional Experiment

Additional experiments show that OPD’s parameter efficiency persists across model scales, RL algorithms, datasets, and architectural components, while its embedding updates remain compact.

E.1 Additional Experiment

  • Across 1.5B–32B models, OPD achieves reasoning performance comparable to RL with substantially smaller parameter update norms.The results support that this efficiency generalizes across model sizes and reflects suppression of functionally redundant updates.Source: §E.1 · p. 19
  • Across examined RL algorithms, OPD consistently has superior parameter update efficiency throughout training, with comparable or superior reasoning performance.The advantage holds regardless of the teacher RL method’s learning dynamics or convergence behavior.Source: §E.1 · p. 19
  • Across model scales, OPD embedding updates show smaller shifts and higher similarity to base representations than RL.This indicates more constrained updates in functionally peripheral embedding layers, avoiding unnecessary drift.Source: §E.1 · p. 19
  • Across broader datasets and algorithmic settings, results support negligible embedding contribution, middle-layer MLP dominance, and redundancy suppression across architectural components.These findings reinforce that the component-level properties are stable rather than artifacts of specific experimental conditions.Source: §E.1 · p. 19

E.2 Detailed Setup of Sliding-Window Intervention Analysis · F Property 2 Additional Experiment

The sections describe a sliding-window intervention for isolating layer- and module-level update contributions, alongside comparisons showing OPD’s reasoning gains relative to RL at different update-norm conditions.

E.2 Detailed Setup of Sliding-Window Intervention Analysis

  • Localized parameter-update injection isolates the marginal functional contribution of updates at different depths and supports layer- and module-level attribution of performance changes.The analysis is intended to examine whether redundant RL updates concentrate in functionally non-critical regions.Source: §E.2 · p. 20
  • Each Transformer layer has Attention and MLP modules, and the intervention window extends eight layers on either side of its center, up to 17 layers, truncated at model boundaries.Neighboring layers are included to reduce the high variance of single-layer interventions while retaining spatial resolution.Source: §E.2 · p. 20
  • Two intervention types replace only Attention or MLP parameters within the window, keeping all other parameters fixed to the base model.Source: §E.2 · p. 20
  • The analysis evaluates valid window centers on MATH500, averaging accuracy across four independent forward passes for each intervened model.The stated centers run from l = 1 through L − 8.Source: §E.2 · p. 20

F Property 2 Additional Experiment

  • Figure 8 reports higher reasoning gains for OPD than RL at an identical update norm, while Figure 9 reports equal reasoning accuracy with smaller OPD update norms.Figure 8 compares final checkpoints; Figure 9 compares intermediate checkpoints throughout training.Source: Figure 8 · p. 21, Figure 9 · p. 21

E.2 Detailed Setup of Sliding-Window Intervention Analysis

  • Figure 10 depicts embedding-layer replacement effects, layer-wise update norms, and RL reasoning accuracy after sliding-window intervention on MATH500.Source: Figure 10 · p. 22

F.1 Geometric Metrics for Parameter Update Matrix

This section defines four complementary metrics for characterizing the geometric structure of a parameter update matrix through its singular-value decomposition. Together, they describe update magnitude and how update energy is distributed across singular directions.

F.1 Geometric Metrics for Parameter Update Matrix

  • The update matrix is decomposed as ΔW = UΣV⊤, with rank r and singular values σ1 ≥ σ2 ≥ ··· ≥ σr > 0.The metrics are averaged across all MLP and attention matrices for each model.Source: §F.1 · p. 24
  • The spectral norm is the largest singular value, σ1, measuring the update magnitude along its dominant direction.It corresponds to the maximum amplification induced by ΔW on any input vector.Source: §F.1 · p. 24
  • The spectral-to-Frobenius norm ratio ρ measures leading-direction dominance; values near 1 indicate concentration along a single direction.Smaller values indicate update energy is distributed across multiple directions.Source: §F.1 · p. 24
  • Effective rank measures the entropy of the singular-value spectrum; a smaller value indicates rapid spectral decay and low-dimensional concentration.A larger effective rank indicates a more diffuse distribution of update energy.Source: §F.1 · p. 24
  • The Top-1% subspace norm ratio measures the share of total update energy captured by the leading 1% of singular directions (Cai et al., 2025).A value near 1 indicates the update is effectively confined to an extremely low-dimensional subspace.Source: §F.1 · p. 24

F.2 Cosine Similarity Analysis of Subspaces

The analysis compares early and final Rank-1 update subspaces, finding that OPD establishes stable dominant directions early in training while RL’s directions remain unstable.

F.2 Cosine Similarity Analysis of Subspaces

  • The analysis uses SVD to compare early-stage and final-stage Rank-1 update subspaces via cosine similarity as a measure of directional lock-in.The Rank-1 subspace corresponds to the strongest singular direction and captures the primary low-dimensional structure of update energy.Source: §F.2 · p. 25
  • OPD’s early-to-final subspace cosine similarity reaches up to 0.9 in intermediate layers, while RL’s remains consistently low across modules.The results indicate that OPD stabilizes dominant directions early, whereas RL continues exploring and correcting before converging.Source: §F.2 · p. 25
  • Early stabilization in reasoning-critical intermediate modules supports Property 1: OPD reinforces stable, task-relevant directions while minimizing redundant parameter movement.This compact update pattern is linked to higher parameter efficiency and performance improvements.Source: §F.2 · pp. 25–26

F.3 Trajectory Evolution of Subspaces · F.4 Scaling Effects on Accuracy and Distribution Alignment

OPD’s subspace trajectories are more concentrated and stable than RL’s, while scaling early updates improves accuracy and student–teacher alignment within an intermediate range. The analyses link these effects to early update directions that already encode useful structure and can be partially unlocked by scaling.

F.3 Trajectory Evolution of Subspaces

  • Across modules, t-SNE projections show OPD’s Rank-1 subspaces following narrower, smoother near-linear trajectories than RL’s dispersed, irregular paths.The visualizations use representations from training checkpoints.Source: §F.3 · p. 26
  • OPD consistently has higher EVR0:2 than RL, indicating that its checkpoint representations vary within a more compact, low-dimensional subspace.The analysis uses PCA on checkpoint-wise representation vectors and measures the cumulative variance explained by the first two principal components.Source: §F.3 · pp. 26–27
  • The analyses interpret OPD’s concentrated updates as an implicit low-rank bias that compresses parameter dynamics and stabilizes update directions early.Updates favor a few stable, effective directions rather than indiscriminately exploring the high-dimensional parameter space.Source: §F.3 · p. 27

F.4 Scaling Effects on Accuracy and Distribution Alignment

  • Accuracy improves as β rises, plateaus near β≈0.8, and degrades above approximately 1.2; Figure 14 identifies 0.8 ≤ β ≤ 1.2 as the optimal range.The plateau reflects limits of the early subspace, while excessive scaling may amplify noise or irrelevant components.Source: Figure 14 · p. 28, §F.4 · p. 28
  • KL divergence falls as β increases, stabilizes over the accuracy plateau, and rises again above 1.2, with low KL accompanying accuracy saturation and strong student–teacher alignment.The passage interprets the initial KL reduction as directly driving accuracy improvement.Source: §F.4 · p. 28
  • Early update directions already capture dominant structure in the teacher distribution and align with the final solution, so appropriate scaling can reduce the student–teacher gap.The early subspace’s norm may be insufficient, but scaling partially unlocks its encoded representational capacity.Source: §F.4 · p. 28
  • Scaling the early checkpoint’s norm to match the final model produces richer, finer-grained reasoning steps and clearer logical progression than the unscaled checkpoint.The scaled responses exhibit reasoning habits more similar to the teacher’s.Source: §F.4 · p. 29

F.5 A Local Geometric View of OPD Dynamics

This section examines OPD dynamics through a local geometric lens, modeling OPD as a quadratic optimization problem to characterize early low-rank updates, module-wise suppression, and more concentrated dynamics than RL. It also presents t-SNE visualizations of U1 trajectories under DAPO and OPD across MLP and attention modules.

Local approximation

  • For small early-training displacements, linearizing student logits and approximating reverse KL locally yields a convex quadratic objective when the Fisher matrix is evaluated near the base model.The base-model Fisher approximation is consistent when student logits remain near the base logits and the teacher is not too far from the base model.Source: §F.5 · pp. 29–30

Directional dynamics

  • When the driving term concentrates in a top-k eigenspace and a spectral gap exists, OPD identifies dominant update directions early and mainly increases their magnitude thereafter.Higher-curvature directions saturate faster; the finite-step tail remains small when the driving term has little mass outside the top-k subspace.Source: §F.5 · pp. 31–32
  • Teacher residuals concentrated on functionally important tokens, filtered by the Fisher matrix and mapped through the Jacobian, help concentrate OPD’s parameter-space driving term.The passage gives key reasoning tokens, intermediate reasoning steps, answer tokens, and formatting tokens as examples.Source: §F.5 · p. 32

Module-wise suppression

  • With weak cross-module coupling, modules weakly coupled to the teacher residual receive negligible updates; embedding and bottom/top transformer layers are empirically suppressed.The module-wise condition is that the module’s driving term is near zero and cross-module coupling terms are not dominant.Source: §F.5 · p. 32

Comparison with RL

  • Unlike RL’s often sparse scalar rewards and diffuse, full-support token updates, OPD uses dense token-level residuals, yielding lower-noise, smoother optimization trajectories in practice.The passages expect Tr(Σ_RL) > Tr(Σ_OPD) in practice; RL’s trajectory-dependent credit assignment also raises gradient variance.Source: §F.5 · p. 33

Local geometric summary

  • OPD’s local dynamics explain early low-rank lock-in and functional redundancy avoidance, while producing denser, lower-variance, more directionally concentrated updates than RL.The summary attributes these properties to a concentrated driving term and weakly coupled modules receiving negligible updates.Source: §F.5 · p. 34

F.5 A Local Geometric View of OPD Dynamics

  • The MLP visualizations cover trajectories for MLP, GATE, and UP modules under both DAPO and OPD.The supplied captions identify the plotted trajectories but do not state a comparative geometric finding.Source: Figure 15 · p. 42, Figure 16 · p. 43, Figure 17 · p. 44
  • Attention-module visualizations show U1 trajectories for Q, K, and V under both DAPO and OPD.The supplied captions identify the plotted module types but do not report their visual patterns.Source: Figure 21 · p. 48, Figure 22 · p. 49, Figure 23 · p. 50
  • A further pair of t-SNE visualizations covers U1 trajectories for the overall attention modules under DAPO and OPD.The captions specify scope only, without a stated result about trajectory alignment or separation.Source: Figure 27 · p. 54, Figure 28 · p. 55

Related papers

Loading 2605.11739v3…