PaperlayerView exact revision on arXiv

Source-linked AI summary

SAF-OPD: Stable Advantage Fusion for On-Policy Distillation

Yifan Ding, Xincheng Wei, Yoshua Y. Li, Ziheng Li, Yuquan Lu, Siyu Zhang, Dongsheng Ma, Rongxiang Weng, Xunliang Cai, Yun Chen

arXiv:2607.29209v1cs.LGcs.AI

TL;DR

Fixed-coefficient fusion of RLVR and OPD suffers magnitude and temporal miscalibration that can destabilize training and limit exploration. SAF addresses both with a switchable four-stage pipeline, improving aggregate scores over fixed-coefficient GRPO+OPD by 0.51–2.70% across six model–domain settings while avoiding entropy collapse.

  • Problem

    Fixed-coefficient RLVR–OPD fusion leaves magnitude and temporal miscalibration unresolved, despite their complementary verified and teacher-relative signals.

  • Method

    SAF controls OPD advantages through a switchable four-stage pipeline pairing sparsify-then-compress magnitude control with warm-up-then-anneal temporal control.

  • Results

    0.51–2.70% gains over fixed-coefficient GRPO+OPD occur across all six model–domain settings, with SAF consistently improving aggregate scores and avoiding entropy collapse.

  • Takeaways & Limitations

    Controlling the OPD signal is more effective than discarding it or injecting it at full strength across the evaluated reasoning and code-generation settings.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) broadcasts a single response-level reward to every token, while on-policy distillation (OPD) scores each token against a stronger teacher for a dense advantage but caps performance at teacher quality and discourages exploration beyond it. Their complementarity makes combining RLVR and OPD promising, but we find that fusing the two advantages with a fixed coefficient triggers entropy collapse from two miscalibrations: a magnitude mismatch, where token-level OPD advantages can spike far beyond the bounded RLVR advantage and erase its signal, and a temporal mismatch, where sustained full-strength OPD keeps pulling the student toward the teacher and limits exploration needed to surpass it. We propose SAF, a Stable Advantage Fusion framework that resolves both issues via a lightweight, four-stage pipeline applied only to the OPD advantage: a sparsify-then-compress mechanism for magnitude control paired with a warm-up-then-anneal mechanism for temporal control, with each stage independently switchable and adding negligible overhead. Instantiating RLVR with GRPO, we evaluate SAF across seven mathematical reasoning and code generation benchmarks with Qwen3-1.7B/4B/8B: SAF avoids entropy collapse and consistently outperforms fixed-coefficient GRPO+OPD fusion, improving the aggregate score by 0.51-2.70% across all six model-domain settings while achieving more stable training.

1 Introduction

RLVR with GRPO provides bounded response-level feedback, whereas OPD provides dense token-level teacher feedback but can cap exploration at teacher quality. SAF addresses their magnitude and temporal miscalibration with controllable advantage processing, avoiding entropy collapse and improving performance over fixed-coefficient fusion.

  • Motivation: GRPO broadcasts one group-normalized response-level advantage to every token, while OPD recomputes dense token-level advantages from student–teacher log-probability gaps.GRPO’s broadcast is coarse, whereas OPD supplies feedback on problems the student cannot yet solve.
  • Motivation: OPD can reward teacher matching regardless of correctness, implicitly capping performance at teacher quality and weakening pressure to explore beyond it without verifier rewards.Sustained teacher matching offers no pressure to exceed the teacher absent an explicitly reshaped reward.
  • Problem: Fixed-coefficient fusion creates magnitude and temporal mismatches: unbounded, heavy-tailed OPD tokens can dominate bounded RLVR signals, while sustained teacher guidance limits later exploration.These mismatches manifest in practice as early, persistent policy-entropy collapse and accuracy plateaus below stronger performance.
  • SAF: SAF is a lightweight, four-stage pipeline combining sparsify-then-compress magnitude control with warm-up-then-anneal temporal control.Each stage is independently switchable, and the annealing horizon is set relative to the observed warm-up endpoint.
  • Results: 0.51–2.70% aggregate-score gains over fixed-coefficient GRPO+OPD occurred in all six model–domain settings across seven mathematical-reasoning and code-generation benchmarks.Experiments used Qwen3-8B, Qwen3-4B, and Qwen3-1.7B, and training analysis found that SAF avoids entropy collapse while preserving exploration capacity.

2 Related Work

RLVR provides coarse response-level feedback broadcast across tokens, whereas OPD supplies dense token-level teacher feedback but can retain unbounded token-wise influence. SAF addresses this mismatch with matched-granularity controls that bound token influence and adapt OPD strength using online student–teacher KL divergence.

  • RLVR and OPD: RLVR commonly uses GRPO to broadcast a group-normalized response-level advantage to every token, while follow-up methods refine normalization, clipping, or weighting without changing its coarse granularity.RLVR is critic-free and response-level; Dr.GRPO, DAPO, and GSPO modify scalar processing rather than token granularity.
  • RLVR and OPD: OPD scores student-generated trajectories with teacher token probabilities, reducing train–generation mismatch and providing dense token-level feedback.The passage contrasts OPD with offline distillation on fixed teacher targets and describes OPD as a common post-training stage.
  • SAF: SAF pairs each mismatch with a matched-granularity mechanism, bounding each token’s influence and adapting OPD strength from online student–teacher KL divergence.This contrasts with heavy-tailed OPD magnitudes that can dominate after coarse gating.

3 Method

SAF stabilizes GRPO+OPD fusion by modifying only the OPD advantage through four independently switchable stages. It controls token-level magnitude with sparsification and tanh compression, then controls training-time strength with KL-triggered warm-up and linear annealing.

  • Overview: SAF applies four independently switchable stages only to A_OPD, leaving the GRPO advantage unchanged during fusion.The stages comprise top-k% sparsification, bounded tanh compression, KL-triggered warm-up, and linear annealing.
  • Magnitude controller: Top-k% sparsification retains tokens above each sequence’s own magnitude quantile and zeros the rest, adapting filtering to sequence-specific OPD distributions.Setting k = 100 disables filtering entirely.
  • Magnitude controller: Bounded tanh compression confines surviving OPD values to (−c, c) while preserving near-origin ordering, adding negligible overhead and remaining independently switchable.Together, sparsification and compression address the mismatch that lets a few tokens dominate the sequence gradient.
  • Temporal controller: KL-triggered warm-up ramps OPD strength from zero and stops early when sampled student-to-teacher reverse KL has sufficiently decreased.The criterion uses observed KL reduction rather than a fixed step count, accommodating differences across models and tasks.
  • Temporal controller: Linear annealing starts at coefficient 1 after warm-up and decays to c_min over the remaining budget, with earlier KL termination producing more gradual decay.If warm-up is disabled, annealing spans the full training budget.
  • Implementation: SAF is a drop-in replacement for fixed GRPO+OPD fusion, requiring no additional model, loss, or forward pass.Its implementation stores limited controller state and computes one quantile per sequence plus the sampled-KL stopping statistic.

4 Experiments

Experiments across three Qwen3 sizes and seven reasoning/code benchmarks show that SAF consistently outperforms fixed-coefficient GRPO+OPD while avoiding the instability associated with magnitude and temporal mismatches. Training analyses link these gains to preserved entropy, stronger critic scores, longer responses, and improved validation accuracy.

  • Models and training: SAF combines GRPO with OPD for Qwen3-8B, Qwen3-4B, and Qwen3-1.7B students trained against Qwen3-30B-A3B-Instruct-2507.Mathematics uses 57K DeepMath problems, while code uses 25K Eurus-RL-Code problems; verifiable rewards drive GRPO and teacher scores drive OPD.
  • Aggregate results: SAF improves over GRPO+OPD (fixed) by 0.97%, 1.51%, and 1.85% on mathematics for Qwen3-8B, Qwen3-4B, and Qwen3-1.7B, respectively.On code generation, the corresponding gains are 1.67%, 2.70%, and 0.51%.
  • Aggregate results: 49.46% is SAF’s average across six model–domain settings, exceeding GRPO+OPD (fixed), GRPO-only, and OPD-only by 1.54%, 2.71%, and 1.60%, respectively.The aggregate improvement over fixed fusion spans all three model scales and both domains.
  • Benchmark-level results: SAF is best or tied for best on four of seven Qwen3-8B benchmarks and six of seven Qwen3-4B benchmarks, with MBPP+ the main reported exception.For Qwen3-8B, SAF improves over GRPO-only by 2.84% on mathematics and 2.69% on code; for Qwen3-4B, the gains are 1.57% and 2.19%.
  • Training dynamics: 20.3585 is the largest reported OPD extreme versus 2.4749 for the largest co-occurring GRPO magnitude, with most OPD tokens near zero and a small subset one to two orders larger.This magnitude mismatch motivates top-k filtering and tanh compression.
  • Training dynamics: SAF ends near 0.84 critic score versus approximately 0.77 for fixed fusion and produces roughly 5.3k–6.2k-token responses versus about 4.5k under fixed fusion.Fixed fusion reduces actor entropy from approximately 0.35 to 0.30 and keeps it near that level, while SAF continues improving late-training validation accuracy.

5 Conclusion

The conclusion frames SAF as a solution for fusing response-level GRPO and token-level OPD advantages without destabilization. It addresses fixed-coefficient fusion’s magnitude and temporal mismatches through a four-stage, switchable pipeline.

  • 5 Conclusion: SAF targets unstable fusion of response-level GRPO and token-level OPD advantages caused by magnitude and temporal mismatches.OPD advantages can spike beyond bounded GRPO advantages, while full-strength guidance changes as the student converges toward the teacher.
  • 5 Conclusion: A four-stage, switchable SAF pipeline addresses both mismatches through top-k sparsify-then-tanh compression and temporally adjusted OPD guidance.The passage identifies the pipeline as the framework’s mechanism for preventing either advantage from destabilizing the other.

A Detailed Experiment Settings

Experiments distill Qwen3-30B-A3B-Instruct-2507 into Qwen3 students on verified mathematics and code tasks, comparing GRPO-only, fixed GRPO+OPD, OPD-only, and SAF configurations. Shared rollout and optimization settings isolate advantage construction, while SAF applies staged controls to OPD weighting.

  • Training data and models: Qwen3-30B-A3B-Instruct-2507 serves as teacher for Qwen3-8B, Qwen3-4B, and Qwen3-1.7B students trained on DeepMath and Eurus-RL-Code.DeepMath contains 57K filtered problems of difficulty level at least 6, while Eurus-RL-Code contains 25K problems; a rule-based verifier gives reward 1 for correct answers or passing code tests.
  • GRPO-based methods: GRPO-only, GRPO+OPD (fixed), and SAF share optimization and rollout configurations at each student scale and domain, differing only in advantage construction.These methods run for 300 steps on mathematics and 200 steps on code.
  • OPD-only: OPD-only uses separate mathematics and code configurations, with batch size 1,024, one rollout per prompt, and learning rate 1×10^-6.OPD-only runs for 100 mathematics steps and 50 code steps, with maximum response lengths of 16,384 and 8,192 tokens, respectively.
  • Advantage configurations: GRPO+OPD (fixed) uses unit OPD weighting with a 1:1 fusion, whereas SAF starts from the same coefficient and enables its configured controls.SAF recomputes the top-k threshold per response; warm-up lasts at most 100 steps, may stop at relative student–teacher KL decrease 0.2, and annealing reaches zero at the domain-specific training endpoint.

B Training Dynamics: Supplementary Parameter-Space Analysis

Supplementary parameter-space analyses show that OPD concentrates update energy into fewer directions than GRPO, while fixed fusion produces the greatest parameter drift and SAF reins it in without eliminating it. Relative-drift measurements provide a scale-normalized view of these training-regime differences.

  • Parameter-space update geometry: OPD exhibits substantially lower stable rank than GRPO across the first 100 shared optimization steps, indicating more concentrated update energy.Figure 5 analyzes mean stable rank and cross-method singular-subspace overlaps from layer-wise SVD of update matrices.
  • Layer-wise weight drift: Fixed-coefficient fusion drifts the most in parameter space, whereas SAF reins in drift without fully suppressing it.The comparison covers GRPO-only, OPD-only, fixed-coefficient GRPO+OPD, and SAF on Qwen3-4B mathematical reasoning checkpoints.
  • Layer-wise relative weight drift: 2.3×10^-4 to 3.6×10^-4: GRPO-only relative drift rises between steps 50 and 300, a 1.6× increase.Relative drift normalizes by base-parameter norm to reduce layer-size effects.
  • Layer-wise relative weight drift: 2.2×10^-4 to 6.8×10^-4: OPD-only relative drift grows between steps 10 and 100, showing the fastest per-step increase.The OPD-only trajectory uses its 100-step mathematical-reasoning training horizon.

C Student–Teacher KL Computation

The temporal controller estimates student-to-teacher reverse KL on sampled tokens from student-generated responses, using clipping and masked token averaging. The resulting statistic is logged for monitoring and warm-up control without adding actor regularization.

  • KL estimation: The method estimates reverse KL from sampled response tokens, reusing student-generated responses instead of evaluating full-vocabulary KL at every position.For token y_i,t in context s_i,t, it computes a clipped teacher-to-student log-probability ratio.
  • KL estimation: Clipping safeguards the estimator against numerical overflow and extreme finite-sample contributions while recovering the non-negative reverse-KL form without clipping.The unclipped estimator is r − log r − 1, with r = πT(y_i,t | s_i,t)/πθ(y_i,t | s_i,t).
  • Monitoring and control: The scalar is logged as actor/kl_loss and, with kl_loss_coef=0, serves only as an alignment statistic while informing warm-up early stopping through its relative decrease from KL0.use_kl_loss=True enables computation, and the controller uses (KL0 − KLs)/KL0 in the early-stop criterion.
  • Aggregation: Valid token contributions are aggregated with a token mean that excludes prompt and padding positions through the response mask.The response mask m_i,t ∈ {0, 1} selects positions included in aggregation.

D token-level Raw Signal Analysis

This appendix analyzes the raw OPD and GRPO advantages across the first five consecutive training steps. It defines the advantages, selects representative responses, and reports high-magnitude token positions with local and aggregate statistics.

  • Scope: The analysis tracks two unprocessed training advantages over the first five consecutive training steps.The quantities are examined before the sparsify, compress, warm-up, and anneal stages.
  • Advantage definitions: A_OPD_i,t is the token-level OPD advantage, while A_GRPO_i is the response-level GRPO advantage shared by every token in response i.The implementation fields are raw_opd and raw_grpo, respectively.
  • Response selection: Each training step includes responses with the most negative, near-zero, and most positive A_GRPO_i values.These represent incorrect, neutral/mixed, and correct responses, respectively.
  • Reported statistics: For each response, the appendix reports the 12 positions with largest |A_OPD_i,t|, the first 15 response tokens, and summary statistics over all response tokens.Token positions are arranged horizontally in two token-level tables, followed by a compact summary table.

D.1 Training Step 1

This section presents position-ordered i,t entries alongside the first 15 response tokens with context, repeated across three displayed blocks.

  • Three blocks list i,t entries ordered by position.
  • The displayed material pairs position-ordered i,t information with contextualized response-token excerpts.
  • Each block also identifies the first 15 response tokens with context.

D.2 Training Step 2 · D.3 Training Step 3 · D.4 Training Step 4

D.2–D.4 present ordered-by-position token information alongside the first 15 response tokens with context. The supplied passages provide these labels but no further substantive findings or quantitative results.

  • D.2 Training Step 2: D.2 includes ordered-by-position token information.The passage is labeled “i,t | (ordered by position):”.
  • D.2 Training Step 2: D.2 also includes the first 15 response tokens with context.This description appears explicitly in the section’s passage.
  • D.3 Training Step 3: D.3 includes ordered-by-position token information.The passage is labeled “i,t | (ordered by position):”.
  • D.4 Training Step 4: D.4 includes ordered-by-position token information.The passage is labeled “i,t | (ordered by position):”.

D.5 Training Step 5

The analysis identifies a persistent magnitude mismatch between OPD and GRPO advantages across representative responses and consecutive training steps. OPD extremes can substantially exceed GRPO values, allowing salient OPD tokens to dominate fixed 1:1 fusion updates.

  • Magnitude mismatch: 20.3585 was the reported OPD extreme, versus 2.4749 for the largest absolute GRPO advantage among the examined responses.These values illustrate the scale difference between token-level OPD and verifier-derived GRPO advantages.
  • Magnitude mismatch: In fixed 1:1 fusion, salient OPD tokens can locally dominate the fused update because their contributions exceed the response-level GRPO contribution.The selected-token analysis reports this mismatch across response types and consecutive steps.
Loading 2607.29209v1…