PaperlayerView exact revision on arXiv

Source-linked AI summary

GR-OPD: Group-Relative Sequence Credit for On-Policy Distillation

Yushu Wang, Renkai Yang, Jingcao Xu, Zili Geng, Haihong Tang, Yaohui Jin, Hao He

arXiv:2610.23042v1cs.LG

TL;DR

OPD’s sampled-token updates assign credit locally, leaving later feedback unable to revise credit for earlier actions. GR-OPD aggregates and calibrates sequence-level credit; it outperforms vanilla OPD across five settings and maintains stable training under mismatched thinking patterns.

  • Problem

    Sampled-token OPD assigns each action only its local feedback, so later evidence cannot revise credit for earlier decisions.

    Source: §3.1 · p. 3

  • Method

    GR-OPD broadcasts aggregated token feedback as sequence-level credit, discounts later feedback, and uses same-prompt groups to calibrate relative preferences and retained collective teacher pressure.

    Source: §1 · p. 2

  • Results

    GR-OPD outperforms vanilla OPD in macro Avg@K across all five teacher–student settings, with gains of 10.9 and 11.5 percentage points in two matched-pattern settings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Ablations show that retaining an appropriate amount of collective teacher pressure can improve learning, while excessive pressure or a full feedback horizon can destabilize training.

    Source: §1 · p. 2

    Evaluation is limited to mathematical reasoning and Qwen3 models; generalization to broader domains and model families remains to be established.

    Source: §6 · p. 9

Abstract

from arXiv · show

On-policy distillation (OPD) provides token-level teacher feedback, yet common sampled-token updates assign credit locally and omit the effects of earlier actions on subsequent feedback. We introduce \method{} (Group-Relative On-Policy Distillation), which decouples feedback computation from credit assignment by aggregating token feedback into calibrated sequence-level credit. We establish a theoretical basis for this approach by showing that, under ideal on-policy assumptions, broadcasting raw summed feedback to every response token yields an unbiased estimate of the negative gradient of the sequence-level reverse-KL objective. To support effective learning in practice, \method{} discounts later feedback that may become less reliable under distribution shift and uses same-prompt rollout groups to calibrate teacher guidance. It preserves relative response preferences while retaining a controlled fraction of the group mean, which we term collective teacher pressure. Across five Qwen3 teacher--student settings and six mathematical reasoning benchmarks, \method{} outperforms vanilla OPD in macro Avg@K in every setting, including gains of 10.9 and 11.5 percentage points in two settings with matched thinking patterns. It also maintains stable training under mismatched thinking patterns, where vanilla OPD collapses. Ablations show that retaining an appropriate amount of collective teacher pressure can improve learning, while excessive collective teacher pressure or an unrestricted feedback horizon can destabilize training.

1 INTRODUCTION

GR-OPD reframes OPD credit assignment by aggregating token feedback into calibrated sequence-level credit. Its theoretical basis and experiments support this approach, while group calibration and feedback discounting address practical training concerns.

1 INTRODUCTION

  • OPD computes dense teacher feedback on student-generated trajectories, while sampled-token updates use each token’s teacher–student log-probability difference as a local advantage.This lightweight variant avoids retaining full-vocabulary distributions at every position.Source: §1 · p. 1
  • Effective distillation must organize dense teacher feedback into learning signals and adapt those signals to student capabilities and reasoning patterns.Prior work addresses update variance, feedback reliability, near-future guidance, teacher-specific preferences, and systematic model-family or style discrepancies.Source: §1 · pp. 1–2
  • GR-OPD aggregates token-level feedback into sequence-level credit, discounting later feedback and calibrating shared teacher pressure against same-prompt group preferences.It preserves relative response preferences while retaining a controlled fraction of the group mean, termed collective teacher pressure.Source: §1 · p. 2
  • Under ideal on-policy assumptions, broadcasting summed teacher–student log-ratios to every token recovers the full reverse-KL descent gradient in expectation.This assigns earlier actions credit for later feedback, beyond local token matching.Source: §1 · p. 2
  • GR-OPD outperforms vanilla OPD in macro Avg@K across all five teacher–student settings, with stable training under mismatched thinking patterns.Two matched-pattern settings show gains of 10.9 and 11.5 percentage points; ablations find that excessive collective teacher pressure or a full feedback horizon can destabilize training.Source: §1 · p. 2

2 RELATED WORK

Related work develops OPD across settings and examines sequence credit, group-relative signals, teacher calibration, and feedback reliability. GR-OPD aggregates teacher feedback across each student-sampled response and uses same-prompt groups to calibrate its sequence-level signal.

2 RELATED WORK

  • MiniLLM and Distributional DAgger distinguish local action matching from actions’ influence on future state visitation; GR-OPD aggregates feedback across a response so later feedback informs earlier actions.TOPD instead uses near-future teacher continuations and optimal-transport alignment to identify trajectory divergence and guide local updates.Source: §2 · p. 3
  • Group-relative methods use multiple responses to a prompt for relative signals, while GR-OPD applies same-prompt groups to relative preferences in trajectory-level teacher feedback.GC-OPD separately normalizes trajectory-level OPD scores and verifier rewards, then distributes their disagreement across token-level advantages.Source: §2 · p. 3
  • Teacher-signal calibration methods adjust different components of guidance; GR-OPD retains a controlled fraction λ of the shared same-prompt mean, termed collective teacher pressure.The cited approaches include teacher-family offsets, recurring style-related disagreement, and teacher log-probability changes from pre-tuning checkpoints.Source: §2 · p. 3
  • Prior work addresses potentially less reliable supervision on longer student-generated prefixes through rollout truncation, trajectory filtering, or position-dependent weighting.These approaches motivate accounting for feedback reliability and optimization granularity when converting dense supervision into action-level learning signals.Source: §2 · p. 3

3 FROM TOKEN CREDIT TO SEQUENCE CREDIT

The section motivates sequence-level credit as a way to let later teacher feedback revise credit for earlier actions, then analyzes how broadcasting summed feedback relates to the reverse-KL gradient. Under on-policy assumptions, this broadcast recovers the complete gradient in expectation.

3.1 MOTIVATION

  • Sequence aggregation can reverse locally misleading feedback: in the illustration, local feedback favors incorrect B, while trajectory credit favors correct A.Sampled-token OPD assigns each action only its local feedback and cannot use evidence revealed later to revise an earlier decision.Source: §3.1 · p. 3, Figure 1 · p. 4
  • Sequence-mean teacher feedback distinguishes correct from incorrect responses under cross-pattern and matched-pattern settings, with reported AUROCs of .75 and .73, respectively.Across five settings, JH(H) measures how strongly scores from the first H tokens favor correct responses; on fixed responses it peaks around 1,500–3,000 tokens, with a cube-root horizontal scale.Source: §3.1 · pp. 3–4, Figure 1 · p. 4

3.2 UNBIASED TRAJECTORY BROADCAST

  • An early action affects both immediate teacher–student agreement and later prefixes; the common sampled-token update keeps only local action matching, omitting future state-visitation credit.The analysis separates the reverse-KL gradient into these local and future components.Source: §3.2 · p. 4
  • Under on-policy sampling, broadcasting summed feedback R to every response token yields an unbiased estimate of the negative reverse-KL gradient.The sequence-level signal preserves the complete gradient in expectation; the derivation accounts for both the sampling distribution and R’s dependence on θ.Source: §3.2 · pp. 4–5

4 CONSTRUCTING RELIABLE GROUP-RELATIVE SEQUENCE CREDIT

GR-OPD turns token-level teacher feedback into calibrated sequence-level credit, discounting later feedback and calibrating scores against same-prompt rollouts. Its estimator broadcasts each response’s advantage to its tokens in a PPO-style update.

4.1 FROM SEQUENCE SCORES TO STABLE LEARNING SIGNALS

  • Feedback–correctness association peaks between roughly 1,500 and 3,000 tokens; discounted scoring retains full-response evidence while reducing the influence of later positions.The observed peak varies by teacher–student pairing.Source: §4.1 · p. 5
  • Directly optimizing discounted trajectory scores destabilizes all five settings, motivating calibration that preserves relative preferences while retaining a controlled amount of the group mean.Verifier scores rise briefly and then decline as mean response length approaches the generation cap.Source: §4.1 · p. 5

4.2 THE GR-OPD ESTIMATOR

  • GR-OPD scores same-prompt rollouts with discounted feedback, calibrates relative preferences with a controlled fraction of collective teacher pressure, then broadcasts advantages for a clipped update.The common within-group standard deviation scales the two calibration components; γ controls feedback horizon and λ controls retained collective teacher pressure.Source: Figure 2 · p. 6, §4.2 · p. 6

5 EXPERIMENTS

Across five Qwen3 teacher–student pairings, GR-OPD improves macro Avg@K over vanilla OPD and remains stable in mismatched-thinking settings. Ablations show that collective teacher pressure and feedback horizon require calibration: excessive pressure or an unrestricted horizon can destabilize training.

5.1 SETUP

  • Experiments use five Qwen3 pairings, DAPO-Math training prompts, and six mathematical reasoning benchmarks comprising 1,042 problems.The reported metric is the macro average of per-problem sample accuracy (Avg@K); runs use 221 steps and a maximum response length of 28,672 tokens.Source: §5.1 · p. 7

5.2 MAIN RESULTS

  • GR-OPD raises macro Avg@K and every displayed suite score across all five pairings, while remaining trainable where vanilla OPD collapses.Matched-pairing macro Avg@K rises from 23.333 to 27.800, 34.118 to 45.066, and 30.320 to 41.774; cross-pattern GR-OPD reaches 29.571 and 24.463.Source: §5.2 · p. 7, Table 1 · p. 7

5.3 CALIBRATING COLLECTIVE TEACHER PRESSURE

  • In matched Instruct pairings, λ=.25 or .2 sustains verifier-reward gains, but full pressure retention (λ=1) leads to length growth and reward collapse.For a mismatched pairing, even λ=.15 causes deterioration; λ=.05 improves stability, but final verifier reward remains below λ=0. The supported amount depends on teacher–student compatibility, not a universal retention fraction.Source: §5.3 · pp. 8–9
  • Table 2 reports batch-level normalized and retained pressure magnitude proxies; cross-pattern runs show larger normalized pressure than matched runs.These are batch-level proxies, not per-prompt normalized pressure magnitudes or signed shared components.Source: Table 2 · p. 9

5.4 CONTROLLING THE FEEDBACK HORIZON

  • With λ fixed, the full horizon (γ=1) eventually brings declining reward and growing responses in matched pairings, while γ=.99 continues improving with more gradual growth.In a mismatched pairing with λ=0, γ=1 also leads to reward decline and length growth, whereas γ=.99 continues to improve with relatively stable length.Source: Figure 3 · p. 8, §5.4 · p. 9
  • Under γ=1, late feedback can become associated with response length and less aligned with verifier reward, potentially reinforcing continuations that do not advance the solution.Discounting with γ=.99 reduces later log-ratios’ influence while retaining earlier downstream evidence for credit assignment.Source: §5.4 · p. 9

6 CONCLUSION

GR-OPD turns token-level teacher feedback into calibrated sequence-level credit, with theoretical grounding under ideal on-policy assumptions. Experiments show consistent gains over vanilla OPD on mathematical reasoning benchmarks, while the evaluation scope and token-level credit allocation remain limitations.

6 CONCLUSION

  • GR-OPD discounts later feedback and calibrates sequence credit by preserving relative response preferences while controlling retained collective teacher pressure.Under ideal on-policy assumptions, broadcasting summed teacher feedback recovers the negative gradient of the sequence-level reverse-KL objective in expectation.Source: §6 · p. 9
  • Across five Qwen3 teacher–student settings, GR-OPD consistently improves over vanilla OPD on mathematical reasoning benchmarks and remains stable under mismatched thinking patterns.The evaluation is limited to mathematical reasoning and Qwen3 models.Source: §6 · p. 9
  • Evaluation is limited to mathematical reasoning and Qwen3 models, so generalization to broader domains and model families remains to be established.The method also broadcasts a single sequence-level credit rather than explicitly differentiating credit among response tokens.Source: §6 · p. 9

AI USE STATEMENT

The authors describe the uses of language models in preparing the manuscript and state that they reviewed the assisted work and take responsibility for the final manuscript.

AI USE STATEMENT

  • Language models assisted with literature, experimental design, result interpretation, manuscript drafting, mathematical exposition, figures, and code; the authors reviewed the work and take responsibility.Source: AI USE STATEMENT · p. 10

A PROOFS

The proofs decompose sequence reverse-KL credit into local matching and future state-visitation effects, and examine how weighted feedback, group calibration, and practical estimator choices affect the resulting claims.

A.1 SEQUENCE-GRADIENT DECOMPOSITION; A.2 ALTERNATIVE RETURN-TO-GO PROOF OF PROPOSITION 1

  • The gradient separates local action matching from future state-visitation credit, and the return-to-go and broadcast identities establish their relationship under the stated on-policy derivation.The alternative proof shows that the past-feedback score term has expectation zero, yielding the same raw-return gradient identity.Source: §A.1 · p. 12, §A.2 · p. 13

A.3 TAIL INFLUENCE UNDER EXPONENTIAL DISCOUNTING

  • Exponential discounting bounds the influence of later feedback, but the bounds do not establish that later feedback is useful or guarantee an optimal discount or stable updates.For nonincreasing weights, the tail weight after position H is at most (L − H)/L; at γ = 1 it equals (L − H)/L.Source: §A.3 · p. 13

A.4 WEIGHTED OBJECTIVES

  • Nonincreasing fixed position weights correspond to nonnegative combinations of prefix-KL objectives, while the practical score's normalization, group statistics, and clipping fall outside the fixed-weight result.The proof assumes a fixed teacher, on-policy student sampling, common sequence support, and fixed maximum-length padding.Source: §A.4 · pp. 13–14

A.5 ALGEBRAIC INTERPRETATION OF COLLECTIVE TEACHER PRESSURE

  • For a fixed group of scores and standard deviation, λ changes the shared offset while preserving pairwise advantage differences and response ordering, but this does not ensure invariant PPO gradients or training trajectories.The shared offset is signed and may raise or lower all advantages together; the decomposition does not identify separate semantic sources of guidance.Source: §A.5 · p. 14

B PRACTICAL ESTIMATOR BOUNDARIES

  • The practical estimator's realized-length normalization, group-standard-deviation scaling, clipping, and repeated PPO updates change the gradient estimator, so raw-return unbiasedness does not transfer to the practical loss.The expectation result applies to raw returns and a single score-function update.Source: B PRACTICAL ESTIMATOR BOUNDARIES · p. 15

C EXPERIMENTAL DETAILS; C.1 TRAINING DATA

  • The experiments use five Qwen3 pairings with distinct Base and Instruct reasoning protocols, and train on the English DAPO-Math-17k-Processed configuration without additional difficulty or correctness filtering.The training data description specifies 14,116 canonical training questions; a fixed-response diagnostic selects 1,024 questions and uses a 512-question confirmation subset.Source: C EXPERIMENTAL DETAILS · p. 15, §C.1 · p. 15

C.2 TRAINING CONFIGURATION

The training configuration specifies rollout and optimization batch structure, separate feedback and advantage clipping, EOS alignment for selected pairings, and pinned model and software versions.

C.2 TRAINING CONFIGURATION

  • Each rollout batch contains 64 prompts with eight responses each, and its 512 responses form eight optimization minibatches and eight optimizer updates per rollout step.Training uses one PPO epoch per rollout batch.Source: §C.2 · p. 15
  • GR-OPD clips token-level feedback before trajectory aggregation and clips the calibrated teacher advantage separately before the policy update.Both value-clipping operations use [−5, 5] and are distinct from PPO probability-ratio clipping.Source: §C.2 · p. 15
  • For Base-student/Instruct-teacher pairings, teacher-scoring EOS targets are mapped to align termination probabilities while preserving the sampled student response.The table summarizes training configuration; model and tokenizer checkpoints are pinned to public revisions, and the software versions are recorded.Source: §C.2 · pp. 15–16, Table 3 · p. 16, Table 4 · p. 16

C.3 EVALUATION PROTOCOL

The evaluation protocol defines suite-level and macro Avg@K, answer verification, and reference capabilities, then describes a fixed-response diagnostic comparing teacher-score horizons.

C.3 EVALUATION PROTOCOL

  • Macro Avg@K is the unweighted mean of six suite scores, with evaluation covering 1,042 questions in total.Avg@K is the fraction of correct responses among K samples per question, averaged within a suite.Source: §C.3 · p. 16, Table 5 · p. 16
  • Correctness is determined by extracting and verifying the final boxed or parsed mathematical answer, with extraction failures, parsing errors, and timeouts scored zero.Responses reaching the generation limit remain in evaluation and are judged using the same procedure.Source: §C.3 · p. 17

C.4 TEACHER AND STUDENT BASE CAPABILITIES

  • Pre-OPD reference scores are reported with different generation caps and are not matched method comparisons.Table 6 reports Macro Avg@K and mean response length averaged equally over the six suites.Source: §C.4 · p. 17, Table 6 · p. 17

C.5 FIXED-RESPONSE HORIZON DIAGNOSTIC

  • The fixed-response horizon diagnostic compares standardized teacher scores across prefix lengths on the same responses; positive values indicate higher scores on correct responses on average.It averages groups within prompts and prompts equally, and does not compare policies trained with different horizons.Source: §C.5 · p. 17

C.6 TRAINING DIAGNOSTICS FOR FEEDBACK HORIZON

The training diagnostic tracks response length, verifier reward, and their associations with trajectory scores under two feedback horizons; the observed correlations describe evolving batches rather than causal effects.

C.6 TRAINING DIAGNOSTICS FOR FEEDBACK HORIZON

  • The diagnostics track batch means and correlations pooled across responses, with centered moving averages used to display the curves.The visualization uses seven-step windows for length and correlations, and an eleven-step window for verifier reward.Source: §C.6 · pp. 17–18
  • With γ = 1, response length grows as trajectory scores become increasingly associated with length, while their rank association with verifier reward turns negative; with γ = .99, length grows more slowly and score–verifier correlation remains positive.The plotted panels report mean response length, mean verifier reward, Pearson score–length correlation, and Spearman score–verifier correlation.Source: §C.6 · p. 18, Figure 4 · p. 18
  • The correlation contrast suggests trajectory scores can increasingly favor length without continuing to favor correct answers, but it does not establish causation or isolate particular continuations.The associations describe evolving batches.Source: §C.6 · p. 18

C.7 PRESSURE MAGNITUDE DIAGNOSTICS

The pressure-magnitude diagnostics show larger normalized collective pressure in cross-pattern than matched Instruct pairings, while training outcomes depend on both pressure magnitude and compatibility with student behavior.

C.7 PRESSURE MAGNITUDE DIAGNOSTICS

  • The normalized proxy is the median over training steps of each batch’s absolute mean trajectory score divided by its mean within-prompt group standard deviation plus 10^-8; the retained proxy multiplies this by λ.It is a ratio of batch summaries, not an average of per-prompt normalized magnitudes or a measure of an individual prompt’s signed shared component.Source: §C.7 · p. 18
  • 3.16–3.61 in matched Instruct pairings versus 6.08 and 7.23 in cross-pattern pairings; larger shared-score offsets accompany pattern mismatch in these runs.Within matched pairings, the normalized proxy varies little across retention settings: 3.16–3.21 and 3.56–3.61.Source: §C.7 · p. 18
  • At λ = .25, the retained proxies are 0.79 and 0.90 across pairings, so the same retention fraction can preserve different magnitudes.Most of the retained-proxy difference within the reported settings comes from the retention fraction.Source: §C.7 · p. 18
  • In Inothink 1.7, λ = .5 retains a proxy of 1.60 and is followed by reward decline, while λ = .25 and .2 retain 0.79 and 0.63 and support sustained improvement.The cross-pattern runs retain no shared component at λ = 0, though their relative response preference term still uses teacher feedback.Source: §C.7 · pp. 18–19
Loading 2610.23042v1…