PaperlayerView exact revision on arXiv

Source-linked AI summary

When and Where to Trust the Teacher: Unifying On-Policy Distillation and GRPO through Entropy-Calibrated Credit Assignment

Jie Zhang, Jingxiao Yang, Zhehao Huang, Yuhang Liu, Xiaolin Huang

arXiv:2609.28385v1cs.LGcs.AI

TL;DR

Verifiable-reward training gives mathematical reasoning models final-answer supervision but little token-level guidance, while teacher preferences on student-generated responses need not indicate correctness. UECR-GRPO combines verifier and teacher signals in response ranking and redistributes task credit across tokens. Across five benchmarks, it exceeds the strongest baseline in average accuracy at both Qwen3 student scales.

  • Problem

    Verifier rewards supervise final-answer correctness but provide little guidance on individual tokens, while teacher preferences on student-generated responses need not reflect correctness.

    Source: §1 · p. 1

  • Method

    UECR-GRPO combines verifier and teacher utility before group normalization, then redistributes verifier-derived token credit using signed teacher gaps and entropy-calibrated, zero-sum adjustments.

    Source: §1 · pp. 2–3, §4.2.1 · p. 6

  • Results

    UECR-GRPO achieves higher average accuracy than compared training objectives across five benchmarks and two Qwen3 scales; its Avg@12 is 17.21% for Qwen3-1.7B and 65.09% for Qwen3-4B.

    Source: §5.2 · p. 8, §6 · p. 9

  • Takeaways & Limitations

    Across the evaluated benchmarks and model scales, UECR-GRPO separates trajectory ranking from token credit assignment while achieving higher average accuracy than the compared objectives.

    Source: §6 · p. 9

    A positive teacher–old-policy token gap indicates relative preference for the realized token, not mathematical validity, causal importance, or a successful final response.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) supervises mathematical reasoning through final-answer correctness, but provides little guidance on individual tokens. On-policy distillation (OPD) supplies dense feedback on student-generated responses, yet teacher preference need not reflect correctness. Recent hybrids combine OPD and verifier-derived advantages or reweight task credit using teacher ratios. However, teacher guidance enters after verifier-based group normalization, and token reweighting need not preserve the total task credit assigned to each response. We introduce Unified Entropy-Calibrated Credit Redistribution for GRPO (UECR-GRPO), which integrates verifier and teacher signals within a single GRPO-style update at both the response and token levels. \emph{Path-Utility Unification} (PUU) combines verifier reward and a teacher-to-anchor path log-ratio in a single KL-regularized objective. Its on-policy implementation uses a length-normalized teacher score and combines both rewards before group normalization and PPO clipping, allowing teacher evidence to influence the response ranking. \emph{Entropy-Calibrated Redistribution} (ECR) then uses the signed teacher--old-policy token gap to redistribute the verifier-derived component. Full-vocabulary teacher entropy attenuates uncertain guidance, while a response-wise zero-sum projection preserves the total task credit and its token-wise sign before clipping. Across five mathematical reasoning benchmarks, UECR-GRPO achieves average \(\mathrm{Avg@12}\) accuracies of 17.21\% and 65.09\% with Qwen3-1.7B and Qwen3-4B students, respectively, exceeding the strongest baseline at each scale by 0.89 and 0.56 percentage points.

1 INTRODUCTION

Verifier rewards identify whether a mathematical response succeeds but give every token the same response-level credit, while teacher preferences provide denser feedback that need not track correctness. UECR-GRPO combines verifier and teacher evidence in response ranking and redistributes task credit across tokens, with reported accuracy gains over the strongest baselines.

1 INTRODUCTION

  • Verifier rewards provide coarse response-level credit, while identical binary rewards within a group can eliminate the group-relative task signal.A single advantage is broadcast to every token, including decisive steps, stylistic tokens, and the first local error.Source: §1 · p. 1
  • Teacher feedback scores student-generated prefixes and can distinguish trajectories with tied verifier rewards, but teacher preference need not reflect correctness.A teacher may prefer an ultimately incorrect path, discourage a valid unfamiliar derivation, or transfer style rather than mathematical substance.Source: §1 · p. 1
  • Teacher scores distinguish 84.3–98.4% of verifier-degenerate groups, but conflict with verifier correctness on 39.1–50.8% of correct–incorrect pairs.On 347 judgeable verifier-tied pairs, teacher preference agrees with blinded Opus 4.8 process-quality judgments in 56.8% of cases.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • PUU combines verifier utility and a teacher-to-anchor path log-ratio in one KL-regularized objective, with both signals combined before group normalization in its on-policy implementation.This allows teacher evidence to influence response ranking before the clipped GRPO-style update.Source: §1 · pp. 2–3
  • ECR uses signed teacher–old-policy token gaps and teacher entropy to redistribute verifier-derived credit while preserving each response’s total task credit and token-wise sign before clipping.This separates local credit assignment from response-level scaling.Source: §1 · pp. 2–3
  • UECR-GRPO reports average accuracy gains of 0.89 and 0.56 percentage points over the strongest baseline for 1.7B and 4B students, respectively.The method combines verifier and teacher signals before group normalization and uses entropy-calibrated redistribution for token-level credit.Source: §1 · p. 3

2 RELATED WORK

Related approaches combine verifier rewards and teacher supervision at different stages, but they leave open how teacher evidence can affect group-relative response comparison while preserving response-level task credit during token redistribution. UECR-GRPO uses a shared response utility and entropy-calibrated token adjustments to address these distinctions.

RLVR and group-relative optimization

  • PPO, GRPO, DAPO, and GSPO refine policy optimization, but token credit in these methods remains determined by sequence-level rewards.GRPO derives advantages from within-prompt reward statistics; DAPO and GSPO modify clipping, sampling, loss reduction, length handling, or importance ratios.Source: §2 · p. 3

On-policy distillation and joint RL–KD

  • PUU forms a shared response utility before computing group-relative advantages, unlike joint RL–KD and reward-gated or sign-gated OPD approaches described here.Generalized Knowledge Distillation, KDRL, and G-OPD are also discussed as approaches to on-policy distillation or joint RL–KD.Source: §2 · p. 3

Teacher-guided credit assignment

  • Distilled RL reweights positive GRPO advantages with teacher-to-old-policy ratios, whereas ECR handles both advantage signs and preserves the arithmetic sum of verifier-derived credit.ECR also calibrates its guidance using teacher entropy.Source: §2 · p. 3
  • ATOD adds token-level OPD advantage after group normalization, so its teacher evidence does not affect the response rankings or group statistics used to construct the GRPO advantage.T-DUR reweights turns using teacher–student disagreement and student uncertainty.Source: §2 · p. 3
  • RLSD, RLCSD, StepOPSD, TASPO, SGCD, and SC-GRPO use other forms of teacher-, hint-, or action-level weighting, while ECR redistributes token credit and preserves each response’s verifier-credit total.TASPO uses mean-preserving privileged-information weights at the action level; ECR instead uses entropy-calibrated teacher gaps.Source: §2 · p. 3

Teacher uncertainty

  • ECR gates local adjustments with full-vocabulary teacher entropy and recovers the broadcast PUU advantage as confidence vanishes.The comparison also describes entropy-aware OPD using forward KL in high-entropy teacher regions and T-DUR using student sampled-token surprisal.Source: §2 · p. 3

KL-regularized control and inference

  • PUU applies KL-control and control-as-inference ideas to joint verifier and teacher utility, using a token mean instead of the exact path log-ratio sum before group-relative normalization.The cited background connects KL control to changes of measure from anchor dynamics and control-as-inference to reward-tilted trajectory distributions.Source: §2 · p. 3

3 PRELIMINARIES AND PROBLEM FORMULATION

The section establishes response and policy notation, then contrasts sequence-level verifier supervision with dense teacher preference on student-visited states. It frames the design choice as when teacher evidence enters the update and how response-level ranking is separated from token-credit redistribution.

3 PRELIMINARIES AND PROBLEM FORMULATION

  • A response y is a token sequence generated for prompt x; the behavior policy collects responses, the student is updated, and the frozen teacher scores them.A separate frozen reference policy may regularize the actor update, and the on-policy algorithm uses the old policy as its anchor.Source: §3 · pp. 3–4

3.1 GRPO: TASK-VALID BUT SEQUENCE-LEVEL SUPERVISION

  • GRPO samples multiple responses per prompt and uses verifier reward to form a within-group advantage, but broadcasts that advantage across each response’s tokens.When group reward variance is zero, the verifier-derived update vanishes entirely.Source: §3.1 · p. 4

3.2 OPD: DENSE PREFERENCE ON STUDENT-VISITED STATES

  • The teacher-to-old-policy log-probability gap measures teacher preference for each sampled token relative to the behavior policy.The gap is dense preference, not process correctness: a positive value does not establish mathematical validity, causal importance, or a successful final response.Source: §3.2 · p. 4

3.3 THREE WAYS OF COMBINING REWARD AND TEACHER EVIDENCE

  • Independent-loss hybrids combine clipped GRPO and OPD surrogates separately, so teacher evidence cannot alter the reward ranking used to form the GRPO advantage.Because clipping is nonlinear, adding the clipped losses is not equivalent to combining their evidence first.Source: §3.3 · p. 4
  • Post-normalization approaches add token-level OPD advantage or multiply normalized GRPO advantage by a teacher ratio after verifier ranking has been fixed.The passage identifies ATOD and Distilled RL as integrated actor updates rather than independent-loss baselines.Source: §3.3 · p. 4
  • Pre-normalization trajectory unification combines verifier reward and teacher evidence into one response utility before group-relative advantages are computed.Token credit is handled separately by redistributing the verifier-derived component within each response.Source: §3.3 · p. 4

4 UNIFIED TRAJECTORY UTILITY AND CONSTRAINED TOKEN CREDIT

UECR-GRPO uses a two-layer design: PUU unifies verifier and teacher evidence for response-level advantages, and ECR redistributes verifier-derived credit across tokens. PUU combines signals before group normalization, while ECR calibrates token guidance by teacher uncertainty and preserves each response’s task-credit budget.

4.1 PUU: PATH-UTILITY UNIFICATION

  • PUU combines verifier utility, a teacher-over-anchor path log-ratio, and anchor KL regularization in one trajectory-level objective.The summed on-policy distillation signal is an exact path log-density ratio.Source: §4.1 · p. 5

4.1.1 ON-POLICY GROUP-RELATIVE REALIZATION

  • On policy, PUU uses a length-normalized teacher-score surrogate and combines it with verifier reward before group normalization, allowing teacher evidence to change response ordering.The surrogate prevents response length from implicitly changing the teacher contribution’s scale.Source: §4.1.1 · p. 5

4.2 ECR: ENTROPY-CALIBRATED REDISTRIBUTION

  • Initialization analyses find that high-entropy prefixes contain most large teacher–old-policy gaps, motivating uncertainty calibration and response-wise credit preservation.Without centering, local token signals move the mean weight away from one and change total verifier-derived credit.Source: §4.2 · p. 6

4.2.1 DIRECTION AND CONFIDENCE

  • ECR uses the signed teacher–old-policy gap for local direction and full-vocabulary teacher entropy to attenuate guidance under uncertainty.Larger gaps increase direction magnitude, while higher entropy decreases confidence; direction also reverses with the task-advantage sign.Source: §4.2.1 · p. 6

4.2.2 ZERO-SUM RESPONSE-WISE PROJECTION

  • A confidence-weighted response-wise zero-sum projection centers token adjustments, keeping mean token weights at one and all weights positive.This preserves the response-wise additive mass and token-wise sign of the task component.Source: §4.2.2 · p. 7

4.2.3 FINAL ACTOR ADVANTAGE

  • The final advantage redistributes only verifier-derived credit; the teacher component introduced by PUU remains unchanged.When the redistribution residual vanishes, including at ρ = 0, the method reduces to PUU.Source: §4.2.3 · p. 7

5 EXPERIMENTS

UECR-GRPO achieves the strongest five-benchmark average at both student scales, while its experiments separately examine response-level signal variation and token-credit localization.

5.1 EXPERIMENTAL SETUP

  • Evaluation averages Avg@12 accuracy equally across five contests, using a fixed 12-sample decoding setup.Main comparisons pair Qwen3-1.7B with a frozen Qwen3-4B-GRPO teacher and Qwen3-4B with a frozen Qwen3-8B-Math-GRPO teacher.Source: §5 · p. 7, §5.1 · p. 8

5.2 MAIN RESULTS

  • 17.21% Avg@12 for Qwen3-1.7B exceeds ATOD-aligned by 0.89 points, and 65.09% for Qwen3-4B exceeds Distilled RL by 0.56 points.The 1.7B model leads on four benchmarks, whereas the 4B model trails the best baseline on three, so aggregate gains are not uniform.Source: §5.2 · p. 8

5.3 ABLATION STUDIES

  • In AMC 2023 pseudo-groups, verifier rewards are tied in 40–53% of groups, while unified utility varies in 56–90%.In mixed groups, the teacher component contributes 17–25% of summed absolute component magnitude, while unified advantage stays close in scale to verifier-only GRPO; these are signal magnitudes, not gradient or performance contributions.Source: §5.3 · pp. 8–9, Figure 4 · p. 8
  • Full ECR outperforms both component removals and yields positive invalid-minus-valid task-weight gaps, exceeding all 100 entropy-position shuffles.The offline diagnosis supports localization of negative task credit on identified errors while preserving each response’s task-credit budget; it does not establish a causal accuracy gain.Source: §5.3 · p. 9, Figure 5 · p. 9, Table 3 · p. 9

6 CONCLUSION

The conclusion presents UECR-GRPO as separating trajectory ranking from token credit assignment. Its accompanying technical material describes the clipped update, the PUU target and decomposition, and constraints and limits on ECR redistribution.

6 CONCLUSION

  • UECR-GRPO combines verifier and teacher utility before group normalization, then redistributes verifier-derived credit with entropy-calibrated teacher gaps while preserving each response’s task-credit budget.Across five benchmarks and two Qwen3 scales, it reports higher average accuracy than compared training objectives.Source: §6 · p. 9

A CLIPPED ACTOR UPDATE AND TEMPERATURE SEMANTICS

  • The clipped actor surrogate uses the current-to-old policy ratio and token advantage, with clipping as the practical proximal mechanism.The actor log probabilities use rollout temperature Tr = 0.7; the teacher gap instead compares teacher and old policy at raw T = 1.Source: A CLIPPED ACTOR UPDATE AND TEMPERATURE… · p. 12, §B.1 · p. 13

B.1 SETUP AND SUPPORT ASSUMPTIONS

  • The PUU target is a KL-regularized path distribution under support and finite-partition-function assumptions, with verifier reward and teacher preference tilting the same target.The implementation uses a masked mean rather than the token sum in the path-ratio identity, changing utility for variable-length responses.Source: §B.1 · pp. 12–13

C EXACT DECOMPOSITION OF THE PUU ADVANTAGE

  • The shared PUU group denominator supports an exact advantage decomposition; separately standardizing rewards by different standard deviations would break it and make the effective teacher coefficient group-dependent.Source: C EXACT DECOMPOSITION OF THE PUU… · p. 13

D CONSTRAINED TOKEN-CREDIT PROJECTION

  • ECR’s zero-sum projection preserves the arithmetic task-credit budget and token-wise sign for the verifier-derived component before clipping.It does not preserve the full PUU advantage sign, gradient norm, or parameter update after clipping.Source: D CONSTRAINED TOKEN-CREDIT PROJECTION · p. 14

D.1 ENTROPY COMPUTATION AND LIMITING CASES

  • As teacher entropy grows, confidence and the local redistribution proposal decrease; zero redistribution also results when ρ = 0 or token directions are identical.When confidence vanishes at every valid token, each token receives the original PUU advantage.Source: §D.1 · p. 14

E DIRECT COMPARISON WITH CLOSEST METHODS

The comparison distinguishes PUU’s pre-normalization response utility from teacher-guidance methods that act on an already normalized GRPO advantage. It also contrasts Distilled RL’s positive-advantage reweighting with ATOD’s token-level OPD addition.

E.1 DISTILLED RL

  • Distilled RL applies normalized teacher-to-old-policy ratios after verifier group normalization, so it cannot change response ranking and reweights only positive GRPO advantages.Its geometric normalization constrains the token-ratio product to one.Source: §E.1 · p. 14

E.2 ATOD

  • ATOD adds a turn-weighted token-level OPD advantage to an already normalized GRPO advantage in one clipped actor update, unlike PUU’s pre-normalization response utility.T-DUR combines normalized turn-level teacher–student disagreement with student sampled-token surprisal and weights only the OPD term.Source: §E.2 · p. 14

F.1 SIGNAL DIAGNOSIS ACROSS TRAINING

The signal diagnosis audits verifier-tied groups and variation in teacher utility, treating these as signal properties rather than training outcomes.

F.1 SIGNAL DIAGNOSIS ACROSS TRAINING

  • The audit reports the share of groups with nonzero teacher-utility variation, teacher–verifier ranking conflicts in non-degenerate groups, and a blinded audit of verifier-tied pairs.Source: §F.1 · p. 15
  • At initialization, 41.3% of groups are verifier-tied.A verifier-degenerate group has identical binary rewards and therefore zero within-group task advantage.Source: §F.1 · p. 15
  • The reported initialization tie rate is illustrated in Figure 1.Source: §F.1 · p. 15

F.2 TOKEN-CREDIT DIAGNOSIS AT INITIALIZATION

An initialization audit examines token-credit calibration and response-wise projection before training. It finds that high-entropy tokens account for most large teacher gaps, while projection preserves unit task-credit mass.

F.2 TOKEN-CREDIT DIAGNOSIS AT INITIALIZATION

  • The audit uses a shared initial 1.7B student and frozen 4B teacher, sampling eight responses for each of 300 prompts.This yields 2,400 trajectories and 21.55M valid response tokens, with no optimizer update or test benchmark involved.Source: §F.2 · p. 15
  • Entropy strata are set once from global step-0 quantiles, and tied near-deterministic values can collapse repeated boundaries.Accordingly, the authors call them strata rather than strict deciles.Source: §F.2 · p. 15
  • 82.4% of top-10% teacher gaps fall in the highest entropy quartile, and calibration reduces their mean magnitude by 37%.Entropy strata use global step-0 quantiles; tied near-deterministic values may collapse repeated boundaries.Source: §F.2 · p. 15
  • The projection gives B_i = 1 analytically, with maximum observed numerical error 2.22 × 10^-16; unprojected values range from 0.98 to 1.02.These quantities measure task-credit mass before clipping, not gradient norms or parameter updates.Source: §F.2 · p. 15

F.3 OFFLINE PUU GROUP-SIGNAL AUDIT

The offline PUU audit tests whether unified utility adds response-level signal in verifier-degenerate groups and examines the teacher component's share. Its signal-geometry results are diagnostic, not estimates of training or accuracy effects.

F.3 OFFLINE PUU GROUP-SIGNAL AUDIT

  • The audit samples 12 responses for each of 40 AMC 2023 problems and partitions them into groups of eight and four for diagnosis only.The frozen 4B teacher and 1.7B base anchor score the realized tokens; these pseudo-groups do not alter training.Source: §F.3 · pp. 15–16
  • 40–53% of groups are verifier-degenerate, while nontrivial unified-utility variation appears in 56–90% of groups.The teacher-range statistic declines from 86% at step 30 to 21% at step 515, so added response-level variation is strongest early in training.Source: §F.3 · p. 16
  • The audit compares verifier-only GRPO advantages with the exact PUU decomposition in mixed groups containing both verifier outcomes.It uses selected checkpoints and response-mean absolute magnitudes for each plotted component.Source: §F.3 · p. 16
  • The teacher share ranges from 17% to 25%, while the mean magnitude of the unified advantage remains close to the GRPO advantage.Under this construction, the teacher term is visible without overwhelming the task component.Source: §F.3 · p. 16
  • This signal-geometry audit uses pseudo-groups and a fixed base-model anchor, unlike the practical update's behavior-policy anchor.Its plotted component magnitudes do not estimate parameter-gradient or accuracy contributions; controlled training variants are reported separately in Table 3.Source: §F.3 · p. 16

F.4 OFFLINE SENSITIVITY OF VERIFIER-CONSISTENT ORDERING TO α

The appendix audits PUU’s verifier-consistent ordering on held-out rollouts and ECR’s token-credit localization, alongside aligned-run configurations and implementation checks. These results are bounded to offline audits; retraining-based sensitivity remains future work.

F.4 OFFLINE SENSITIVITY OF VERIFIER-CONSISTENT ORDERING TO α

  • The fixed-rollout sweep reused held-out AMC 2023 responses across 17 checkpoints, with groups of 12 or 24 samples rather than the training group size of eight.It does not establish within-class process quality or estimate performance from retraining at another α.Source: §F.4 · pp. 16–18
  • At α = 1, both group-balanced conflict rate and any-inversion group rate are zero across 451 mixed groups and 22,811 pairs; this is an offline audit, not retraining.The 95% empirically safe-group threshold ranged from 5.3 to 9.3 across checkpoints, while the median ranged from 12.6 to 21.3; the result is not a universal guarantee.Source: Figure 6 · p. 17, §F.4 · p. 17

F.5 OFFLINE ECR CREDIT-LOCALIZATION AUDIT

  • The ECR localization audit selected incorrect responses without using teacher or ECR signals, then compared token weights with blinded step-validity judgments.Its localization statistic contrasts judge-invalid and judge-valid tokens, with inference clustered by prompt.Source: §F.5 · p. 18
  • D = 0.0048 for Full ECR, beyond all 100 entropy-position permutations with mean gap 0.0024; PUU is exactly zero in this localization audit.The projection preserves a unit response-wise mean weight, reallocating rather than increasing verifier-derived credit; the audit does not prove causal accuracy gains.Source: §F.5 · p. 18

F.6 ALIGNED CONFIGURATIONS

  • Aligned runs pair Qwen3-1.7B, trained for 515 steps on DeepMath-103K problems at difficulty levels 5–7, with a Qwen3-4B-GRPO teacher, and Qwen3-4B with a frozen Qwen3-8B-Math-GRPO teacher.Figures 7 and 8 track training reward, response length, actor entropy, gradient norm, advantage statistics, and fixed-decoding validation accuracy.Source: §F.6 · pp. 18–19, Figure 7 · p. 19, Figure 8 · p. 19

F.7 REMAINING SENSITIVITY AND ADDITIONAL CONTROLS

  • Retraining-based sensitivity to α, ρ, τδ, and τH remains future work; the completed offline α audit does not estimate counterfactual training performance.Source: §F.7 · p. 19

F.8 EFFICIENCY AND BATCH-FACT AUDITING

  • Efficiency and batch-fact auditing logs timing, throughput, memory, and trajectory counts, and fails runs with altered batches, duplicated prompts, dropped trajectories, or incorrect step counts.Source: §F.8 · p. 20

F.9 REGRESSION AND SMOKE TESTS

  • Regression checks verify PUU recovery at ρ = 0, projection invariants, token-score alignment, single-entry reference KL, and native-GRPO recovery when teacher scoring is disabled.CPU tests precede 1-, 5-, and 10-step GPU runs.Source: §F.9 · p. 20

G REPRODUCIBILITY CHECKLIST

The checklist specifies artifacts, telemetry, and run-integrity checks for reproducibility.

G REPRODUCIBILITY CHECKLIST

  • Archive checkpoints, tokenizer and chat-template assets, manifests, source and configuration hashes, plus reward, normalization, decoding, extraction, selection, and per-problem records.Source: G REPRODUCIBILITY CHECKLIST · p. 20
  • Log shared reward, advantage, PPO, KL, length, truncation, trajectory, token, and memory telemetry; disable teacher metrics rather than recording zeros for Vanilla GRPO.Source: G REPRODUCIBILITY CHECKLIST · p. 20
  • Additionally log method-specific teacher and PUU/ECR quantities, response-wise budget error, and maximum |AU_i|, with numerical tolerances stated before training.Source: G REPRODUCIBILITY CHECKLIST · p. 20
Loading 2609.28385v1…