PaperlayerView exact revision on arXiv

Source-linked AI summary

Can We Trust the Teacher? Decoupled Credit Direction-Magnitude for Self-Distillation

Yugu Li, Zehong Cao, Peizhen Li, Yang Zhang, Siyi Hu, Jianglin Qiao

arXiv:2609.34848v1cs.AI

TL;DR

Trajectory-level RLVR provides outcome feedback, while OPSD supplies denser teacher supervision that can misjudge local credit. DCSD separates credit direction from magnitude to calibrate teacher supervision, and improves overall student reasoning performance over outcome-based RL and existing self-distillation methods across 11 mathematical and multimodal benchmarks.

  • Problem

    RLVR typically assigns credit at trajectory level, whereas OPSD’s coupled teacher signal can misjudge local step direction and magnitude.

    Source: §1 · p. 1

  • Method

    DCSD determines step direction through belief-margin probing and contribution magnitude through marginal information gain, then uses the signals to calibrate teacher supervision.

    Source: §1 · p. 2

  • Results

    Across 11 mathematical and multimodal reasoning benchmarks, DCSD improves overall student reasoning performance over outcome-based RL and existing self-distillation methods.

    Source: §1 · p. 3, §5 · p. 11

  • Takeaways & Limitations

    Separating direction from magnitude provides a way to calibrate privileged teacher supervision rather than directly inheriting its credit.

    Source: §1 · p. 2

    Limitations in visual evidence can constrain subsequent reasoning and local credit estimation, which credit calibration alone cannot address.

    Source: §4.2 · p. 9

Abstract

from arXiv · show

RLVR provides reliable trajectory-level credit, while OPSD offers dense supervision for token-level credit. This exposes a fundamental coupling when updating step-level credit direction and magnitude with teacher supervision, preventing steps from receiving reliable credit directions and contribution magnitudes, while making both vulnerable to teacher judgment errors and preference variance, as supported by our theoretical analysis. To separate credit direction from its contribution magnitude, we introduce \textit{Decoupled Credit Self-Distillation (DCSD)}, which theoretically decouples credit direction and magnitude into two reliable signals and uses them to calibrate privileged teacher supervision. Specifically, we design belief-margin probing to determine credit direction and marginal information gain to quantify credit magnitude, enabling step-to-token credit assignment for policy optimization. Across 11 benchmarks, DCSD achieves the best overall scores against GRPO, OPSD, RLSD, and RLCSD. Compared with base models, DCSD improves the overall score by 8.45 points on mathematical reasoning and 7.01 points on multimodal reasoning, while correcting the credit direction for 6\% of tokens and yielding a 1.5$\times$ reduction in token credit magnitude.

1. Introduction

RLVR supplies outcome feedback but typically assigns credit to whole trajectories, while OPSD offers denser teacher supervision that can still misjudge local credit. DCSD separates credit direction from contribution magnitude to calibrate teacher supervision, and the paper reports the best overall scores against GRPO, OPSD, RLSD, and RLCSD.

1. Introduction

  • RLVR assigns reliable outcome feedback at trajectory level, while OPSD uses privileged teacher information for denser supervision but can miscredit individual reasoning steps.Trajectory-level credit may reinforce incorrect or unnecessary steps in successful trajectories and give no positive credit to useful steps in unsuccessful ones.Source: §1 · p. 1
  • Oracle self-distillation is directionally consistent with local RL advantage, but its magnitude does not directly measure a step’s contribution and practical teachers can corrupt its direction.Source: §1 · p. 2
  • DCSD determines direction with belief-margin probing and magnitude with marginal information gain, then uses the two signals to calibrate teacher supervision.This separation is intended to preserve fine-grained token-level information without letting teacher variation freely alter established step-level credit.Source: §1 · p. 2
  • Across mathematical and multimodal reasoning benchmarks, DCSD achieves the best overall scores against GRPO, OPSD, RLSD, and RLCSD.The authors report that calibrated teacher signals improve student overall performance.Source: §1 · p. 3

2. Why Direct Teacher Signals Produce Coupled Credit

The paper formalizes local step credit using changes in eventual success probability, then shows that an oracle-conditioned self-distillation signal has the correct direction. Practical teacher judgment errors and preference variation can reverse that direction, while trajectory-level anchoring sacrifices local flexibility.

2.1. Oracle Local Advantage

  • An oracle local advantage compares a step’s success probability with the preceding state’s eventual success probability, but intermediate-state success probabilities are unknown and trajectory advantage cannot distinguish local contributions.Source: §2.1 · p. 3

2.2. Oracle Signal Determines the Correct Credit Direction

  • The log-ratio between the student policy conditioned on eventual success and its ordinary policy is directionally consistent with oracle local advantage, but its magnitude is not thereby established as contribution magnitude.Source: §2.2 · p. 3

2.3. Direct Teacher Signal Deviates from the Oracle Signal

  • Direct OPSD credit combines teacher judgment deviation and preference-dependent variation, either of which can reverse the oracle direction and couple direction with credit magnitude.Source: §2.3 · p. 4
  • RLSD and RLCSD anchor local credit to trajectory advantage, so harmful steps in successful trajectories cannot receive negative credit and useful steps in unsuccessful trajectories cannot receive positive credit.Source: §2.3 · p. 4

3. Method: Decoupled Credit for Calibrating Teacher Signals

DCSD derives step-level credit from student representations and answer beliefs rather than directly estimating intermediate-state values or inheriting teacher credit. It separately estimates direction and relative magnitude, then uses teacher evidence only to allocate calibrated step credit across tokens.

3.1. From Value Estimation to Information-Attributed Credit

  • DCSD identifies reasoning steps from representation transitions and attributes information in the student’s hidden-state features, avoiding direct estimation of intermediate-state values.The paper notes that value estimation can introduce approximation bias or sampling variance, and that sequential information attribution does not assign correctness labels.Source: §3.1 · pp. 4–5

3.2. Belief-Margin Probe Design for Decoupled Credit Direction

  • Belief-margin probing scores candidate answers with a fixed readout suffix, using the change in the student’s support for the correct answer to set step direction.When the belief-margin change does not cross a confidence threshold, DCSD falls back to trajectory-level direction to account for readout mismatch or incomplete candidate coverage.Source: §3.2 · p. 5

3.3. Information-Gain Principle for Decoupled Credit Magnitude

  • Marginal information gain measures information a step adds beyond preceding reasoning and is normalized to assign bounded relative credit magnitude within the response.Theorem 3 connects the information gain to oracle credit through evidence-model bounds rather than claiming pointwise recovery of oracle magnitude.Source: §3.3 · p. 6

3.4. Calibrated Teacher Supervision for Step-to-Token Credit Assignment

  • The teacher supplies token-level evidence only after step direction and magnitude are established, with its influence bounded during within-step allocation.The factorization separates step direction, step magnitude, and teacher-based within-step allocation; Theorem 4 states that calibrated supervision preserves step credit for every teacher realization under its stated assumptions.Source: §3.4 · pp. 6–7

4. Experiments and Results

Experiments evaluate DCSD in mathematical and multimodal reasoning, examining benchmark performance, training behavior, teacher-signal deviation, and reasoning efficiency. DCSD improves over the reported baselines in most task comparisons, with ZeroBench an exception.

4.1. Experimental Setup

  • Experiments use Qwen3-4B with DAPO-17K for math and Qwen3-VL-8B-Instruct with MMFineReason-123K for multimodal reasoning, evaluating mean@4 across six and five benchmarks.Teacher-signal deviation is assessed by varying privileged information for fixed student responses across five conditions.Source: §4.1 · pp. 7–8

4.2. Main Performance Results

  • DCSD consistently improves over GRPO, OPSD, and RLSD on mathematical tasks; multimodally, it improves over the strongest baseline RLSD on four tasks, with ZeroBench the exception.The mathematical gains include challenging multi-step tasks and higher-accuracy benchmarks.Source: §4.2 · p. 8, Table 1 · p. 9

4.3. Training Dynamics and Credit Analysis

  • About 6% of tokens receive direction corrections, while magnitude attenuation remains stable; DCSD reaches stronger validation accuracy despite training rewards comparable to GRPO and RLSD.Case studies show DCSD selectively reverses OPSD signals that reward incorrect steps or penalize correct ones.Source: §4.3 · pp. 9–10

4.4. Teacher Signal Deviation and Calibration Effects

  • DCSD has the lowest TDR and MAE across privileged-information conditions C0–C4 against the condition-matched Qwen3-32B reference.Its TDR advantage also appears under C0, with no privileged answer information.Source: Table 2 · p. 10, §4.4 · pp. 10–11

4.5. Calibrated Teacher Signal Improves Reasoning Efficiency

  • Across six benchmarks, DCSD improves unweighted mean Pass@1 by 4.02% and Pass@4 by 3.74% over RLCSD while reducing mean response length by 6.3%.Responses are shorter on every benchmark in this comparison.Source: Table 3 · p. 11, §4.5 · p. 11

5. Conclusion

DCSD separates step-credit direction from contribution magnitude to calibrate privileged teacher supervision. The analysis motivates this separation by showing that trajectory outcomes need not reflect local credit and that an oracle-consistent log-ratio does not encode contribution scale directly.

5. Conclusion

  • Across 11 mathematical and multimodal reasoning benchmarks, DCSD improves overall student reasoning performance over outcome-based RL and existing self-distillation methods.It determines direction through belief-margin probing and magnitude through marginal information gain; future work could extend the framework to longer-horizon and interactive reasoning.Source: §5 · p. 11

A. Related Work

  • DCSD addresses local credit by separating a step’s update direction from its contribution magnitude, while retaining teacher evidence for fine-grained token supervision.Existing approaches enrich or stabilize privileged supervision, anchor it to outcome rewards, selectively route or weight distillation, or contrast teacher signals across contexts.Source: A. Related Work · p. 17

B. Analysis of Direct Teacher Signal with Coupled Credit

  • Direct teacher supervision, positive outcome-anchored reweighting, and contrastive outcome-anchored correction do not by themselves guarantee oracle-consistent local credit.The analysis separates teacher deviations into judgment error and preference-dependent variation.Source: B. Analysis of Direct Teacher Signal… · p. 17

B.1. Step-Level Reasoning and Oracle Local Credit

  • A trajectory-level outcome does not require every local contribution to share its sign; successful responses can contain harmful steps and unsuccessful ones useful steps.The oracle local advantage’s sign indicates whether the next prefix has a higher or lower continuation success probability.Source: §B.1 · p. 18

B.2. Deriving the Success-Conditioned Oracle Signal

  • Under success conditioning, the oracle log-ratio is directionally consistent with local advantage, but equal log-ratios can correspond to different advantage magnitudes.For example, transitions from 0.1 to 0.2 and from 0.4 to 0.8 both yield log 2, but advantages of 0.1 and 0.4, respectively.Source: §B.2 · p. 19
  • Local advantage magnitude depends on the preceding value and a nonlinear transformation of the success-conditioned log-ratio; DCSD uses relative magnitude weights to allocate credit across steps.The analysis assumes positive probabilities on the relevant support for finite-log results.Source: §B.2 · p. 19

B.3. Judgment Error and Preference-Dependent Variation

The analysis distinguishes changes in teacher scores caused by semantic privileged content from variation across its realizations. It shows that direct OPSD can misstate both local credit direction and magnitude, including in a normalized example.

B.3. Judgment Error and Preference-Dependent Variation

  • Preference-dependent variation is defined by changing the realization while holding the student, state, step, teacher parameters, and semantic content fixed.The exact decomposition separates the realization-averaged teacher’s deviation from the success-conditioned oracle from a particular realization’s deviation from that average.Source: §B.3 · pp. 19–20
  • Averaging teacher probabilities is generally different from averaging log-probabilities, and the step-level signal connects to summed token-level teacher log-ratios.The realization average is taken over complete step probabilities.Source: §B.3 · pp. 19–20

B.4. Direct OPSD: Local Flexibility Without an Oracle Guarantee

  • In the normalized example, the beneficial step C has oracle advantage 0.3, yet realization u1 gives it teacher probability 0.3 against an averaged probability of 0.6, producing a negative log-ratio.Direct teacher supervision does not enforce the sufficient condition for preserving oracle direction; its log-ratio magnitude bound is not a bound on oracle advantage magnitude.Source: §B.4 · p. 21

B.5. RLSD: Outcome Anchoring and the No-Crossing Limitation

The appendix analyzes the limits of outcome-anchored teacher methods and details DCSD’s separate constructions for local direction, relative magnitude, and calibrated token allocation. DCSD’s teacher signal is reserved for within-step redistribution after student-side credit signals are fixed.

B.5. RLSD: Outcome Anchoring and the No-Crossing Limitation

  • Teacher evidence can change RLSD credit magnitude but cannot reverse trajectory-level direction; when local oracle direction conflicts, outcome anchoring cannot recover it.A zero multiplier yields no update, and trajectory anchoring does not identify local contribution magnitude.Source: §B.5 · pp. 21–22

B.6. RLCSD: Contrastive Preference Cancellation and Its Limits

  • RLCSD can suppress shared preference effects, but residual variation may remain and outcome-preserving modulation cannot recover an opposing oracle local direction.Its contrastive scale is not constructed to equal the absolute oracle-advantage scale.Source: §B.6 · p. 22

B.7. Implications for Decoupled Credit for Teacher Calibration

  • DCSD addresses complementary limits by selecting step direction and relative magnitude separately, then using teacher evidence only for within-step token allocation.Teacher variation can redistribute credit within a step but cannot change its established direction or total magnitude.Source: §B.7 · p. 23

C. Details of DCSD Workflow

  • The workflow appendix details marginal-information, belief-margin, and hierarchical step-to-token credit constructions after fixing the student rollout and trajectory advantage.The reasoning-step partition and local credit signals are constructed from the fixed student trajectory.Source: C. Details of DCSD Workflow · p. 23

C.1. Detailed: From Value Estimation to Information-Attributed Credit

  • Detected representation transitions define reasoning steps, whose conditional response information is attributed to each step; marginal gains also bound answer information and expected squared oracle advantage.The realized partition is held fixed when the implemented information attribution is computed.Source: §C.1 · p. 24, §C.3 · p. 28

C.2. Detailed: Belief-Margin Probe Design for Decoupled Credit Direction

  • Belief-margin probing uses student answer scores and the known correct answer to select local direction, with trajectory-level fallback when evidence is insufficient.Oracle certification is conditional on the theorem’s readout-and-coverage error budget, which the online probe does not directly evaluate.Source: §C.2 · pp. 24, 26

C.3. Detailed: Information-Gain Principle for Decoupled Credit Magnitude

  • Marginal representation information sets relative step magnitude: redundant representation structure yields less gain, and if all gains vanish, every step receives zero credit.Weights are maximum-normalized, so the largest-gain step retains unit relative magnitude rather than the weights summing to one.Source: §C.3 · pp. 27–28

C.4. Detailed: Calibrated Teacher Supervision for Step-to-Token Credit Assignment

DCSD applies privileged teacher information only after step direction and relative magnitude are fixed, using it to allocate credit among tokens within each step. The proof appendix states a calibration-and-coverage condition for the belief-probe direction guarantee.

C.4. Detailed: Calibrated Teacher Supervision for Step-to-Token Credit Assignment

  • Teacher–student token evidence is converted into bounded positive weights, normalized within each step to redistribute credit without changing its established direction or total magnitude.The allocation is bounded against concentration; with ϵw = 0 it is uniform, and for a one-token step the weight is 1.Source: §C.4 · pp. 28–29
  • The token-credit construction uses a common reference scale κτ|Aτ| multiplied by the sum of maximum-normalized step weights αk.When αk = 0 a step receives zero credit, and when Aτ = 0 all token credits are zero.Source: §C.4 · pp. 29–30

D. Proofs of Theoretical Results in DCSD

  • The theorem’s direction guarantee requires finite readout-calibration and candidate-coverage error, rather than empirical threshold selection alone.The online probe does not directly evaluate Ξk.Source: D. Proofs of Theoretical Results in DCSD · p. 30

D.1. Proof of Theorem 1: Sequential Information Attribution as Step Credit

The proof assigns each ordered feature group its conditional information about a target beyond the preceding groups. These increments are nonnegative, sum to total information, and for response prediction equal the gain in best attainable expected log score.

D.1. Proof of Theorem 1: Sequential Information Attribution as Step Credit

  • Each step’s contribution is conditional mutual information: the added information in Φk about target Υ given X and prior groups Φ<k.Subtracting the information in the preceding feature prefix leaves the increment for the current group.Source: §D.1 · p. 31
  • The contribution is nonnegative and is zero exactly when adding Φk leaves the conditional target distributions unchanged, equivalently when Υ is conditionally independent of Φk.Source: §D.1 · p. 31
  • Summing sequential contributions recovers the total information in all feature groups because the empty feature prefix carries zero information.Source: §D.1 · p. 31
  • The attribution applies to sequential internal feature groups, with response targets measuring response construction and evidence targets measuring information about the answer-relevant latent variable.Source: §D.1 · pp. 30–32
  • For response prediction, adding Hk improves the best attainable expected log score by exactly Ik.Source: §D.1 · p. 32
  • Refining a feature group redistributes its raw information contribution but preserves the total; fixed log-determinant geometry also satisfies diminishing returns.Source: §D.1 · p. 32

D.2. Proof of Theorem 2: Oracle-Consistent Credit Direction

The proof relates a step’s probe-selected belief-margin change to its oracle log-odds change under calibrated readouts and bounded error. Signed thresholds then give a sufficient condition for oracle-consistent direction on selected steps.

D.2. Proof of Theorem 2: Oracle-Consistent Credit Direction

  • With the continuation policy fixed, the proof compares success probabilities before and after a step through log-odds, retaining failed or unparsable completions as incorrect outcomes.Source: §D.2 · p. 32
  • Readouts at both boundaries use the same finite candidate field, including the correct answer and at least one competitor, with calibration error bounded by Ξk.Source: §D.2 · p. 33
  • The readout and coverage error bound controls the discrepancy between probe-margin change and oracle log-odds change.Source: §D.2 · p. 33
  • A probe-selected step is oracle-consistent when its signed threshold is crossed and Ξk is below the corresponding threshold.Source: §D.2 · p. 34
  • When selected steps satisfy the certificate, their directions are correct while unselected steps retain the trajectory outcome sign.Source: §D.2 · p. 34

D.3. Proof of Theorem 3: Information Gain Bounds Credit Magnitude

Under the stated evidence model, marginal information gain quantifies a step’s contribution to latent-answer information and bounds answer information independently of the verifier. The proof connects that gain to oracle-credit behavior, including its null cases and dependence on the answer map.

D.3. Proof of Theorem 3: Information Gain Bounds Credit Magnitude

  • Accumulated evidence attenuates later marginal gain in covered directions, while repeated nonzero observations may retain positive gain.Source: §D.3 · p. 35
  • Under the evidence model, step gain ΔFk is the conditional information about latent answer Θ and sums to total information across the steps.Source: §D.3 · pp. 35–36
  • ΔFk bounds answer information for every answer map and is the least verifier-independent upper bound on that information.Source: §D.3 · p. 36
  • The information gain bounds oracle-credit moments, and zero gain makes the step oracle-null for every answer map.Source: §D.3 · p. 37
  • With representations and evidence fixed, realized oracle magnitude can still vary with the verifier when ΔFk is positive.Source: §D.3 · p. 38
  • DCSD sets αk to zero exactly when ΔFk is zero; otherwise its relative magnitude follows the information envelope, and certified steps have oracle-consistent direction.Source: §D.3 · p. 38

D.4. Proof of Theorem 4: Calibrated Teacher Supervision Preserves Step Credit

The theorem proof states that positive within-step teacher weights preserve aggregate step credit while assigning token coefficients that inherit certified step direction. The appendix also documents experiments and reports lower DCSD diagnostic errors than RLSD and OPSD across conditions.

D.4. Proof of Theorem 4: Calibrated Teacher Supervision Preserves Step Credit

  • Positive within-step teacher weights preserve aggregate step direction and total magnitude, and tokens inherit the oracle-consistent direction of certified parent steps.Source: §D.4 · pp. 39–40

E. Experimental Details and Result Analysis Protocols

E.2. Construction of the C0–C4 Diagnostics

  • Diagnostics compare sampled-token MAE and full-vocabulary top-1 disagreement against a matched-prompt Qwen3-32B operational oracle, averaging equally across C0–C4.Source: §E.2 · pp. 41–42
  • DCSD has lower MAE and TDR under every condition; condition-averaged MAE is 0.192993 nat/token versus RLSD 0.198465 and OPSD 0.217870, while TDR is 9.8218%, 10.1566%, and 12.1243%, respectively.Source: Table 6 · p. 42, §E.2 · p. 42

E.3. Training-Dynamics Definitions

  • The logged correction_rate is the fraction of valid response tokens whose selected direction differs from sign(Aτ), counting both positive and negative corrections.Source: §E.3 · p. 42

E.4. Baseline Training Settings and Result Sources

The mathematical baseline runs use Qwen3-4B, shared DAPO-17K data, prompt batches, and rollout budgets, with final evaluation at step 70. DCSD adds post-processing costs, but generation remains the dominant shared expense.

E.4. Baseline Training Settings and Result Sources

  • All mathematical baseline runs use Qwen3-4B in thinking mode, shuffled DAPO-17K data, shared prompt batches and rollout budgets, and step-70 checkpoints for final evaluation.The prepared data contain 17,917 source records; Table 7 documents baseline training settings.Source: §E.4 · p. 42, Table 7 · p. 43
  • OPSD uses sampled-token self-distillation, while RLSD and RLCSD use their specified teacher-refresh and rollout-context settings; multimodal evaluation follows the RLSD source script.RLSD refreshes a frozen teacher every 20 iterations; RLCSD uses same-question eligible rollout contexts and skips groups without usable contexts.Source: §E.4 · p. 43

E.5. Computational Overhead

  • 1.6481 GPU-hours per step is DCSD’s total training cost, 23.2% above OPSD; generation remains dominant despite 0.3463 GPU-hours per step in other processing.The added processing extracts and analyzes hidden representations, induces reasoning steps, computes marginal information gains, and performs belief-margin probing.Source: §E.5 · p. 43, Table 8 · p. 43

F. Case Studies

The case studies compare local credit signals with the semantic contribution of selected reasoning spans. DCSD agrees with the stated judgments in several examples, but stable signs do not always imply semantically appropriate credit.

F. Case Studies

  • −0.015295 for DCSD agrees with the negative judgment on an incorrect angle reduction, unlike OPSD’s positive mean of +0.005286.The boxed chain reduces 23π/14 incorrectly to 2π −π/14 rather than 2π −5π/14.Source: Figure 4 · p. 44
  • +0.037267 for DCSD gives positive credit to a correct complex-number simplification, while OPSD assigns −0.066991.The boxed calculation (5 + 10i)/5 = 1 + 2i completes the correct evaluation.Source: Figure 5 · p. 45
  • +0.035223 for DCSD gives positive credit to the correct outer-annulus area, whereas OPSD assigns −0.048977.The intermediate area is 13π; the requested difference between the three region areas is 7π.Source: Figure 6 · p. 46
  • +0.038157 for DCSD gives positive credit to the correct winning-move inference, unlike OPSD’s −0.042883.Taking four tokens from nine leaves five, a losing position for the player to move.Source: Figure 7 · p. 47

Credit case 5: correct remainder calculation

  • +0.012545 for DCSD gives positive credit to the correct remainder calculation, while OPSD assigns −0.021065.The calculation supplies the divisibility condition n + 2 | 39; it does not assert that P(−2) vanishes.Source: Figure 8 · p. 48, Credit case 5: correct remainder… · p. 48
  • DCSD remains negative on valid conjugate rationalization (−0.002488, −0.000041), although the second mean is near zero.The step is described as useful and deserving positive local credit; both Base and OPSD switch from small negative to small positive means across prompts.Source: Figure 9 · p. 49
  • A formatting separator warrants near-zero magnitude without a correct positive or negative sign, yet DCSD remains negative with substantial magnitude.Base and OPSD switch from negative to positive across prompts; DCSD’s stable suppression does not match the near-zero target.Source: Figure 10 · p. 50
  • A final-answer heading also warrants near-zero magnitude; DCSD’s stable positive means do not establish semantically correct positive credit.The heading adds no derivation, and the actual answer appears outside the frame.Source: Figure 11 · p. 51

Related papers

Loading 2609.34848v1…