DIAL-OPD: Learning More from Fewer Tokens in On-Policy Distillation

1 Introduction

• Random selection at 80% retention raises MATH-500 accuracy from 30.2% to 38.4% for a 0.6B student, but the benefit disappears when the student matches the 4B teacher.The passage attributes this pattern to the smaller student's limited capacity to absorb dense supervision.Source: Figure 1 · p. 2, §1 · p. 2
• At 10% retention, low-low tokens make up 20.6% of selected tokens but 49.8% of absolute reward, and replacing them improves learning across retention ratios.The log-ratio reward treats (0.08, 0.8) and (10^-4, 10^-3) alike despite the latter probabilities being negligible under both models.Source: §1 · p. 2
• DIAL-OPD weights reward magnitude by the logarithmic mean of teacher and student probabilities, with β controlling how strongly probability scale affects selection.It retains the highest-scoring tokens within each response and applies the original OPD loss only to those tokens.Source: §1 · p. 3
• At 40% retention, DIAL-OPD exceeds Vanilla OPD and the strongest full-token OPD variant in mean accuracy across all settings by up to 5.25 and 1.24 points, respectively, and doubles AIME25 Pass@16 from 13.33% to 26.67%.At 20% retention, it exceeds the strongest token-selection baseline at the same ratio by up to 5.23 points in mean accuracy and 10.33 points on GSM8K Avg@4.Source: §1 · p. 3
• Moderate β weighting suppresses low-low tokens while preserving useful disagreements, whereas excessive weighting can discard valuable supervision.Token-level studies report that DIAL-OPD filters high-reward tokens with limited reasoning value while preserving supervision important to reasoning correctness.Source: §1 · p. 3

2 Preliminaries

• OPD minimizes reverse KL divergence between the teacher's and student's next-token distributions over the vocabulary on student-generated responses.The setup uses x sampled from the data distribution and y sampled from the student policy.Source: §2 · p. 3
• Sampled-token OPD reduces the communication and memory costs of full-vocabulary distributions by querying teacher log-probabilities only for student-generated tokens.Source: §2 · p. 3
• Student rollouts provide an unbiased Monte Carlo estimate, with the token-level reward treated as a stop-gradient weight.The derivation also explains the stop-gradient treatment of the reward.Source: §2 · p. 3
• The figures cover reward-based selection, low-low token and absolute-reward shares, and low-low replacement.Source: Figure 3 · p. 4, Figure 4 · p. 4, Figure 5 · p. 4

3.1 More Supervision Does Not Imply Better Learning

• At 80% retention, random selection reached 38.4% MATH-500 accuracy versus 30.2% for full-token Vanilla OPD, an 8.2-point gain.With a Qwen3-4B student and the same teacher, 30% retention reached 72.6% accuracy versus 71.2% for full-token Vanilla OPD.Source: §3.1 · p. 4

3.2 Which Tokens Are Worth Learning From?

• At 10%, 15%, and 20% retention, Reward-Top reached 25.6%, 30.4%, and 34.4% accuracy, outperforming Reward-Bottom and Random at each ratio.Reward-Top runs improved little during the first 350 steps and did not consistently beat full-token Vanilla OPD.Source: §3.2 · p. 5

3.3 Low–Low Supervision Can Hinder Learning

• At 10%, 15%, and 20% retention, Low-low replacement exceeded Reward-Top accuracy by 12.8, 7.2, and 6.4 points, respectively.Low-low tokens made up 20.59%, 13.49%, and 10.26% of selected tokens but contributed 49.80%, 41.43%, and 38.01% of total absolute reward at those retention ratios.Source: §3.3 · p. 5

4 Method

• DIAL-OPD weights OPD reward magnitude by the logarithmic mean of student and teacher probabilities, reducing low-low token scores when β > 0.The method selects the highest-scoring tokens at the chosen retention ratio and applies the original signed OPD reward only to selected tokens.Source: §4 · p. 5, §4.1 · p. 6, §4.3 · p. 7

4.1 Bridging Log-Probability and Probability Spaces

• At β = 0 selection uses reward magnitude, at β = 1 it uses the absolute probability gap, and intermediate values interpolate between them.Values above 1 strengthen probability weighting.Source: §4.1 · p. 6

4.2 The Role of β in DIAL-OPD

• Increasing β suppresses low-low disagreement, but β > 1 drives scores to zero when only one probability vanishes; β = 1 preserves a finite, nonzero limit..Source: §4.2 · pp. 6–7

4.3 DIAL-OPD

• DIAL-OPD selects tokens from candidate positions in each student-generated response and recomputes scores and masks from fresh rollouts at each training step.Unselected tokens remain in context but are excluded from the loss.Source: §4.3 · p. 7

• Qwen3 students at 0.6B and 1.7B are trained with 4B and 8B teachers on DeepScaleR for 800 steps.Training details are given in Appendix B.Source: §5 · p. 7
• Evaluation covers GSM8K, MATH-500, AMC23, AIME24, AIME25, Minerva Math, and OlympiadBench.Source: §5 · p. 7
• Baselines include full-token Vanilla OPD, Random, PowerOPD, ExOPD, TIP, SA-OPD, TA-OPD, Prefix OPD, and CC-OPD.Token-selection methods are compared at matched retention ratios.Source: §5 · p. 7

6.1 The Effect of β on Learning

• On MATH-500, both students learn faster and reach higher final accuracy near β = 1, while excessive probability weighting reduces performance.The 0.6B student peaks at 44.8% for β = 1.2; the 1.7B student peaks at 61.8% for β = 1.Source: §6.1 · pp. 7–8

6.2 Main Results

• At 40% retention, DIAL-OPD beats full-token OPD across all four model pairs, with mean-accuracy gains up to 5.25 points over Vanilla OPD and 1.24 points over the strongest full-token variant.The Vanilla OPD comparison reaches 5.25 points for 4B→0.6B; the strongest full-token variant comparison reaches 1.24 points for 4B→1.7B.Source: §6.2 · p. 8, Table 1 · p. 8, Table 2 · p. 9
• At 20% retention, DIAL-OPD has the highest seven-benchmark mean accuracy in all eight model-pair and retention combinations, exceeding the strongest existing selection method by up to 5.23 points.The 5.23-point gain is for 4B→1.7B over CC-OPD at β = 1.Source: §6.2 · pp. 8–9, Table 1 · p. 8, Table 2 · p. 9
• With a 4B teacher and 40% retention, DIAL-OPD exceeds the strongest full-token variant using an 8B teacher by 0.95 and 1.76 mean-accuracy points for 0.6B and 1.7B students, respectively.Both comparisons use β = 1.Source: Table 1 · p. 8, §6.2 · p. 9, Table 2 · p. 9

6.3 Understanding DIAL-OPD’s Selection Behavior

• At 20% retention, raising β from 0 to 1 reduces selected low-low tokens from 8.26% to 0.29%; at β = 2, their share falls to 0.02%, but useful disagreements may also be suppressed.The selection-rate visualization uses hexagon colors for selection rates and dotted lines to delimit the low-low region.Source: §6.3 · pp. 9–10, Figure 7 · p. 10
• DIAL-OPD filters high-reward but low-value tokens such as serialVersionUID (−28.13) and Therefore (+4.52), while retaining useful supervision for an incircle formula (+0.94) and an arithmetic digit (−0.81).The contextual comparison shows that reward magnitude alone does not establish a token’s reasoning value.Source: §6.3 · p. 10, Figure 8 · p. 10

7 Conclusion

• DIAL-OPD uses the logarithmic mean to combine relative disagreement with absolute probability scale, selecting a fixed fraction of response tokens while preserving the signed OPD reward.The conclusion reports that moderate weighting suppresses low-low tokens while preserving useful disagreements, whereas excessive weighting can discard valuable supervision.Source: §7 · p. 10

A Related Work

• Prior OPD work addresses how to construct teacher supervision and optimize distillation objectives, while Fu et al. [8] report that one-query training recovers much of full-data OPD’s improvement.Self-distillation, reward transformations, geometric targets, policy-gradient methods, and entropy-dependent KL supervision are among the approaches reviewed.Source: A Related Work · p. 15
• Selective-supervision studies assess token learning value or use trajectory structure; DIAL-OPD targets the risk that disagreement favors tokens assigned low probability by both models.The reviewed token-level methods include uncertainty, divergence, surprise, teacher support, and task sensitivity criteria.Source: A Related Work · p. 15

B Experimental Details

• Students use Qwen3-0.6B-Base and Qwen3-1.7B-Base, and teachers use instruction-tuned Qwen3-4B and Qwen3-8B checkpoints; training uses AdamW, BF16, and rollouts of up to 1,024 response tokens.All methods use learning rate 5 × 10−7 and batch size 32; prompt, padding, and terminal EOS positions are excluded from training loss.Source: B Experimental Details · p. 16
• Main evaluation samples four responses on GSM8K, MATH-500, and OlympiadBench, eight on Minerva Math, and sixteen on AMC23, AIME24, and AIME25; Mean is the unweighted average of Avg across seven benchmarks.Avg denotes average correctness, Pass success in at least one sampled response, and Len mean response length.Source: B Experimental Details · p. 16

B.1 Supervision Density Experiment

• For supervision-density experiments, each response samples K = max(1, round(ρN)) eligible positions without replacement, refreshes masks every training step, and applies the original signed OPD loss only to selected positions.The same-size retention sweep uses Qwen3-4B-Base as student and Qwen3-4B as teacher.Source: §B.1 · p. 16

B.2 Baseline Implementations

• Baseline adaptations enforce matched per-response retention, including input-dependence ranking for SA-OPD, teacher-support ranking for TA-OPD, and initial-rollout fractions for Prefix OPD.Because the teacher’s pre-RL checkpoint was unavailable for CC-OPD, the authors use a smaller instruction-tuned model from the same family as reference.Source: §B.2 · p. 16

B.3 Low-Low Token Analysis

• The low-low analysis pools selected-token counts and absolute reward sums across responses before calculating low-low shares; replacement candidates are ranked by absolute reward while retaining original reward signs in the OPD loss.Replacement strategies use settings matched to the corresponding Reward-Top baseline.Source: §B.3 · p. 16

C Derivation of the Sampled-Token OPD Update

• The derivation assumes strictly positive, differentiable student and teacher probabilities over a finite vocabulary, bounded rollout lengths, and regularity for interchanging differentiation and expectation.The fixed-context objective differentiates student distributions at rollout-policy contexts and evaluates the gradient at the rollout parameters.Source: C Derivation of the Sampled-Token OPD… · p. 17
• For a fixed context, the negative gradient of reverse KL becomes an expectation of the teacher–student log-ratio multiplied by the student’s log-probability gradient.The log-ratio serves as the token-level reward in the REINFORCE form.Source: C Derivation of the Sampled-Token OPD… · p. 17
• Averaging the conditional identity over rollout contexts yields the sampled-token update, with each sampled token providing a Monte Carlo sample of the conditional expectation.The estimate is unbiased by conditional expectation and linearity even though tokens within a response need not be independent.Source: C Derivation of the Sampled-Token OPD… · p. 18
• The stop-gradient surrogate reproduces the required gradient at the rollout policy; it is not obtained by differentiating the sampled log-ratio alone.The analytical KL gradient accounts for the reward’s explicit parameter dependence, whose additional term cancels by normalization.Source: C Derivation of the Sampled-Token OPD… · p. 18
• The unbiasedness result concerns conditional reverse-KL gradients at sampled contexts, not derivatives through how earlier token choices change later contexts.Contexts are refreshed from new student rollouts during training, so holding their distribution fixed during differentiation does not make training off-policy.Source: C Derivation of the Sampled-Token OPD… · pp. 17–18

D Probability Weighting Ablation

• At 40% retention, β = 1 improves seven-benchmark mean accuracy by 6.96 percentage points and GSM8K Avg@4 by 15.96 points over β = 0.The comparison uses a Qwen3-4B teacher and Qwen3-1.7B-Base student; β = 0 corresponds to Reward-Top.Source: D Probability Weighting Ablation · p. 18

E.1 Proof of Proposition 1

• Scaling both probabilities by ε preserves their ratio, while β > 0 reduces the selection score and larger β strengthens suppression.At β = 0, the score is unchanged under the same proportional scaling.Source: §E.1 · pp. 18–19

E.2 Proof of Proposition 2

• The score is symmetric in teacher and student probabilities, so either probability approaching zero has the same limiting behavior.The derivation restricts attention to positive probabilities; the score need not be defined at zero.Source: §E.2 · p. 19
• As b approaches zero, the score diverges for 0 ≤ β < 1, tends to a for β = 1, and tends to zero for β > 1.For β > 1, the zero is a limit for arbitrarily small positive b, not the score at finite b.Source: §E.2 · p. 20