Influence-Directed Distillation: Solving the Diversity Bottleneck in Sampled-Token On-Policy Distillation

• Sampled-token OPD frequently improves pass@1 while pass@k plateaus, failing to transfer the teacher’s generation diversity.Source: §1 · p. 1
• IDA-OPD preserves entropy-expanding updates and applies divergence-adaptive advantage shrinkage to entropy-contracting updates, strongest at small teacher–student discrepancies.Source: §1 · pp. 1–2
• IDA-OPD improves pass@k under the same sampled-token budget, matches strong teacher-informed methods at lower information cost, and broadly maintains pass@1.Source: §1 · p. 2
• Entropy-contracting updates, including many low-discrepancy positions, explain the diversity distillation failure.Source: §1 · pp. 1–2
• First-Order Local Entropy Influence separates each update’s entropy effect into the teacher–student log-probability gap and the student’s local probability structure.Source: §1 · pp. 1–2

• OPD optimizes reverse KL between student and teacher next-token distributions on student-induced prefixes.Source: §2 · p. 2
• Exact reverse-KL computation requires full-vocabulary teacher logits at every prefix, making it prohibitively expensive.Source: §2 · p. 2
• Sampled-token OPD estimates reverse KL by drawing one token from the student distribution and querying only that token’s teacher log-probability.Source: §2 · p. 2
• Although locally unbiased, the one-token estimator can reduce diversity and cause premature entropy collapse.Source: §2 · p. 2

3.1 Advantage Is Insufficient

• On real training positions, advantage alone has a weak relationship with measured entropy change because identical advantages span both signs.This motivates modeling the advantage jointly with the student’s current distribution.Source: §3.1 · p. 2, §3 · p. 2
• The same positive advantage can decrease or increase entropy depending on whether it reinforces a high- or low-probability token.High-probability reinforcement sharpens the distribution, whereas low-probability reinforcement spreads probability mass.Source: §3.1 · p. 2, Figure 1 · p. 3

3.2 First-Order Local Entropy Influence

• Figure 1 shows that IH(y) tracks measured one-step entropy change and predicts its sign more closely than the advantage.This connects the analytical decomposition to real training positions.Source: Figure 1 · p. 3, §3.2 · p. 3
• Under a local logit-space step, the entropy change satisfies ∆H = η · IH(y) + O(η2), with IH(y) = AyDy.The influence’s sign provides a token-wise signal for identifying entropy-contracting and entropy-expanding updates.Source: §3.2 · p. 3

3.3 Where Entropy Loss Originates

• Cumulative entropy contraction peaks near δy ≈0, where many individually weak updates accumulate into the dominant entropy drain.A secondary entropy-loss cluster remains at the high-divergence negative tail.Source: Figure 2 · p. 3

3.2 First-Order Local Entropy Influence

• First-Order Local Entropy Influence decomposes the leading entropy effect into the teacher–student log-probability gap and the student’s local probability structure.The advantage scales the update, while the student distribution determines its direction-dependent effect.Source: §3.2 · pp. 2–3

4.1 Diagnosing the Entropy Drain

• High-divergence updates contribute to entropy loss, but cumulative contraction is dominated by the low-discrepancy region near δy ≈0.The dominant effect arises from the large number of highly aligned tokens, each with minimal individual impact.Source: §4 · pp. 3–4
• Uniformly penalizing all updates with IH(y) < 0 can suppress critical high-divergence teacher signals while failing to target the main entropy drain.The discrepancy distribution motivates selective rather than uniform intervention.Source: §4 · pp. 3–4

4.2 Divergence-Adaptive Shrinkage

• IDA-OPD attenuates Ay only at entropy-contracting positions, using a symmetric relative disagreement weight wy to measure remaining correction need.The method preserves entropy-expanding updates and requires only sampled-token teacher information.Source: §4.1 · p. 4
• The retention weight wy is small when teacher and student agree and approaches 1 as disagreement grows, without introducing signal beyond Ay.This makes attenuation strongest for low-discrepancy updates and weakest for high-discrepancy corrections.Source: §4.1 · p. 4
• The shrinkage is sign-preserving: it never reverses teacher corrections, vanishes near agreement, and leaves high-disagreement corrections intact.Proposition 1 formalizes near-quadratic attenuation as Ay →0 and near-lossless preservation as |Ay| →∞.Source: §4.1 · p. 4

Main Results

• IDA-OPD attains the highest pass@16 across all four mathematics benchmarks at both 4B and 8B scales.It improves over standard OPD by +8.3 on HMMT Feb, +4.6 on AIME 2024, and +4.3 on AIME 2025 in the 4B setting.Source: §5.1 · p. 4
• IDA-OPD matches or exceeds teacher-informed AOPD and EOPD while avoiding their expensive top-K teacher distributions.It uses a teacher-free entropy diagnostic and no full-vocabulary teacher information.Source: §5.1 · p. 5
• +2.1/+0.9 on MBPP+ and +1.3/+1.0 on LiveCodeBench: IDA-OPD improves pass@1/pass@16 in the code domain.This extends the observed diversity-recovery pattern beyond mathematics.Source: §5.1 · p. 5

Pass@k Performance

• +4 to +5 points: IDA-OPD’s pass@k advantage over OPD emerges by k = 8–16 and persists through k = 64.The methods are close at k = 1, where a single high-probability mode can suffice.Source: §5.2 · p. 5, Figure 3 · p. 5

Ablation Study

• The influence gate and divergence-adaptive shrinkage are complementary, while linear shrinkage best balances suppressing low-discrepancy drain against preserving high-discrepancy corrections.Using sign(Ay) misidentifies entropy-risk positions, whereas the linear mapping tops every reported Table 4 column.Source: §5.3 · p. 6, §5.4 · p. 6
• Hard masking protects entropy but overprotects: discarding every entropy-contracting update sacrifices knowledge signals and causes the worst pass@1 drop.The ablation shows that entropy control must preserve useful teacher corrections.Source: §5.5 · pp. 6–7

Token-Level Entropy Analysis

• IDA-OPD stays closer to the teacher’s token-entropy distribution by avoiding OPD’s low-entropy pile-up and retaining more high-entropy tokens.OPD places approximately 74% versus the teacher’s 57% in the 0.0–0.1 bin, while IDA-OPD remains near 55%.Source: §5.6 · p. 7

Main Results

• IDA-OPD retains OPD’s cheap O(L) teacher-information footprint, adding only an entropy term that reuses student logits from the forward pass.Teacher-informed EOPD and AOPD use truncated top-K teacher distributions with O(LselK), K ≫1.Source: §5.7 · p. 7

On-Policy Distillation for Language Models

• On-policy distillation trains on student-sampled prefixes, reducing the train–inference mismatch present in offline distillation.OPD variants differ in the per-token teacher information they require.Source: §6 · p. 7

Token Selection and Reweighting in OPD

• Token-selection methods reweight updates using teacher–student divergence, token importance, teachability, or optimization granularity.Examples include TIP, Token Teachability, and Filter-Then-Reweight.Source: §6 · p. 7
• Other methods decouple exploration from imitation, reshape sampled-token credits, or restore entropy with forward-KL penalties at uncertain tokens.AOPD, REOPOLD, negative-sample reinforcement, and EOPD represent these strategies.Source: §6 · p. 7
• EOPD and AOPD achieve strong diversity protection but reintroduce full or top-K teacher distributions, sacrificing sampled-token efficiency.These methods locate salient positions using teacher–student disagreement or uncertainty.Source: §6 · p. 7

7 Conclusion

• First-Order Local Entropy Influence flags whether sampled-token updates expand or contract entropy without requiring full teacher distributions.The proxy is described as cheap and teacher-free.Source: §7 · p. 7
• IDA-OPD preserves entropy-expanding updates and applies divergence-adaptive advantage shrinkage to entropy-contracting updates.The intervention uses the influence signal to modify only contracting updates.Source: §7 · p. 7
• IDA-OPD consistently improves pass@k over methods with the same sampled-token budget, reaches parity with the strongest teacher-informed methods at lower cost, and broadly maintains vanilla OPD’s pass@1.The reported comparisons use no full-vocabulary teacher information.Source: §7 · p. 7