PaperlayerView exact revision on arXiv

Source-linked AI summary

AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals

Duy Nguyen, Hanqi Xiao, Archiki Prasad, Zaid Khan, Anirban Das, Austin Zhang, Sambit Sahu, Hyunji Lee, Elias Stengel-Eskin, Mohit Bansal

arXiv:2605.20643v1cs.LGcs.AIcs.CL

TL;DR

Self-distillation offers on-policy token-level guidance, but choosing a privileged teacher view is difficult because its best type can vary by task. AVSD reconstructs supervision from multiple views, combining shared consensus with selectively gated residual support; it outperforms single-view self-distillation and GRPO across the reported math and code benchmarks.

  • Problem

    The best type of privileged information can vary by task, while self-distillation methods typically choose one view despite the possibility that the teacher relies on information unavailable to the student at inference.

    Source: §1 · p. 2

  • Method

    AVSD separates multi-view teacher signals into shared consensus and view-specific residuals, adding residual support only when its direction agrees with consensus and its magnitude is proportionate.

    Source: §1 · p. 2

  • Results

    AVSD improves over single-view self-distillation and GRPO across math benchmarks and over single-view self-distillation on code; its Qwen3-8B math gain over the strongest baseline is 3.1%.

    Source: §1 · p. 3, §3.2 · p. 8

  • Takeaways & Limitations

    Analyses indicate that AVSD produces more reliable token-level credit signals than sparse-reward GRPO and single-view self-distillation.

    Source: §4.2 · p. 10, §6 · p. 11

    Evaluation focuses primarily on math and code; broader settings such as tool-use agents remain to be validated.

    Source: A Limitations and Broader Impact · p. 14

Abstract

from arXiv · show

Self-distillation enables language models to learn on-policy from their own trajectories by using the same model as both student and teacher, with the teacher being conditioned on privileged information unavailable to the student. Such information can come in different types or views, such as solutions, demonstrations, feedback, or final answers. This setup provides dense token-level feedback without relying on a separate external model, but creates a fundamental asymmetry: the teacher may rely on view-specific information that the student cannot access at inference time. Moreover, the best type of privileged information is often task-dependent, making it difficult to choose a single teacher view. In this work, we address both these challenges jointly by introducing AVSD (Adaptive-View Self-Distillation), a novel method of self-distillation with multiple privileged-information views, which reconstructs token-level supervision by separating stable cross-view consensus from view-specific residual signals. AVSD identifies the consensus signal shared across views, which provides a reliable update direction, and then selectively adds the view-specific residual signal to adjust the update magnitude when it both aligns with the consensus direction and remains proportionate to the consensus signal. Experiments on math competition benchmarks (AIME24, AIME25, and HMMT25) show that AVSD consistently outperforms both single-view self-distillation baselines and GRPO, achieving average Avg@8 gains of 3.1% and 2.2% over the strongest baselines on Qwen3-8B and Qwen3-4B, respectively. Moreover, on code-generation benchmarks (Codeforces, LiveCodeBench v6) using Qwen3-8B, AVSD outperforms the single-view self-distillation baseline by 2.4% on average.

1 Introduction

Self-distillation can provide dense, on-policy guidance without an external teacher, but the best privileged view varies by task and no single view is guaranteed to be optimal. AVSD combines multiple views by retaining their shared token-level signal and selectively adding reliable view-specific support.

1 Introduction

  • A single privileged view is not uniformly optimal: full solutions, partial reasoning, and final answers can yield different performance across datasets.The best choice can depend on the task and model regime, and each view has trade-offs in information and alignment with the student's reasoning.Source: §1 · p. 2, Figure 1 · p. 2
  • AVSD uses multiple privileged-information views and separates their teacher signals into a shared consensus signal and view-specific residual support.The consensus supplies a reliable update direction; the residual captures additional support from one or a few views.Source: §1 · p. 2
  • AVSD adds residual support only when views agree on its promote-or-suppress direction and its magnitude remains proportionate to the consensus signal.This lets AVSD use complementary information while preventing any single view from dominating the token-level signal.Source: §1 · p. 3
  • AVSD improves over the strongest baseline by 3.1% on Qwen3-8B and 2.2% on Qwen3-4B math benchmarks, and by 2.4% over single-view self-distillation on Qwen3-8B code benchmarks.The reported math results also compare AVSD with GRPO; the code evaluation covers Codeforces and LiveCodeBench v6.Source: §1 · p. 3

2 AVSD: Adaptive-View Self-Distillation

AVSD distills from multiple privileged teacher views by anchoring token updates in cross-view consensus and selectively adding view-specific residuals. It gates those residuals to preserve the consensus direction while retaining useful complementary support.

2 AVSD: Adaptive-View Self-Distillation

  • AVSD separates shared consensus from view-specific residuals, then adds residual support only when views agree on update direction and its magnitude remains proportionate.This is intended to retain complementary signals without letting them freely determine the student update.Source: §2 · p. 3, §2.3 · p. 6

2.1 Problem Setup

  • At each student prefix, multiple privileged views induce teacher distributions that reveal how each view would promote or suppress the same generated token.The student sees the prompt, while the teacher conditions on training-time privileged information; token-level advantages encode promote-or-suppress updates.Source: §2.1 · pp. 3–4

2.2 Consensus and Residual Signal

  • The geometric target emphasizes tokens supported across views, while the arithmetic target retains support from any view; their gap measures support omitted by strict consensus.Directly distilling the permissive arithmetic target can be noisy when support reflects view-specific artifacts.Source: §2.2 · pp. 4–5

2.3 Gated Reconstruction of Token-Level Supervision via Consensus and Residual Signal

  • The alignment gate suppresses residuals when views conflict, while the magnitude gate bounds their contribution so they cannot reverse the consensus update.The gate is low when positive and negative per-view updates cancel, and limits residual influence relative to consensus.Source: §2.3 · p. 6
  • AVSD trains by on-policy reverse-KL distillation to a reconstructed target, with gradients flowing through the student distribution while teacher distributions remain fixed.The views share the same generated trajectory, so AVSD adds teacher-context evaluations rather than student rollouts, with modest overhead in practice.Source: §2.3 · p. 7

3 Experiments

AVSD is evaluated on math and code reasoning using three base models, competition benchmarks, and single-view and RL baselines. It achieves the best reported math averages across the models and improves code averages over the base model and OPSD.

3.1 Experimental Setup

  • Math training uses OpenThoughts with full-solution, final-answer, and partial-solution views; evaluation covers AIME 2024, AIME 2025, and HMMT 2025.Code training uses 5K Codeforces Python examples with reference implementations, algorithmic hints, and execution feedback; evaluation includes 100 held-out Codeforces problems and LiveCodeBench v6.Source: §3.1 · p. 7
  • The comparisons include SFT, GRPO, and OPSD, with OPSD using the same reverse-KL objective but a single privileged view.The OPSD teacher uses a full solution for math and a reference implementation for code.Source: §3.1 · pp. 7–8

3.2 Results: AVSD Improves Math and Code Reasoning over GRPO and OPSD

  • AVSD achieves the best math Avg@8 for all three base models, exceeding the strongest baseline by 2.2% on Qwen3-4B and 3.1% on Qwen3-8B.For DeepSeek-R1-Distill-Qwen-7B, it improves over the strongest self-distillation baseline by 2.3% on average.Source: Table 1 · p. 8, §3.2 · p. 8
  • On Qwen3-8B, AVSD improves average code Avg@8 on Codeforces and LiveCodeBench over the base model and OPSD by 4.9% and 2.4%, respectively.Qwen3-4B and DeepSeek-R1-Distill-Qwen-7B also achieve the best code average among compared methods.Source: Table 1 · p. 8, §3.2 · p. 8

4 Analysis

The analysis compares target aggregation, token-credit reliability, and the effects of adding privileged views. Across these tests, AVSD performs best in the ablation comparison, gives more reliable token-level credit, and generally benefits from additional views.

4.1 Ablating Consensus and Arithmetic Targets

  • AVSD achieves the best Avg@8 across datasets, while consensus-only and arithmetic-only targets are not uniformly optimal.The ablation compares single-view OPSD, consensus-only distillation, arithmetic-only distillation, and full AVSD on Qwen3-4B.Source: §4.1 · p. 8, Figure 3 · p. 9

4.2 Token-Level Credit Allocation

  • 29.6% of examples leave GRPO without a learning signal, while incorrect-sign credit falls from OPSD’s 12.3% to AVSD’s 2.9% of high-impact tokens.For a single incorrect rollout, AVSD also gives more coherent credit on key tokens than single-view teachers.Source: §4.2 · p. 9, Figure 4 · p. 9

4.3 Scaling the Number of Views

  • Moving from one to three math views improves Avg@8 by 2.9% on AIME24 and 2.8% on AIME25; a fourth view adds only 0.7% and 0.2%, respectively.With four views, performance remains comparable on HMMT25.Source: §4.3 · p. 10
  • Moving from one to three code views improves Codeforces by 2.0%, and final performance generally improves as complementary views are added.The fourth code view provides a clearer benefit than the fourth math view.Source: §4.3 · p. 10

5 Related Work

Prior work uses privileged information in several training approaches, but does not settle how to construct token-level signals when multiple views are available. AVSD distinguishes shared cross-view support from view-specific support and derives its aggregation from the reverse-KL learning signal.

Learning from Privileged Information

  • Prior work uses privileged or derived information in sparse-reward RLVR [Shao et al., 2024, DeepSeek-AI, 2025] and self-improvement or bootstrapping [Zelikman et al., 2022, Wang et al., 2023].Other approaches use privileged information for exploration or curriculum construction.Source: §5 · p. 10
  • Existing approaches generally improve trajectories, hints, or curricula rather than construct token-level learning signals from simultaneously available privileged views.AVSD instead aggregates multiple views by separating cross-view consensus from view-specific residual support.Source: §5 · p. 11

On-Policy Self-Distillation and Teacher Aggregation

  • Prior self-distillation and ensemble approaches do not characterize stable versus context-specific teacher signals; AVSD derives its aggregation structure from the reverse-KL learning signal.Ensemble distillation methods aggregate teachers by averaging logits or votes.Source: §5 · p. 11

6 Conclusion

The conclusion presents AVSD as multi-view privileged self-distillation that reconstructs token-level learning signals from shared structure across teacher views. Experiments on math and code benchmarks show improvements over single-view self-distillation and GRPO, alongside more reliable token-level credit signals.

6 Conclusion

  • AVSD reconstructs token-level learning signals from shared structure across privileged views and improves over single-view self-distillation and GRPO on math and code benchmarks.The analyses indicate that it produces more reliable token-level credit signals.Source: §6 · p. 11

A Limitations and Broader Impact

AVSD’s validation and guarantees have explicit scope boundaries, while the appendix supplies additional derivations, implementation details, and discussion of other losses. The limitations section also flags data quality and broader-impact concerns.

A Limitations and Broader Impact

  • AVSD focuses primarily on math and code, depends on constructed-view and data quality, and remains to be validated in broader settings such as tool-use agents.Noisy annotated math solutions may produce less reliable teacher signals; the advantage interpretation and bounded-residual guarantee require further study for objectives beyond reverse KL.Source: A Limitations and Broader Impact · p. 14
  • Stronger reasoning models may be misused, and training data can contain biases or artifacts, so careful evaluation, deployment safeguards, and high-quality data remain necessary.AVSD’s design does not remove the need to evaluate unintended behavior.Source: A Limitations and Broader Impact · p. 14

B Additional Details for AVSD

  • Appendix B covers the reverse-KL sampled-token advantage, pooled-target derivations, gated reconstruction and boundedness, the training algorithm, and applicability to other losses.Source: B Additional Details for AVSD · p. 14

B.1 Reverse-KL Token Advantage

  • The reverse-KL objective yields a token-level distillation advantage; its sampled-token estimator uses the generated token at the visited student prefix.The constant 1 acts as a baseline because the expected student log-probability gradient is zero.Source: §B.1 · p. 15

B.2 Derivations of Two Pooled Targets

The appendix relates geometric and arithmetic pooled targets to opinion pooling, extends them to weighted views, and establishes that the reconstructed advantage preserves the consensus update direction.

B.2 Derivations of Two Pooled Targets

  • The geometric and arithmetic targets correspond respectively to log-linear and linear opinion pooling, capturing cross-view consensus and view-marginal support.The pooling perspective extends to non-uniform view weights.Source: §B.2 · p. 16
  • Uniform view weights keep the main experiments parameter-free and avoid additional view-reliability hyperparameters.Weighted targets use view-specific reliability weights in the pooled objectives.Source: §B.2 · p. 16

B.3 Bounded Residual Property

  • The bounded residual can strengthen a positive consensus update or soften a negative one, but cannot flip the consensus-induced direction.Source: §B.3 · p. 16

B.4 Detailed Training Algorithm

AVSD trains on student-generated trajectories while evaluating multiple privileged views at each prefix. The added batched teacher evaluations increase training time modestly without requiring extra student rollouts.

B.4 Detailed Training Algorithm

  • Training time rises from 41.5s to 46.7s per step, a 1.13× relative cost, because AVSD adds batched teacher evaluations but shares the student rollout.The reported comparison uses Qwen3-4B under the same batch size and hardware setup.Source: §B.4 · p. 16
  • The algorithm uses M = 3 views in the experiments and updates the student by minimizing the AVSD loss.Source: §B.4 · p. 17
  • The procedure constructs privileged views, samples an on-policy rollout, and evaluates student and view-conditioned teacher distributions at each prefix.Source: §B.4 · p. 17

B.5 Applicability to other Distillation Losses

AVSD’s reconstructed teacher target can in principle be paired with other distribution-matching losses, but its sampled-token policy-gradient interpretation and guarantees are most directly aligned with reverse-KL on-policy self-distillation.

B.5 Applicability to other Distillation Losses

  • The multi-view reconstructed target is not tied to one divergence and can in principle be paired with forward KL or symmetric objectives such as Jensen–Shannon divergence.Source: §B.5 · p. 17
  • Forward KL takes expectation under the reconstructed teacher, requiring full-vocabulary supervision or teacher-sampled tokens and losing the same sampled-token advantage interpretation.Source: §B.5 · p. 18
  • AVSD’s main design and guarantees are most directly aligned with reverse-KL on-policy self-distillation; testing whether reconstruction helps other objectives remains future work.Source: §B.5 · p. 18

C Implementation Details of Experiments

The experiments disable Qwen3 thinking mode during training and specify shared evaluation settings and GPU configurations.

C Implementation Details of Experiments

D Additional Experiments

Additional experiments find that privileged-view strengths vary by task and that AVSD outperforms the strongest single-view code baseline on both reported benchmarks. They also examine residual-gate use, token-credit reliability, example views, and the training prompt.

D Additional Experiments

E Examples of Multi-View Privileged Information

F Prompts

  • The training prompt frames a reference—which may be an answer, hint, partial solution, or full solution—as hidden guidance.Source: F Prompts · pp. 24–25

Related papers

Loading 2605.20643v1…