PaperlayerView exact revision on arXiv

Source-linked AI summary

DualOPSD: Adaptive Privileged Teachers for On-Policy Self-Distillation

Yutong Chen, Guangfu Guo, Zhichao Xu, Kunpeng Liu

arXiv:2608.26019v1cs.LGcs.AI

TL;DR

OPSD’s privileged teacher remains fixed while the student’s distribution and style change, motivating adaptive teacher learning. DualOPSD alternates clipped student learning with full teacher adaptation on the same trajectory. It improves accuracy at larger Qwen3 scales, reduces truncation across scales, and shows scale-dependent gains.

  • Problem

    OPSD uses a fixed privileged teacher even as the student distribution and output style change during training.

  • Method

    DualOPSD alternates a clipped student update with full reverse-KL teacher adaptation toward the updated student on the same trajectory.

  • Results

    DualOPSD improves avg@12 over OPSD by 23.61, 13.89, and 10.00 points on AIME 2024, AIME 2025, and HMMT 2025 at Qwen3-8B.

  • Takeaways & Limitations

    Across 1.7B, 4B, and 8B, DualOPSD reduces truncation, while its accuracy effect improves at 4B and 8B but not at 1.7B.

  • Takeaways & Limitations

    Evidence covers one model family, one training seed, and the same 90 competition problems at each scale; broader replication is needed.

Abstract

from arXiv · show

On-policy self-distillation (OPSD) uses a privileged copy of the student model to provide dense supervision without an external teacher. OPSD keeps this privileged teacher fixed, even though the student distribution and output style change during training. We propose DualOPSD, an asymmetric alternating framework that adapts both policies. The student first learns from the privileged teacher. The teacher then moves toward the updated student distribution on the same student trajectory. This update makes later supervision responsive to the learner and does not require another rollout. On Qwen3-8B in non-thinking mode, DualOPSD improves avg@12 over OPSD by 23.61, 13.89, and 10.00 points on AIME 2024, AIME 2025, and HMMT 2025. Results at 1.7B and 4B show that the accuracy gain depends on model scale. Across all three scales, DualOPSD reduces truncation. The 4B diagnostic also shows lower KL in both directions between the teacher and student.

1 INTRODUCTION

OPSD provides dense on-policy supervision through a privileged policy, but its fixed teacher cannot adapt to changing student prefixes and style. DualOPSD alternates conservative student learning with teacher adaptation on the same trajectory.

  • OPSD uses a privileged copy of the student to score every student prefix, providing dense supervision without an external teacher or verifier call during optimization.
  • A fixed privileged teacher can repeatedly present style-related signals that the student’s pointwise clipping rejects.Clipping protects the student update but gives the frozen teacher no feedback about rejected supervision.
  • DualOPSD alternates a clipped student update with full teacher adaptation toward the updated student on the same trajectory.The adapted teacher supplies targets for later student steps and changes future gradients without another rollout.
  • Teacher adaptation improves avg@12 at 4B and 8B but not at 1.7B, indicating that its accuracy effect depends on model scale.

2 BACKGROUND

On-policy distillation combines student-generated state coverage with dense token-level distribution matching. OPSD uses a privileged reference-conditioned policy and clips large signed vocabulary contributions to limit style-driven gradients.

  • SFT provides dense supervision on fixed expert prefixes, whereas RLVR samples current-policy responses but supplies sparse verifier feedback.
  • On-policy distillation evaluates token divergences along prefixes sampled from the student, reducing the state distribution gap between training and inference.Full-vocabulary distillation supervises every vocabulary entry at each completion position.
  • OPSD constructs a student conditioned on x and a privileged teacher conditioned on (x, r), with only the teacher seeing the verified reasoning trace.Both policies score the same student-sampled prefix.
  • The pointwise clipped KL surrogate caps large signed vocabulary terms before summation, preventing style tokens from overwhelming the student update.The clipped sum is not itself a divergence, and the frozen teacher cannot respond to repeatedly rejected style signals.

3 DUALOPSD: ADAPTIVE PRIVILEGED TEACHER LEARNING

DualOPSD replaces fixed teacher targets with a sequential closed loop: the student updates from clipped privileged supervision, then the teacher adapts on the same student trajectory. Separate adapters and optimizer states preserve the asymmetric roles.

  • 3.1 FROM FIXED TARGETS TO CLOSED LOOP ADAPTATION: Fixed-teacher OPSD evaluates the student loss on the current student trajectory while reusing the initial teacher throughout training.The teacher therefore does not track changing student prefixes or style.
  • 3.1 FROM FIXED TARGETS TO CLOSED LOOP ADAPTATION: DualOPSD updates the student first and then updates the privileged teacher using the same trajectory and the updated student parameters.The resulting teacher supplies targets at the next step, changing later supervision.
  • 3.2 ASYMMETRIC ALTERNATING UPDATES: Both updates reuse one student rollout; the teacher does not generate a response and adds one privileged forward and backward pass after the student step.
  • 3.2 ASYMMETRIC ALTERNATING UPDATES: The procedure samples one response, scores student prefixes, performs sequential updates, and returns updated student and teacher policies.
  • 3.2 ASYMMETRIC ALTERNATING UPDATES: DualOPSD retains the upper clip for the student so large style contributions do not dominate its update.
  • 3.2 ASYMMETRIC ALTERNATING UPDATES: The teacher minimizes reverse KL DKL(pS∥pT) with the stopped student distribution as target, while gradients pass only through teacher logits.Teacher vocabulary contributions are unclipped, unlike the student’s clipped privileged signal.
  • 3.2 ASYMMETRIC ALTERNATING UPDATES: The student and teacher are implemented as separate LoRA adapters with separate parameters and AdamW optimizer states on one frozen base model.Only one adapter is active at a time.

4 EXPERIMENTS

Experiments compare posttraining methods across three Qwen3 scales and competition-mathematics benchmarks, showing that DualOPSD’s accuracy benefit is scale-dependent while truncation decreases consistently. Diagnostics indicate that adaptation lowers teacher–student KL, with moderate additional training cost.

  • Experimental setup: Experiments use Qwen3-1.7B, Qwen3-4B, and Qwen3-8B in non-thinking mode, with 29,434 mathematics problems for training.All methods start from the same base checkpoint within each scale; student rollouts are limited to 1,024 tokens.
  • Experimental setup: Evaluation covers AIME 2024, AIME 2025, and HMMT February 2025 using 12 samples per problem and avg@12 as the primary accuracy measure.Answers are extracted from boxed completions and checked with math_verify, with a project-verifier fallback when parsing fails.
  • Main results: DualOPSD changes avg@12 relative to OPSD by −3.61/−1.94/−1.94 points at 1.7B, +13.06/+5.00/+6.67 at 4B, and +23.61/+13.89/+10.00 at 8B.The task order is AIME24, AIME25, and HMMT25; the gain grows from 4B to 8B but reverses at 1.7B.
  • Main results: Lower truncation is consistent across scales but does not explain accuracy: pooled 4B truncation falls from 9.63% with OPSD to 2.31% with DualOPSD.At 1.7B the rate falls from 3.24% to 1.20%, and at 8B from 8.43% to 3.33%; accuracy rises at 8B but falls at 1.7B.
  • Training cost: At 4B, DualOPSD training takes about 1.40 hours on one A100 versus 1.00 hour for OPSD.The added cost comes from an additional teacher forward and backward pass, while student optimizer steps and sampled trajectories remain fixed.
  • Teacher and student adaptation dynamics: DualOPSD reaches DKL(pT ∥pS) = 0.171 at step 100, 50.2% below OPSD, while DKL(pS∥pT ) = 0.064, 45.9% below OPSD.The diagnostic uses shared valid tokens, fixed prefixes, and masks to isolate adapter coupling.

5 ABLATION STUDY

The ablation study tests how the teacher update and its clipping affect performance. Results favor learning the teacher with the full reverse-KL signal while retaining conservative student supervision.

  • Teacher update ablation: DualOPSD-C replaces the standard teacher objective with a lower-clipped variant to test whether adaptation benefits from the full reverse-KL signal.The lower clip removes gradients from vocabulary contributions at or below τ and operates on signed terms, so the objective is not a divergence.
  • Teacher–student coupling: Figure 2 measures adapter coupling with full-vocabulary KL using identical contexts, fixed reference prefixes, and 3,983 valid tokens per checkpoint.The vertical axis uses a linear scale.
  • Teacher update ablation: 28.80% at step 100 exceeds the 23.43% OPSD mean by 5.37 points under the same student update.DualOPSD-C rises from 23.15% at step 50 to 28.80% at step 100, separating teacher adaptation from additional student steps.
  • Teacher update ablation: The full teacher signal raises the step-100 mean by another 2.87 points over the tested lower clip.The gains are 1.39, 1.67, and 5.56 points on AIME24, AIME25, and HMMT25.

6 RELATED WORK

Related work spans supervised fine-tuning, reinforcement learning with verifiable rewards, classical distillation, and privileged-information methods. DualOPSD combines on-policy student states with dense privileged supervision while adapting a same-model teacher.

  • Supervised fine-tuning and reinforcement learning: SFT provides dense token supervision on fixed expert prefixes, whereas RLVR samples on-policy responses and receives sparse sequence-level verifier feedback.These approaches trade dense token feedback against on-policy state coverage.
  • Knowledge distillation: Classical and sequence-level distillation transfer soft teacher probabilities but usually assume an external teacher.MiniLLM and GKD extend generative distillation with reverse KL, policy gradients, on-policy sampling, or multiple divergences.
  • On-policy self-distillation: OPSD creates a privileged teacher from the same base model, which scores token distributions along student responses without an external teacher or verifier.Its privileged target remains fixed during training.
  • DualOPSD: DualOPSD extends OPSD by adapting the privileged teacher toward the updated student on the same trajectory, changing later supervision without another rollout.This combines the student-state coverage associated with on-policy methods and dense token feedback.

7 CONCLUSION AND LIMITATIONS

DualOPSD forms an asymmetric closed loop in which the student learns conservatively from a privileged target and the teacher adapts on the same trajectory. Across Qwen3 scales, it reduces truncation, while accuracy gains depend strongly on model size and the evidence remains limited in scope.

  • Conclusion: DualOPSD reduces truncation across 1.7B, 4B, and 8B Qwen3 models without requiring another rollout.Its accuracy improves over OPSD at 4B and 8B but not at 1.7B.
  • Conclusion: The 4B diagnostic shows lower KL in both directions between teacher and student during training.This indicates that policy coupling changes under teacher adaptation.
  • Limitations: The evidence covers one model family, one training seed, and the same 90 competition problems at each scale.The 8B result therefore needs replication across seeds and broader answer auditing.
  • Limitations: The 1.7B reversal shows that DualOPSD's accuracy effect is not independent of model scale.Lower KL alone does not establish better reasoning or continued use of the privileged reference.

A.1 EXACT TRAINING PROMPTS

The appendix specifies the student and teacher prompts, their shared chat-template settings, and the evaluation configuration. It also identifies Table 4 as reporting pooled generation behavior and wall time for 1.7B and 8B models.

  • Exact training prompts: The student prompt requests step-by-step reasoning and places the final answer within \boxed{}.The teacher prompt supplies a reference solution before requesting independent reasoning toward the same final answer.
  • Reported appendix measurements: Table 4 reports pooled truncation, mean completion length without truncation, and one-A100 wall time for 1.7B and 8B models.The caption states that these generation statistics are pooled over AIME24, AIME25, and HMMT25.
  • Exact training prompts: Both message configurations use the Qwen chat template with add_generation_prompt=True and enable_thinking=False.The evaluation message omits the initial Problem: label while retaining the final-answer instruction and chat settings.

A.2 OPTIMIZATION AND UPDATE ORDER

Each batch produces one student rollout per problem, then updates the student before recomputing logits and updating the teacher on the same token IDs and mask.

  • Update order: Each batch of 32 prompts produces one student rollout per problem, with each rollout limited to 1,024 tokens.The student adapter is scored and updated first.
  • Update order: After the student update, its logits are recomputed without gradients before the teacher adapter is activated.The teacher is scored with privileged prompts on the same token IDs and mask, then updated.
  • Scoring implementation: Full-vocabulary scoring uses token segments of length 1,024 and row microbatches selected for each model scale.

A.3 BASELINE BUDGETS

The appendix defines baseline training budgets, wall-time accounting, truncation detection, and answer-verification procedures for the compared methods.

  • Baseline budgets: SFT uses 100 steps, global batch 64, maximum sequence length 16,000, and learning rate 5 × 10^-6.
  • Baseline budgets: GRPO uses 200 steps, effective batch 32, eight generations of at most 8,192 tokens, two iterations, β = 0, and temperature 1.0.
  • Baseline budgets: PiDistill and BRTS use four privileged candidates per problem, 1,024-token rollouts, and 100 steps.They are comparison variants defined in this paper rather than cited external methods.
  • Cost accounting: Reported wall time includes rollout generation, scoring, optimization, verifier execution when used, checkpoint I/O, and systems overhead on one A100.These measurements compare actual implementations and are not hardware-independent measures of algorithmic FLOPs.
  • Evaluation rules: A sample is truncated when its finish reason is length or when it reaches 32,768 − |tokenized prompt| tokens.Verification parses the last balanced \boxed{...} expression, with fallback extraction and legacy verification for failures.
Loading 2608.26019v1…