Source-linked AI summary
VISTA: Verifier-Informed Student-to-Teacher Adaptation for On-Policy Self-Distillation
Zewen Ding, Zezhong Wu, Zhou Tao, Shida Wang, Shizhuo Hou, YongXiang Hua, Haoyu Cao, Linli Xu
TL;DR
Standard OPSD may misdirect problem-only reasoning because its privileged teacher is treated as a fixed superior target despite uneven teacher–student alignment. VISTA adapts the teacher toward the student on outcome-verified rollouts at the largest-disagreement positions while retaining standard student training, achieving the highest Avg@12 at every evaluated scale.
Problem
Standard OPSD relies on the teacher-superiority assumption, treating the privileged distribution as a better target throughout the student rollout even when valid problem-only alternatives differ.
Method
VISTA gates student-to-teacher adaptation by rollout correctness and applies it at the top-k positions with largest teacher–student KL divergence while preserving the standard OPSD student update.
Results
VISTA achieves the highest three-benchmark Avg@12 at all three Qwen3 scales, improving over reported OPSD by 0.6, 0.7, and 2.1 points at 1.7B, 4B, and 8B.
Takeaways & Limitations
The results support student supervision from outcome-verified rollouts and identify student-to-teacher adaptation as a promising direction for OPSD.
Takeaways & Limitations
Student behavior is not generally a better teacher target, and updating every token could erode complementary privileged knowledge; VISTA therefore relies on verified rollouts and sparse position selection.
Abstract
from arXiv · showhide
On-policy self-distillation (OPSD) improves reasoning by training a problem-only student on its own rollouts using dense token-level supervision from a privileged teacher that also sees a reference solution. However, standard OPSD treats the teacher distribution as a fixed target along the student's rollout and updates only the student %, although -- even though privileged conditioning does not guarantee that the teacher always provides the most appropriate target for problem-only reasoning. This one-way supervision can therefore misdirect the student when the teacher distribution is misaligned with valid student reasoning. We therefore introduce Verifier-Informed Student-to-Teacher Adaptation (VISTA), which preserves the standard OPSD student update while using outcome-verified rollouts to adapt the teacher toward the student distribution. Within each verified rollout, VISTA further restricts this adaptation to the top-$k$ positions with the largest teacher--student KL divergence. Notably, VISTA reuses the rollout and loss function from standard OPSD, introducing no additional sampling or separate reward objective. Across AIME24, AIME25, and HMMT25 with Qwen3 models at 1.7B, 4B, and 8B, VISTA achieves the highest Avg@12 at every scale, improving over OPSD by $0.6$, $0.7$, and $2.1$ points, respectively. These results demonstrate the value of student supervision from outcome-verified rollouts and highlight student-to-teacher adaptation as a promising direction for OPSD.
1 Introduction
Standard OPSD can misdirect problem-only reasoning because privileged teacher distributions are not always superior targets. VISTA addresses this by selectively adapting the teacher on outcome-verified student rollouts while retaining standard student training, and reports stronger performance across scales.
- OPSD assumes the privileged teacher is a better token-level target at every position, although reference conditioning does not guarantee superiority for problem-only reasoning.
- Reference-specific teacher preferences can prompt the student to mention information unavailable at inference or suppress valid alternative reasoning and self-correction.
- VISTA enables student-to-teacher adaptation only on outcome-verified rollouts and at the top-k positions with the largest teacher–student KL divergence.
- VISTA preserves the standard OPSD student update and reuses the same rollouts and loss function without additional sampling or a separate reward objective.
- VISTA achieves the highest three-benchmark Avg@12 at all three Qwen3 scales and state-of-the-art performance in eight of nine scale–benchmark settings.
- VISTA improves over reported OPSD by 0.6, 0.7, and 2.1 points at the 1.7B, 4B, and 8B scales, respectively.
2 Related Work
Prior work regulates privileged supervision mainly on the student side, while VISTA makes the teacher adaptive using verified rollouts and selective disagreement-based updates.
- Privileged-context distillation gives a teacher access to side information unavailable to the student while transferring predictive distributions to the student.
- Outcome verifiers provide sequence-level signals for whether mathematics or code rollouts reach valid final results, and RLVR uses them as policy-gradient rewards.
- Existing selective methods regulate student updates using outcome advantages, annotated reasoning spans, or teacher–student disagreement.
- Unlike these methods, VISTA applies selectivity to teacher adaptation by using outcome-verified problem-only rollouts and high-disagreement positions.
3 Method: VISTA
VISTA extends OPSD with selective student-to-teacher adaptation while retaining one-way student supervision. It gates teacher feedback to outcome-verified rollouts and top-k positions with the largest teacher–student disagreement.
- OPSD preliminaries: OPSD trains a problem-only student on its own rollouts to match a privileged teacher’s next-token distribution at every prefix.The teacher additionally observes a reference solution, while the student sees only the problem.
- Teacher-superiority assumption: Standard OPSD relies on the teacher-superiority assumption that the privileged distribution is a better target than the student throughout the rollout.Reference conditioning does not guarantee uniform suitability for problem-only reasoning.
- Teacher-superiority assumption: Teacher–student disagreement can misdirect learning by supporting reference-specific continuations or suppressing valid alternatives and self-correction.A representative shared-prefix case shows the student following valid reasoning while the teacher emits a reference-specific phrase.
- VISTA training objective: VISTA retains the OPSD student loss and adds a teacher loss that uses the current student trajectory and paired distributions without a new sampling loop or separate reward-optimization phase.Stop-gradient separates the student and teacher updates; the student uses forward KL, while teacher adaptation uses reverse KL.
- Rollout and position selection: Teacher adaptation occurs only on verifier-accepted rollouts, because failed outcomes reveal an error without identifying its location.This outcome gate prevents unidentified student errors from directly supervising the teacher.
- Rollout and position selection: Within eligible rollouts, VISTA updates the teacher only at up to k positions with the highest teacher–student KL divergence.Selective updates preserve disagreement and avoid rapidly shrinking the distributional gap.
4 Experiments
VISTA is evaluated on competition-level mathematical reasoning across Qwen3 scales, benchmarks, and ablations. It performs best when teacher adaptation uses verified rollouts and selectively targets high-disagreement positions.
- Evaluation: VISTA is evaluated on Qwen3-1.7B, Qwen3-4B, and Qwen3-8B using AIME24, AIME25, and HMMT25 with Avg@12.The evaluation follows the matched OPSD protocol and compares against Base, SFT, GRPO, SDPO, and standard OPSD.
- Main results: 0.6, 0.7, and 2.1 points are VISTA’s improvements over reported OPSD at the 1.7B, 4B, and 8B scales, respectively.VISTA achieves the highest three-benchmark Avg@12 at every scale and eight state-of-the-art results across nine scale–benchmark settings.
- Outcome-gate ablation: 66.9 Avg@12 is achieved by VISTA with outcome-verified rollouts, versus 64.5 when adapting only on verifier-rejected rollouts.Random gating and all-rollout adaptation reach 65.6 and 65.8, respectively, compared with OPSD’s 64.8.
- Rollout-position selection: 66.9 is the best score among position-selection variants, with VISTA’s top-k KL rule outperforming first-k, random-k, and last-k alternatives.The comparison uses Qwen3-8B and Avg@12 across three benchmarks.
- Top-k selection size: 78.6 on AIME24 and 73.9 on AIME25 are obtained at k = 16, while selection sizes below 16 and above 32 underperform intermediate choices.The sweep evaluates k ∈{8, 16, 32, 64} and an All setting.
- Teacher-side diagnostics: Explicit reference attribution falls from 63% initially to 48% at steps 75 and 100 during teacher adaptation.The analysis uses validated student trajectories and interprets the decline as weaker preference for reference-dependent continuations.
- Teacher capability: VISTA maintains perfect teacher Pass@12 while teacher Avg@12 gradually increases during training.This analysis evaluates the teacher with reference answers and reports that teacher adaptation does not compromise its problem-solving performance.
5 Conclusion
VISTA extends OPSD with selective student-to-teacher adaptation while preserving standard student training and inference-time deployment. Across scales and benchmarks, it combines verified feedback with high-disagreement positions and improves reported performance.
- Conclusion: VISTA adapts the privileged teacher toward student distributions at top-k KL-divergence positions within outcome-verified rollouts.The method preserves the standard student update and reuses the existing rollout and KL objective without additional sampling or a separate reward objective.
- Conclusion: VISTA achieves the highest Avg@12 at every Qwen3 scale across three competition-math benchmarks.Ablations favor outcome-verified, high-disagreement feedback over indiscriminate or random teacher adaptation.
- Conclusion: Teacher-side analyses indicate better support for validated student trajectories, less explicit reference attribution, and preserved privileged-teacher capability.Only the problem-only student is retained at inference.
B.1 Complete VISTA Training Procedure
The complete VISTA minibatch procedure keeps the standard OPSD student branch and adds a verifier-gated, top-k teacher-adaptation branch. Both parameter blocks are updated synchronously after evaluating the student and teacher on shared student prefixes.
- Minibatch procedure: For each problem–reference pair, the problem-only student samples one rollout, and student and teacher distributions are evaluated at the same student prefixes.The student loss is active at every rollout position.
- Teacher adaptation: The teacher loss activates only for verifier-accepted completed rollouts and the min(k, |y|) positions with the largest raw teacher-first KL scores.Rejected rollouts receive no teacher adaptation loss.
- Gradient routing: VISTA retains the standard OPSD student branch over all rollout positions and adds the masked teacher branch before synchronous parameter updates.The rollout and teacher-update mask are fixed for the update.
- Update strength: The implementation uses ηT instead of λ, with λ = ηT /ηS expressing the relative teacher-update strength.The student and teacher learning-rate arguments encode the same coefficient relationship.
B.2 Training Configuration
VISTA retains OPSD’s student-side pipeline while adding a trainable teacher branch, using matched Qwen3 models, OpenThoughts reasoning data, and problem-only versus reference-conditioned prompts.
- Lineage and data: VISTA initializes independent student and teacher adapters from the same instruct-tuned Qwen3 model and changes OPSD mainly through the trainable teacher branch.The student branch remains optimized at every valid rollout position.
- Data and budget: Training uses up to 30K OpenThoughts mathematical-reasoning problem–reference pairs, one rollout per prompt, batch size 32, and 100 updates.The reported schedule samples about 3,200 prompt–rollouts in total.
- Prompting: The student receives only the problem with thinking disabled, while the teacher receives the problem and reference solution with thinking enabled.The prompt construction follows OPSD’s procedure.
- Optimization: The core configuration uses full-vocabulary forward KL with β = 0, verifier-accepted rollouts, raw-KL top-k positions, and k = 32 for 1.7B/4B and k = 16 for 8B.Training uses ηS = 4 × 10−6, ηT = 3 × 10−6, and λ = 0.75.
B.3 Evaluation Configuration
The evaluation reuses OPSD’s released evaluation logic and applies a fixed benchmark, prompting, sampling, answer-checking, and checkpoint-reporting protocol.
- The implementation reuses OPSD’s problem prompt, Qwen3 chat templating, boxed-answer extraction, mathematical-equivalence checker, and Avg@N/Pass@N aggregation.
- All 30 problems from AIME 2024, AIME 2025, and HMMT 2025 are evaluated with a fixed step-by-step prompt requiring boxed final answers.
- Each problem receives 12 completions from one vLLM sampling call using temperature 1.0, top-p 0.95, disabled top-k, min-p 0, seed 42, and a 38,912-token limit.
- A completion is correct only when its final balanced boxed expression is mathematically equivalent to the ground truth; missing boxed expressions count as incorrect.
- Evaluation covers saved checkpoints at steps 25, 50, 75, and 100, reporting each benchmark’s highest Avg@12 and the unweighted mean of the three peaks.
C.1 Controlled Ablation Protocols
Controlled ablations vary which rollouts activate teacher adaptation and which rollout positions receive updates, while preserving the common OPSD token horizon.
- All controlled ablations use Qwen3-8B and the protocols specified for the evaluation and ablation settings.
- Outcome gate: VISTA enables teacher updates only on verifier-accepted rollouts, unlike all-rollout, random, and reverse controls.
- Outcome gate: The random control activates teacher updates with probability 0.6, matching the roughly 60% verifier acceptance rate during VISTA training.
- Position selection: Position controls select the first, last, random, or largest-KL positions, with m = min(k, |y|) positions selected per rollout.
- Position selection: Sparse variants rank candidate positions by raw teacher-first KL scores, while the all-position setting updates every valid position.
- Position selection: Candidate positions are restricted to the first 1,024 rollout tokens, matching the OPSD distillation-loss horizon.
C.2 Teacher-Side Analysis Protocols
Teacher-side analyses track checkpoints, inspect verified student trajectories and reference-conditioned teacher responses, and examine performance throughout training.
- The analysis evaluates the initial Qwen3-8B teacher and checkpoints saved at steps 25, 50, 75, and 100 from the reported training run.
- Valid student trajectories: The valid-trajectory study uses 100 automatically verified and manually reviewed problem-only Qwen3-8B trajectories from randomly sampled training problems.
- Explicit reference attribution: The reference-attribution probe samples one privileged-teacher response per problem and uses an LLM judge to identify explicit references to privileged input.
- Privileged-teacher capability: The privileged-teacher capability probe uses 30 validation problems and 12 privileged-teacher responses per problem.
- Training analysis reports the Qwen3-8B student’s Avg@12 at steps 25, 50, 75, and 100 across all three benchmarks.
D.2 Student Adaptation Scale Under Sparse and All-Position Updates
The analysis compares effective student adaptation under top-16 and all-position teacher updates, then connects reduced all-position adaptation to two-sided teacher–student mismatches.
- The experiment compares Qwen3-8B top-16 and all-position runs at checkpoints 25, 50, 75, and 100 using base-referenced effective LoRA updates.
- The reported statistics include global update scale, mean absolute entry scale, maximum absolute entry scale, and the relative student/teacher scale ratio.
- Student adaptation scale: 15.7%: at step 100, top-16 student D(2) is 1.624 versus 1.404 for all-position, extending its 3.2% step-25 advantage.
- Student adaptation scale: At step 100, top-16 mean and maximum entry scales are respectively 15.2% and 13.7% larger than all-position.
- Student adaptation scale: At step 100, the student-to-teacher scale ratio is 1.42× for top-16 versus 1.31× for all-position, supporting insufficient subsequent student updates after broad teacher adaptation.
- Two-sided teacher–student mismatch: Reference conditioning can over-support reference-dependent continuations or under-support valid problem-only alternatives at the same student-generated prefix.