Source-linked AI summary
OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning
Yuxiao Yang, Xiaoyun Wang, Weitong Zhang
TL;DR
On-policy self-distillation can become unstable when privileged teacher responses mismatch student reasoning patterns. OGLS-SD contrasts teacher logits from correct and failed on-policy rollouts to calibrate token-level guidance, improving performance and stability over standard OPSD on mathematical reasoning benchmarks.
Problem
Privileged self-distillation can mismatch student responses because the teacher uses information unavailable to the student, potentially destabilizing training and harming reasoning.
Source: §1 · pp. 1–2, §3.1 · p. 4Method
OGLS-SD averages privileged teacher logits over correct and failed on-policy rollouts, contrasts them, and uses the resulting steered logits for self-distillation.
Source: §1 · p. 2Results
Takeaways & Limitations
Outcome-guided calibration of privileged teacher signals supports more effective language-model self-improvement in the evaluated reasoning setting.
Source: §1 · p. 2, §6 · p. 8A more comprehensive assessment would evaluate OGLS-SD on larger models, different model families, and broader reasoning tasks.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
We study on-policy self-distillation (OPSD), where a language model improves its reasoning ability by distilling privileged teacher distributions along its own on-policy trajectories. Despite its promise, OPSD can suffer from training instability due to a pattern mismatch between teacher and student responses. Self-reflected teacher responses may introduce reflection-induced biases and response templates that miscalibrate token-level supervision, ultimately harming the student's reasoning ability. To mitigate this issue, we propose OGLS-SD, an outcome-guided logit-steering framework that leverages verifiable outcome rewards to calibrate privileged teacher logits. Specifically, OGLS-SD contrasts teacher logits induced by successful and failed on-policy trajectories, constructing an outcome-discriminative steering direction for token-level guidance. Experiments on mathematical reasoning benchmarks show that OGLS-SD stabilizes self-distillation and improves performance over standard OPSD and other variants.
1 Introduction
On-policy self-distillation offers dense guidance from a model’s own trajectories, but privileged teacher signals can mismatch student responses and destabilize training. OGLS-SD contrasts guidance from correct and incorrect rollouts to calibrate teacher logits; the paper reports improved stability and performance over standard OPSD and naive RLVR integrations.
1 Introduction
- RLVR commonly applies sparse sequence-level outcome rewards, whereas OPD uses student on-policy generations to construct token-level guidance.OPD is presented as a promising alternative for denser supervision.Source: §1 · p. 1
- Privileged self-reflection can concentrate supervision on one solution pattern and shift reasoning styles, potentially suppressing self-checking and harming student reasoning.Multiple valid reasoning paths may exist for the same problem.Source: §1 · pp. 1–2
- OGLS-SD verifies groups of on-policy rollouts, averages teacher logits by outcome, and contrasts the pools to steer token-level guidance.The steered logits are added to anchor teacher logits to form the teacher distribution.Source: §1 · p. 2, Figure 1 · p. 2
- OGLS-SD improves stability and performance over standard OPSD and naive integrations of RLVR objectives.The authors attribute the improvement to outcome-guided calibration of privileged teacher signals.Source: §1 · p. 2
2 Related Works
Related work spans on-policy distillation, self-distillation with privileged information, and steering methods. The paper positions OGLS-SD as calibrating the dense teacher signal itself to address privileged-information mismatch.
2.1 On-policy Distillation
- OPD samples student-policy rollouts and applies teacher logits along those trajectories, providing dense guidance on states the student visits.OPD can reduce reliance on or partially replace RLVR-based post-training (Yang et al., 2025; Team et al., 2026).Source: §2.1 · pp. 2–3
2.2 LLM Self-Distillation
- OPSD uses the model itself as teacher with privileged information, while reported instability has motivated RLVR hybrids that leave the dense teacher target largely unchanged (Kim et al., 2026; Yang et al., 2026a; Li et al., 2026a).The paper instead calibrates token-level teacher signals using correct and incorrect on-policy rollouts.Source: §2.2 · p. 3
2.3 Steering LLM Generation
- Although steering is widely studied for generation control, its use as dense training-time guidance for on-policy distillation remains underexplored; EX-OPD addresses standard OPD, whereas this work studies OPSD with privileged-information mismatch (Yang et al., 2026b).OGLS-SD uses steering both to amplify the teacher signal and to mitigate that mismatch.Source: §2.3 · pp. 3–4
3 Preliminaries
The preliminaries define student rollouts, outcome labels, and privileged information, then describe OPSD and contrastive steering. They frame the central mismatch as direct matching to teacher information unavailable to the student at test time.
3 Preliminaries
- The setup distinguishes correct and incorrect student rollouts using verifiable rewards and treats privileged information as teacher-only training context.Examples include an expert solution or reference reasoning trace unavailable to the student at test time.Source: §3 · p. 4
3.1 On-Policy Self-Distillation
- In OPSD, a teacher conditioned on privileged information supplies token-level distribution-matching guidance along student-generated on-policy responses.The objective uses a stop-gradient teacher target and a token-level matching loss such as KL divergence.Source: §3.1 · p. 4
- Directly matching a privileged teacher creates mismatch because the student cannot access the privileged solution; the ideal non-privileged target is the marginal teacher over such information.Practical estimates from few privileged traces can retain sharper distributions, reduced exploratory-token support, or shifted reasoning styles.Source: §3.1 · p. 4
3.2 Guidance in Large Language Models
- Contrastive steering modifies a base next-token distribution using positive and negative conditions; OGLS-SD instantiates the direction with logits from verified correct and incorrect rollouts.The steered distribution is represented through logits before softmax.Source: §3.2 · pp. 4–5
4 Method
OGLS-SD addresses OPSD pattern mismatch by building teacher guidance from verified positive and negative on-policy contexts and steering logits by their contrast. It also limits steering on already-correct rollouts and regularizes short verified-positive ones.
4.1 Motivation
- Single reference paths can bias OPSD toward one solution pattern, while reflection-specific templates may suppress useful original reasoning behaviors.Source: §4.1 · p. 5
4.2 Outcome-Labeled Rollouts
- OGLS-SD samples policy rollouts, verifies them with dataset labels or rule-based verifiers, then partitions them into correct and incorrect contexts.Source: §4.2 · p. 5
4.3 Maintaining Mixed Guidance Logits
- It aggregates teacher logits over correct contexts and, when available, the reference solution, alongside incorrect contexts, reducing single-solution bias.Source: §4.3 · p. 5
4.4 Outcome-Guided Logit Steering
- The positive–negative logit difference supplies outcome-discriminative steering, intended to suppress shared reflection artifacts while preserving the base on-policy response distribution.Source: §4.4 · p. 6
- Steering loss applies only to incorrect rollouts, avoiding unnecessary full-vocabulary changes to already-correct behavior; it is zero when no incorrect rollout exists.Source: §4.4 · p. 6
- To curb length drift from steering incorrect responses, short verified-positive rollouts receive last-K-token SFT regularization, anchoring correct trajectories and final-answer format.Source: §4.4 · p. 6
5 Experiments
Across five mathematical reasoning benchmarks, OGLS-SD outperforms baselines on most benchmarks. The experiments compare training and reasoning behaviors and test which components contribute to its performance and stability.
5.1 Setup
- Experiments train Qwen3-1.7B and Qwen3-4B on OpenThought, using eight rollouts per question and forward KL distillation on the first 1,024 rollout tokens.Evaluation covers AIME 2024/2025, AMC23, HMMT, and MATH500; models use LoRA with the same backbone and training configuration.Source: §5.1 · p. 6
5.3 Main Results
- OGLS-SD consistently outperforms baselines on most benchmarks; Table 1 reports Avg@8 for Qwen3-1.7B/4B across five reasoning benchmarks.Source: Table 1 · p. 7, §5.3 · p. 7
5.2 Baselines
- Baselines include SFT, GRPO, and OPSD; Qwen3-1.7B also uses an outcome-routed hybrid of OPSD on failed rollouts and GRPO on successful ones.Source: §5.2 · p. 7
5.3 Main Results
- Figure 4 reports pairwise percentages preferring OGLS-SD over OPSD across reasoning behaviors, with 50% marking the preference baseline.Source: Figure 4 · p. 7
- On AIME24, OGLS-SD improves more steadily than OPSD, which peaks early and degrades; OPSD also remains lower on seven epistemic reflection markers.Figure 6 reports marker frequencies for Qwen3-1.7B and Qwen3-4B.Source: §5.3 · p. 7, Figure 5 · p. 8, Figure 6 · p. 8
5.4 Ablation Study
- Positive-mean guidance improves slightly over OPSD but remains below OGLS-SD, while positive-only steering peaks early and then degrades.The positive-mean variant averages positive teacher distributions; positive-only steering amplifies positive guidance without using negative rollouts.Source: §5.4 · p. 8
- With G=4, performance decreases slightly but still exceeds the other ablations; the authors identify explicit positive-negative contrast as important for performance and stability.Source: §5.4 · p. 8
6 Conclusion
OGLS-SD calibrates privileged teacher signals by contrasting successful and failed on-policy rollouts. On mathematical reasoning benchmarks, it improves over standard OPSD and produces more stable self-distillation.
6 Conclusion
- OGLS-SD averages teacher logits over successful and failed rollout pools, then contrasts them to reduce solution-specific bias and shared privileged-pattern shifts.This replaces direct distillation from a single privileged teacher distribution.Source: §6 · p. 8
- Experiments on mathematical reasoning benchmarks show that OGLS-SD improves over standard OPSD and leads to more stable self-distillation.Source: §6 · p. 8
Limitations
The evaluation and design study leave boundaries on how broadly OGLS-SD has been assessed. The authors also note risks inherited from the base model and training data.
Limitations
- A broader assessment would require evaluating OGLS-SD on larger models, different model families, and broader reasoning tasks.Source: Limitations · p. 9
- The study follows the original OPSD setup, leaving alternative divergence objectives, teacher updates, and steering schedules for future work.The unstudied choices include EMA or synchronized teacher updates and different steering-coefficient schedules.Source: Limitations · p. 9
- Although the method targets mathematical reasoning rather than sensitive user-facing applications, stronger models may be misused and inherit base-model and training-data risks.Source: Limitations · p. 9
A.1 Case Analysis
In a case comparison, OGLS-SD avoids an incorrect Alice/Bob role mapping that OPSD makes without sufficient reflection.
A.1 Case Analysis
- OGLS-SD revisits the Alice/Bob role assignment and checks the losing-position implications, avoiding the mistake made by OPSD.Both methods identify the relevant losing-position structure, but OPSD draws the wrong conclusion from an incorrect mapping.Source: §A.1 · p. 12
A.2 Diagnostic Analysis: Pattern Drift in Privileged Guidance
The diagnostic compares direct positive guidance with positive-negative contrastive guidance on rollout tokens. Direct guidance can reduce support for deliberative tokens, while contrastive guidance preserves or increases it.
A.2 Diagnostic Analysis: Pattern Drift in Privileged Guidance
- Both teacher distributions come from the same reference model, isolating the effect of teacher construction; support changes are measured against its unconditioned token log-probability.Positive changes indicate increased support for a rollout token, and negative changes indicate reduced support.Source: §A.2 · p. 12
- The comparison uses direct conditioning on a correct guidance trace versus a positive-negative contrastive construction using correct and incorrect traces.Source: §A.2 · p. 12
- Direct positive guidance can reduce support for deliberative tokens such as “wait” and “hold on,” while contrastive guidance preserves or increases their support.The authors attribute this contrast to subtracting shared response-pattern shifts induced by privileged conditioning; most deliberative expressions have negative support changes under direct guidance but near-neutral or positive changes under contrastive guidance.Source: §A.2 · p. 12
- The diagnostic compares teacher constructions on the same rollout tokens using teacher forcing, with a Qwen3-1.7B reference model on AIME 2024.It selects one correct and one incorrect rollout from eight samples per problem and sets λ = 1 for the contrastive construction.Source: §A.2 · p. 12
A.3 Effect of Positive-Tail Regularization
On Qwen3-1.7B with AIME24, the ablation examines positive-tail regularization using Avg@8 and answer-format rate. The regularizer stabilizes answer formatting and later training, while the outcome-guided distillation objective provides the primary learning signal.
A.3 Effect of Positive-Tail Regularization
- Removing the regularizer yields faster early performance gains, suggesting the outcome-guided distillation objective supplies the primary learning signal.The ablation reports Avg@8 and answer-format rate on AIME24 using Qwen3-1.7B.Source: §A.3 · p. 12
- Positive-tail SFT trades some early-stage aggressiveness for improved later-stage stability.Source: §A.3 · p. 13
- Without positive-tail regularization, answer-format rate falls over training, with length drift and reduced final-answer completion; the regularizer supports more stable final performance.The effect is framed as stabilization rather than the primary source of performance gains.Source: §A.3 · p. 13
B.1 Implementation Details
The implementation fixes the teacher to the original base model and uses concise verified completions for positive-tail regularization. The appendix also describes the behavioral judging protocol and the reflection keywords used for measurement.
B.1 Implementation Details
- Training uses a fixed original-base-model teacher; positive SFT applies to the last 128 tokens of verified correct rollouts shorter than 4,096 tokens.The SFT term is intended to anchor concise correct completions and mitigate length drift, not provide full-trajectory supervision.Source: §B.1 · p. 13
- The case study contrasts OPSD's incorrect Alice/Bob role mapping with OGLS-SD's reassignment check, which preserves the correct answer.Source: Figure 7 · p. 14
- OGLS-SD adds training-time computation for teacher-logit polling over positive and negative guidance pools, without changing inference cost.Long-context evaluation is also a major computational cost, with a maximum generation budget of 38k tokens.Source: §B.1 · p. 14
B.2 LLM Judge Details
- For AIME24 behavioral annotation, the judge compares eight rollouts per model through anonymized pairwise comparisons across five reasoning dimensions.Undesirable behaviors are reverse-scored so larger values consistently indicate more desirable behavior.Source: §B.2 · pp. 14–15
B.3 Prompt Details
- The appendix lists keywords for measuring explicit self-reflection and describes a pairwise judge prompt covering self-reflection, revision, confidence, premature commitment, and pattern mismatch.Source: §B.3 · p. 15