Source-linked AI summary
Negative Self-Distillation: Learning to Reason by Avoiding Flaws
Rongcan Pei, Zhepei Wei, Shuyao Xu, Xinyu Zhu, Wei-Lin Chen, Yu Meng
TL;DR
OPSD can degrade complex reasoning by distilling artificially confident trajectories from privileged information, motivating a method that avoids flawed reasoning instead of imitating privileged solutions. NSD generates negative conditions from the model itself, gates divergence to reasoning-relevant tokens, and consistently outperforms OPSD and other self-bootstrapping baselines across reasoning benchmarks.
Problem
OPSD uses privileged information to produce artificially confident reasoning trajectories, which can suppress natural reflection and exploratory reasoning on complex tasks.
Method
NSD uses self-generated negative conditions, token-level gating, bounded unlikelihood, and KL regularization to diverge from flawed reasoning while preserving linguistic priors.
Results
NSD consistently outperforms OPSD and other baselines across seven reasoning tasks and 1.7B, 4B, and 8B models, with ΔAvg gains of +2.3%, +7.5%, and +6.0%.
Takeaways & Limitations
NSD provides a label-free, self-bootstrapped alternative that improves reasoning while mitigating overconfidence and preserving reflection capabilities.
Takeaways & Limitations
NSD may be less effective for extremely small or weak models that cannot generate meaningful negative contrasts.
Abstract
from arXiv · showhide
On-Policy Self-Distillation (OPSD) has emerged as a popular paradigm for large language model (LLM) self-improvement, allowing models to act as their own teachers by leveraging privileged information such as ground-truth solutions. However, recent findings indicate that OPSD can severely degrade the performance of LLMs on complex reasoning tasks: By forcing the student to imitate an artificially confident reasoning trace conditioned on privileged information, OPSD inadvertently suppresses expressions of uncertainty and penalizes the exploratory, self-corrective behaviors required to solve challenging problems. To address this, we introduce Negative Self-Distillation (NSD), a new framework that optimizes LLMs by diverging from flawed reasoning rather than imitating privileged solutions. Instead of relying on ground-truth answers or external supervision, NSD uses the model itself to generate a question-specific negative condition (eg, acting as a ``careless reasoner'') and pushes the student's distribution away from this self-generated negative teacher. Naively applying unlearning objectives to achieve this divergence is problematic, as flawed reasoning tokens are confounded with basic linguistic tokens; indiscriminately penalizing both risks catastrophically degrading the model's foundational language capabilities. We resolve this by designing a dynamic gating mechanism that automatically identifies and isolates reasoning-critical tokens, ensuring gradient updates target only behavioral flaws while preserving the model's linguistic priors. Empirically, NSD consistently outperforms OPSD and other label-free, self-bootstrapping reinforcement learning (RL) baselines.
1 INTRODUCTION
NSD replaces privileged-answer imitation with self-generated negative reasoning conditions, aiming to avoid flawed reasoning while preserving language capabilities. Its gated, label-free training consistently outperforms established self-improvement baselines across model sizes and reasoning tasks.
- Motivation and contribution: OPSD can produce artificially confident, highly linear reasoning trajectories because its teacher is conditioned on privileged information such as ground-truth answers.This motivation contrasts privileged-solution imitation with NSD’s self-generated negative supervision.
- Motivation and contribution: NSD uses the model itself to generate question-specific negative conditions and trains the student to diverge from flawed reasoning without ground-truth solutions or external teachers.The negative teacher can represent behaviors such as a careless reasoner, encouraging avoidance of premature conclusions and other flawed patterns.
- Motivation and contribution: NSD dynamically gates token-level divergence so updates target reasoning-critical tokens rather than ordinary linguistic tokens, preserving pretrained language priors.The gate compares negative and benign reference distributions, while bounded unlikelihood further stabilizes optimization.
- Motivation and contribution: NSD consistently outperforms OPSD, Intuitor, and TTRL across seven reasoning tasks and 1.7B, 4B, and 8B model sizes.The reported advantages also include training efficiency, reduced overconfidence, and preserved reflection capabilities.
2 NSD: NEGATIVE SELF-DISTILLATION
NSD generates negative conditions from unlabeled problems, identifies tokens boosted by those conditions, and suppresses them while regularizing benign tokens. Dynamic gating and sigmoid-bounded unlikelihood address token confounding and unstable gradients.
- Negative conditioning: NSD generates a negative condition for each unlabeled problem, constructs a negative teacher from that condition, and penalizes the student’s alignment with it.The default online strategy samples an initial solution and then creates an adaptive negative condition from that reasoning trace.
- Background and challenges: Standard unlikelihood can penalize grammatical tokens and create loss or gradient explosions on highly confident tokens such as punctuation and spaces.These failure modes motivate separate solutions for token identification and bounded penalty design.
- Token-level adaptive gating: The gating mechanism activates penalties only when the negative teacher raises a sampled token’s probability above the benign reference, filtering ordinary linguistic tokens.A larger positive probability gap produces a heavier penalty, while tokens not boosted by the negative condition are exempt.
- Gated unlikelihood penalty: Sigmoid-bounded gated unlikelihood repels negative reasoning patterns while attenuating penalties on high-probability structural tokens and emphasizing low-to-mid confidence tokens.The bounded objective serves as a safeguard against gating imperfections and unstable updates.
- Regularization and overall objective: NSD adds a point-wise forward KL regularizer using a sampled-token estimator instead of computationally heavy full-vocabulary KL divergence.The regularizer helps preserve the student’s fundamental linguistic priors while the gated penalty targets identified vulnerabilities.
3 EXPERIMENTAL SETUP
The evaluation trains Qwen3 models on label-free MATH data and tests them across seven mathematical reasoning benchmarks against distillation and reinforcement-learning baselines.
- Training setup: NSD, Intuitor, and TTRL discard gold labels during training on MATH across Qwen3-1.7B, 4B, and 8B models.All models are trained for two epochs.
- Baselines: The comparison covers OPSD, Intuitor, and TTRL as representative distillation, internal-feedback reinforcement-learning, and consensus-based reinforcement-learning methods.TTRL is applied to the training dataset for fair comparison.
4 EVALUATION RESULTS
Across model sizes and mathematical benchmarks, NSD delivers the strongest average gains, broader generalization, preserved reflection, and improved training efficiency over competing approaches.
- 4.1 MAIN RESULTS: +2.3%, +7.5%, and +6.0% ∆Avg gains make NSD the best-performing method across Qwen3-1.7B, 4B, and 8B models.∆Avg is averaged across seven benchmarks against the same-size base model.
- 4.1 MAIN RESULTS: NSD’s 4B and 8B models reach peak AIME accuracies of 35.8% and 39.6% while generalizing across diverse mathematical tasks.OPSD and reinforcement-learning baselines excel narrowly on AIME 2024.
- 4.2 NSD INSPIRES REFLECTION: NSD yields up to 7.5 reflection tokens per response, whereas OPSD and Intuitor drop to 2.18 and 0.75.The comparison measures average reflection-token frequency on Qwen3-4B across AIME and HMMT.
- 4.3 NEGATIVE CONDITION VARIANTS STUDY: Question-only conditioning achieves a 7.3% average improvement, close to the default online strategy’s 7.8%, while wiki-irr remains competitive.These results show that simpler negative conditions can retain effectiveness.
- 4.4 EFFICIENCY OF NSD: Wiki-irr reduces NSD latency from 68s to 54s, while single rollouts, parallel prefilling, and scalar-only loss computation improve overall efficiency.NSD requires one rollout per sample and trains faster overall than OPSD.
- Supplementary evaluations: Supplementary evaluations report that NSD continues to outperform all baselines under Pass@8 and thinking-mode settings.An alternative policy-gradient objective is also evaluated in the supplementary experiments.
5 UNDERSTANDING NSD TRAINING OBJECTIVE
NSD’s adaptive gating prioritizes task tokens over style tokens and requires KL regularization to prevent training collapse. A bounded unlikelihood objective further attenuates updates on highly predictable tokens.
- 5.1 ADAPTIVE GATING ANALYSIS: The gating function filters trivial tokens while retaining tokens considered essential for the task.Its analysis distinguishes style tokens from task tokens when evaluating token-level logit distributions.
- 5.1 ADAPTIVE GATING ANALYSIS: NSD’s gating mechanism filters style tokens more effectively than entropy-based OPSD weighting and vanilla OPSD loss.A lower style-task ratio indicates stronger prioritization of task tokens.
- 5.2 KL ABLATION: Removing the KL constraint causes mid-training collapse, with oscillating distributions and a sharp gate-activation drop around step 120.The collapse reflects excessive drift from the reference model and coincides with declining gating effectiveness.
- 5.3 DISCUSSION ON GATED UNLIKELIHOOD: Minor probability fluctuations on highly predictable tokens can activate the gate, but their near-zero gate values keep the resulting divergence negligible.This sensitivity mainly affects tokens such as punctuation when negative and reference probabilities are both near one.
- 5.3 DISCUSSION ON GATED UNLIKELIHOOD: The sigmoid-bounded unlikelihood penalty attenuates gradients on high-probability tokens and provides a safeguard against gating imperfections.The paper reports lower LGU values than pure unlikelihood objectives for such tokens.
6 RELATED WORK
Related work spans external-reward on-policy distillation, label-free reinforcement learning, and training with negative signals. These approaches variously replace external supervision with self-generated rewards, labels, or negative trajectories.
- On policy distillation: Self-distillation removes external teachers by using ground-truth solutions, but can suffer from solution bias and overconfidence.The related-work discussion contrasts this setting with original on-policy distillation based on external reward models.
- Label-free reinforcement learning: Label-free reinforcement learning commonly substitutes self-generated confidence or entropy rewards, consensus pseudolabels, or model-generated gold labels.These methods operate across RLVR and on-policy distillation frameworks.
- Training with negative signals: Negative-signal methods use unlikelihood or combine positive and negative trajectories in distillation and reinforcement-learning frameworks.Prior work also explores exclusively negative-signal RLVR and negative signals for reducing student overconfidence.
7 CONCLUSION
NSD combines negative conditioning with gated unlikelihood training and consistently outperforms existing baselines across seven mathematical reasoning benchmarks and multiple model sizes.
- 7 CONCLUSION: NSD consistently outperforms existing baselines across seven mathematical reasoning benchmarks under various model sizes.The framework combines adaptive gating, sigmoid unlikelihood, flexible negative conditioning, and efficient parallelized computation.
- 7 CONCLUSION: NSD’s adaptive gating isolates genuinely flawed tokens, while sigmoid unlikelihood produces smoother reasoning gradients.The framework also avoids full-vocabulary computations and supports parallel negative-teacher and reference-model passes.
- 7 CONCLUSION: NSD is presented as preserving and stimulating self-reflection while enhancing reasoning capabilities through label-free training.The conclusion identifies this capacity as a potential benefit for post-training.
LIMITATIONS
NSD depends on the student’s ability to generate meaningful negative conditions and, under its online strategy, requires additional generation and forward-pass computation.
- LIMITATIONS: NSD may be less effective for extremely small or weak models that cannot generate meaningful negative contrasts.The limitation is tied to the student model’s capacity to produce useful negative conditions.
- LIMITATIONS: The online strategy requires negative-condition generation and two forward passes through frozen models, increasing rollout computation.Generation-free conditioning and parallel execution can mitigate rollout costs and wall-clock latency.
A ANALYSIS OF CANDIDATE GATED UNLIKELIHOOD GRADIENTS IN NSD AND OPSD OBJECTIVES
NSD’s gated unlikelihood suppresses negative-teacher-aligned tokens while attenuating updates on highly probable structural tokens. Compared with standard unlikelihood and OPSD, its sigmoid-based objective concentrates corrective gradients on mid-confidence, reasoning-critical tokens.
- The NSD loss combines a reference-distribution KL term with the gated unlikelihood term LGU.The adaptive gate is G = max(0, πneg − πref), and LGU penalizes tokens vulnerable to the negative condition.
- Figure 7 compares gradient behavior by token probability and shows that OPSD increases gradients with probability, whereas NSD generally decreases them.The comparison indicates that OPSD emphasizes high-probability tokens, while NSD’s candidate objectives provide more stable patterns.
- Standard unlikelihood produces gradients proportional to πc, risking maximal updates on trivial high-confidence tokens such as punctuation and fixed collocations.When the gate is active and πc approaches 1, the optimizer can strongly update tokens that are unlikely to represent reasoning errors.
- The sigmoid-based GU objective attenuates gradients as student-token probability approaches 1, unlike the unbounded logarithmic penalty.Its numerator contains a (1 − πc) factor, causing gradient magnitude to decay toward zero for highly confident tokens.
- NSD’s sigmoid formulation concentrates its largest gradients near πc ≈ 0.6, targeting ambiguous tokens more likely to be reasoning-critical.The same mechanism suppresses gradients on tokens with πc > 0.9, helping preserve structural fluency.
B COMPARISON OF NSD WITH OTHER METHODS
The comparison framework distinguishes methods by sampling, reward source, teacher dependence, gold-label requirements, and monitor-signal quality. NSD is characterized as an on-policy, self-generated, label-free approach with gating-based noise filtering.
- Table 4 compares methods across sampling source, reward signal, teacher dependence, gold-label use, and monitor-signal quality.
- NSD is described as using on-policy negative conditioning without gold labels, while filtering noise through gating.
C EXPERIMENT DETAILS
The experiments standardize hardware, prompting, and training settings while testing several negative-condition generation variants. NSD’s prompts include question-only, solution-aware, online, offline, and irrelevant-context constructions.
- All experiments use eight NVIDIA A100 80GB GPUs on a single node, with official method hyperparameters and controlled comparison adjustments.OPSD is evaluated with and without LoRA, reporting the best-performing variant.
- All methods use the Qwen3 instruct chat template in non-thinking mode, with an empty <think> block generated automatically.The setting applies to student rollouts, teacher log-probability computation, and downstream evaluations.
- Question-only negative conditions generalize the problem’s mathematical class and instruct a plausible cognitive error without using the current problem’s specific numbers or variables.
- Solution-aware prompts ask a helper model to exploit the student’s reasoning steps, introduce a plausible shortcut error, and forbid a correct verification.The generated attack prompt is targeted to the student’s existing solution rather than only the problem statement.
- The wiki-irrariant uses randomly sampled Wikipedia text as spurious context for the teacher, while the student receives the plain problem prompt.
D ADDITIONAL EXPERIMENTAL RESULTS
Additional experiments evaluate pass@8, thinking-mode performance, policy-gradient optimization, and case-study behavior. NSD remains competitive across these settings, while results vary by objective, negative-conditioning strategy, and task difficulty.
- D.1 ADDITIONAL EVALUATION: The main evaluation reports pass@8, defined as whether at least one of eight sampled solutions is correct, with average improvement measured across seven benchmarks.
- D.2 PERFORMANCE ON THINKING MODE: NSD outperforms all other baselines overall in thinking-mode evaluation on Qwen3-4B, with larger gains on challenging AIME and Olympiad Bench datasets.On AMC, where improvement headroom is limited, methods perform comparably to the base model.
- D.3 ALTERNATIVE OBJECTIVE: POLICY GRADIENT OPTIMIZATION: The policy-gradient formulation assigns strongly negative advantages to high-loss tokens and near-zero or positive advantages to low-loss tokens.This reduces the probability of tokens receiving high NSD loss while leaving low-loss tokens largely unchanged.
- D.3 ALTERNATIVE OBJECTIVE: POLICY GRADIENT OPTIMIZATION: NSD trained with the policy-gradient objective improves the 1.7B model by +5.0% on average, while the original J-objective performs better overall at 4B and 8B.Under policy-gradient training, wiki-irr performs best and online solution-aware is second best.
- D.4 CASE STUDY: In a case study, NSD-trained models explore novel correct solutions absent from both base and OPSD outputs and use reflection steps to avoid erroneous trajectories.