PaperlayerView exact revision on arXiv

Source-linked AI summary

Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation

Sarthak Harne, Chinmay Karkar, Yash Pandya, Ahmed Awadallah, Akshay Nambi

arXiv:2608.04794v1cs.AIcs.LG

TL;DR

The paper asks whether self-distillation with privileged information, used as a lone objective, teaches correctness on difficult tasks. It measures the objective across varied reasoning domains and finds loss declines without matching accuracy gains, attributing the gap to a target shaped by a specific solution rather than correctness.

  • Problem

    Reported self-distillation gains come mainly from comparatively easy tasks, leaving open whether the objective alone optimizes correctness on difficult tasks.

    Source: §1 · p. 1

  • Method

    The paper tests reward-free self-distillation, in which a PI-conditioned version of the model supplies per-token targets to a prompt-only student, and measures the resulting loss and task outcomes.

    Source: §3 · p. 3, §6 · p. 6

  • Results

    Across difficult tasks, per-token loss falls while validation accuracy fails to improve and typically degrades, by up to 3.51 points on average in agentic tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The PI Bias Score and token-level measurements indicate that the target favors a particular reference trajectory rather than encoding correctness, leaving the lone objective decoupled from task success.

    Source: §1 · p. 2, §8 · p. 11

    The study is observational, evaluates Qwen3 at two scales across four domains, and makes no cross-architecture claim.

    Source: §8 · p. 12

Abstract

from arXiv · show

Self-distillation (SD) has emerged as a compute-efficient alternative to reinforcement learning with verifiable rewards: a self-teacher, conditioned on privileged information (PI) about the answer such as a reference solution, supplies dense per-token supervision to a student that never sees it. Reported gains, however, come almost exclusively from narrow, low-difficulty settings, leaving open a basic question: as a lone objective, with no reward term, does SD teach anything? We reproduce SDPO's reported gains in its easy setting, then apply the identical setup to difficult tasks and find that it does not. Across question answering, mathematics, coding, and multi-turn agentic tool use, across reasoning modes, model sizes, and forms of PI, and under both the SDPO and OPSD recipes, the per-token loss falls steadily while validation accuracy does not improve and typically degrades. We explain this failure through a single causal chain from the loss to the model it produces. The chain begins with PI bias: having seen one particular reference solution, the teacher's per-token target is pulled toward that trajectory rather than toward correctness in general, an effect we quantify with a PI Bias Score. Trained to match this target everywhere, the student's objective becomes nearly blind to whether a rollout is correct, and the loss it assigns falls mostly on low-information tokens like stopwords, punctuation, uncertainty markers, rather than those that determine the answer; within correct rollouts the exploratory tokens incur the highest divergence, so it penalizes the hesitation that reasoning requires. The result is a flatter, less decisive student that is no better at reasoning: as a lone objective, SD optimizes a signal decoupled from task success.

1 Introduction

The paper asks whether self-distillation (SD), used alone without reward, optimizes correctness on difficult tasks. It reproduces gains in an easy setting but finds that on harder tasks the loss falls without corresponding accuracy gains, tracing the gap to PI bias and the token-level target.

1 Introduction

  • Across four difficult domains, SD loss falls while validation accuracy fails to improve and typically degrades, by up to 3.51 points on average in agentic tasks and 7.0 on individual benchmarks.The authors reproduce the reported SDPO trend in its easy setting and report the difficult-task outcome under both SDPO and OPSD.Source: §1 · p. 2
  • The PI Bias Score is 0.52 for the in-context solution versus ≤0.02 for a different correct solution, indicating that the teacher favors one trajectory over correctness generally.The score is a teacher-minus-student log-probability ratio computed from forward passes.Source: §1 · p. 2
  • Correct and incorrect rollouts have overlapping per-token loss (≈3.5 × 10−4) and KL (≈0.03), while the teacher–student gap is larger on correct rollouts (0.7) than incorrect ones (0.45).The reported gap suggests learning pressure is highest where the student is already right, allowing loss to fall without accuracy following.Source: §1 · p. 2
  • Stopwords, uncertainty markers, punctuation, and whitespace absorb 55.4% of per-token loss, while answer-determining content receives comparatively little.The loss is distributed across tokens that do not decide the answer.Source: §1 · p. 2
  • Within correct rollouts, off-reference positions carry KL ≈0.31 versus ≈0.08 on-path, so the teacher penalizes exploratory reasoning detours.The authors argue that a teacher conditioned on the solution cannot distinguish a productive detour from a mistake.Source: §1 · p. 2
  • The student becomes flatter and commits earlier without solving more problems, completing the measured chain from PI bias through a correctness-blind loss to a less decisive policy.The authors frame SD as an underspecified family of objectives: dense supervision remains, but its target does not encode correctness.Source: §1 · p. 2

2 Background and Related Work

Prior work positions self-distillation between coarse reward-based training and dense distillation, reporting gains mainly on easier tasks. Related critiques identify leakage and degraded reasoning, while leaving SD alone on difficult tasks untested.

2 Background and Related Work

  • RLVR supplies a binary trajectory-level correctness reward, while GRPO estimates group-relative advantages across rollouts; SD instead offers dense per-token supervision without an external teacher.The background contrasts RLVR’s coarse signal and need for verifiers with SD’s PI-conditioned teacher.Source: §2 · pp. 2–3
  • SDPO and OPSD report gains over GRPO on relatively limited tasks, including SciKnowEval, ToolAlpaca, LiveCodeBench, AIME, and HMMT.SDPO uses per-token KL with feedback-conditioned teachers; OPSD uses forward KL with a teacher conditioned on the reference solution.Source: §2 · p. 3
  • RLSD [Yang et al., 2026a] formalizes conditional mutual-information leakage, while Kim et al. [2026] report degraded reasoning under richer teacher conditioning, including declines of up to ∼40%.The paper notes that these works address theory or degraded outcomes but leave their origin open.Source: §2 · p. 3
  • Prior work had not tested SD as a lone objective on difficult tasks; this paper takes up that gap by connecting leakage and uncertainty-suppression accounts in measured runs.Other cited approaches keep verifiable reward primary and use SD as a small-weight auxiliary regularizer.Source: §2 · p. 3

3 The SD Objective and Two Sources of Bias

Self-distillation trains a prompt-only student to match a PI-conditioned version of the same model along the student’s own rollout. The target is dense across tokens yet reflects a particular privileged solution, creating two sources of bias.

3 The SD Objective and Two Sources of Bias

  • The student sees the prompt alone, while the self-teacher is the same model additionally conditioned on privileged information r; the teacher may be the student or a slow copy.The student is trained along a rollout sampled from its own policy to match the teacher’s next-token distribution.Source: §3 · p. 3
  • The objective uses stop-gradient so gradients flow only through the student, and the paper studies it without a reward term.SDPO and OPSD instantiate the divergence D differently.Source: §3 · p. 3
  • The objective applies a target at every position, placing punctuation, stopwords, and discourse markers on equal footing with content-bearing tokens.This density is the design’s advantage over trajectory-level reward, but it also spreads signal across tokens that do not determine content.Source: §3 · pp. 3–4
  • With a whole reference solution as PI, the target reflects one solution’s answer and phrasing, so token-by-token matching spreads signal across its surface form rather than correctness decisions.RLSD [Yang et al., 2026a] characterizes this dependence as an irreducible conditional mutual information term.Source: §3 · p. 4

4 Experimental Setup

The experiments test SD on four harder domains with Qwen3 models, varied PI, and a reward-free objective, while tracking both loss and task outcomes. The setup also reproduces SDPO’s reported easy-setting behavior.

4 Experimental Setup

  • The evaluation covers general QA, mathematics, coding, and multi-turn agentic tool use, with in-domain and held-out transfer benchmarks.Training uses 2,000 examples for QA, math, and coding, plus the BFCL multiturn split.Source: §4 · p. 4
  • Qwen3-8B think and instruct variants cover each domain, with Qwen3-32B added for general QA; whole-solution PI is the default, alongside short hints and structured skills.The whole-solution PI is a randomly selected correct rollout per query.Source: §4 · p. 4
  • Training minimizes the reward-free objective with symmetric Jensen–Shannon divergence, per-token clipping, and a slow EMA teacher; the OPSD variant is also reported.The study tracks task accuracy or success, per-token loss, student entropy, response length, teacher–student KL, and teacher perplexity.Source: §4 · p. 4
  • On SciKnowEval with whole-solution PI, Qwen3-8B reproduces the qualitative SDPO trend: loss decreases as Biology and Physics validation accuracy rises.The experiment uses SDPO’s codebase and reported hyperparameters.Source: §4 · p. 4

5 The Training Signature of Lone SD

Lone self-distillation steadily optimizes its per-token loss without improving validation performance across difficult reasoning tasks, underperforming on reasoning-heavy benchmarks and across both SDPO and OPSD recipes. The section characterizes this training signature and rules out optimization failure and implementation error as explanations.

5 The Training Signature of Lone SD

  • On MMLU-Pro, the top 50% of tokens consume 98.5% of unclipped loss, so the average can fall while little signal reaches content tokens.This skew mirrors OPSD's reported pattern on math.Source: §5 · p. 5

5.1 Patterns observed during training

  • Across four domains, per-token loss falls while validation accuracy does not improve; entropy rises and response length falls, with agentic instruct mode the sole modest-improvement exception.The authors say shorter responses on reasoning tasks are consistent with reduced exploration before committing.Source: §5.1 · p. 5, Figure 2 · p. 6

5.2 End-task results

  • 3.51 points on average are lost by the think variant in agentic tasks, including 7 on BFCL Multi-Turn Missing Function V4; both modes lose 1.6 across general QA.In-domain and transfer results move together, consistent with the objective failing to install a capability rather than a narrow one.Source: §5.2 · p. 5
  • In coding, the instruct gain on MBPP+ (+12.6) is offset by losses on HumanEval+ (−3.8), CodeElo (−1.7), LCBv6 (−1.5), and CodeForces (−0.9), so performance still declines averaged over benchmarks and modes.Positive averages appear only under the instruct variant; its agentic gains are confined to BFCL, trained on 100 tasks with mean 300-token rollouts.Source: §5.2 · p. 5
  • Under OPSD, 18 of 20 benchmark–mode cells are non-positive; average changes are −2.0 and −0.7 on general QA and −4.3 and −0.4 on math for think and instruct.The authors attribute the shared behavior to Eq. (1)'s structure rather than one recipe's implementation choices.Source: §5.2 · p. 5
  • The authors rule out optimization failure and implementation error: loss decreases smoothly, and the same code, teacher, and clipping reproduce reported gains on SciKnowEval.They identify task difficulty as what changes and conclude that the objective itself remains at issue.Source: §5.2 · p. 5

6 A Mechanistic Account: From Privileged Information to a Misguided Student

Using Qwen3-8B with whole-solution privileged information, the analysis measures a proposed chain from PI bias through correctness-blind loss to a flattened student; changing PI specificity tests the chain’s first link.

6.1 PI bias: the teacher targets one solution, not correctness

  • Hints and skills reduce PI bias but weaken the signal: on DAPO-Math, they lose 5.1 and 3.5 points against a base average of 63.4, while whole-solution PI stays within half a point.The authors describe the two ends of the specificity range as failing for opposite reasons: whole-solution PI ties the target to one trajectory, while hints and skills move little mass toward any target.Source: §6.1 · pp. 7–8, Table 3 · p. 9

6.2 The loss does not track correctness

  • Correct and incorrect rollouts have overlapping per-token loss and KL, while the teacher–student gap is higher on correct rollouts.The gap settles near 0.7 on correct rollouts versus 0.45 on incorrect ones, so the strongest signal falls where the student already succeeds.Source: §6.2 · pp. 8–9, Figure 4 · p. 9

6.1 PI bias: the teacher targets one solution, not correctness

  • Across domains, the PI Bias Score is much higher for the in-context solution than for a different correct solution, which scores like an unrelated one.The contrast indicates that the teacher favors the particular trajectory it saw rather than correctness in general.Source: §6.1 · pp. 6–7, Figure 3 · p. 8

6.3 The loss lands on uninformative tokens

  • Stopwords, uncertainty markers, punctuation, and whitespace absorb 55.38% of per-token loss, while answer-determining content words, numbers, and math symbols absorb comparatively little.Uncertainty markers have the highest average loss per token, followed by stopwords and content words; numbers and special tokens have the lowest.Source: §6.3 · p. 10, Figure 5 · p. 10

6.4 The objective penalizes exploration

  • Within correct rollouts, off-path exploratory tokens incur about four times the KL of on-path tokens, so the objective penalizes exploration even when the answer is correct.The reported KL is approximately 0.31 off-path versus 0.08 on-path.Source: Figure 4 · p. 9, §6.4 · p. 10

6.5 The student flattens rather than sharpens

  • The teacher–student mean log-probability gap approaches zero while its variance persists, indicating cancellation rather than convergence; the student ends flatter and less decisive.Student entropy rises in most domains, and the student commits earlier without solving more problems.Source: §6.5 · pp. 10–11

7 Robustness

The robustness tests find no improvement when varying response length, task difficulty, or model scale; the reported outcomes remain at or below the respective base performance.

Response length

  • −1.1, −0.8, and −1.6 points versus the base average of 63.4 for short, medium, and long DAPO-Math splits; in-domain scores rise +1.4, +1.0, and +0.6.Eight of nine transfer cells fall, and the short split is no better than the long one.Source: §7 · p. 11

Task difficulty

  • All MMLU-Pro difficulty splits fall below the base average of 60.9: easy by 3.6 points, medium by 1.4, and hard by 1.6.Difficulty groups are defined by the base model’s mean success rate at eight samples.Source: §7 · p. 11

Model scale

  • For Qwen3-32B on MMLU-Pro, averages move −0.6 points in think mode and −1.4 in instruct mode; the reported failure signature remains unchanged.The 32B model’s base average is 64.7, compared with 60.9 for Qwen3-8B on the same benchmarks.Source: §7 · p. 11

8 Discussion

The discussion attributes self-distillation’s failure on difficult tasks to a PI-conditioned target that follows one reference trajectory rather than correctness. It contrasts this with settings where imitation may approximate correctness and outlines practical checks, related reward-based methods, and study boundaries.

8 Discussion

  • The failure lies in the target: a PI-conditioned teacher encodes one reference trajectory rather than distinguishing correct continuations from well-phrased incorrect ones.Because the dense loss covers every token, density cannot repair a target that lacks correctness information.Source: §8 · p. 11
  • Imitation can approximate correctness on easy tasks with short responses, bounded answers, little exploration, and few tokens irrelevant to correctness.These conditions weaken as responses lengthen and answers open up.Source: §8 · pp. 11–12
  • A falling SD loss is not evidence of learning; the authors recommend monitoring validation accuracy and using the PI Bias Score to screen candidate PI.The score uses forward passes over a handful of prefixes and four targets, at negligible cost beside training.Source: §8 · p. 12
  • Both ends of the PI-specificity range fail: a large PI Bias Score signals trajectory transfer, while a uniformly small score signals no transfer.The paper argues there is no benign middle setting for PI specificity.Source: §8 · p. 12
  • Reward-based augmentation supplies correctness while SD supplies form; whether combining them repairs the documented failure remains untested here.The authors identify dense targets reflecting a set of correct continuations as an open direction.Source: §8 · p. 12
  • The study is observational, covers Qwen3 at two scales across four domains, and reports some domain differences small relative to run-to-run variation.The authors make no cross-architecture claim and note that hints and skills are model-generated.Source: §8 · p. 12

9 Conclusion

The conclusion finds that a PI-conditioned per-token objective alone does not align its training signal with task correctness in the studied settings. It links this result to teachers anchored to one reference trajectory and the resulting flatter student.

9 Conclusion

  • Across four reasoning-heavy domains, loss decreases while accuracy stays flat or degrades under both recipes and reasoning modes, across trajectory length, task difficulty, and model scale.The study does not claim self-distillation is ineffective in general; its conclusion concerns the PI-conditioned per-token objective used alone in these settings.Source: §9 · p. 12
  • The PI Bias Score indicates that a teacher conditioned on one reference solution learns that trajectory rather than correctness.The resulting loss cannot distinguish correct from incorrect rollouts and concentrates on already-correct trajectories, low-information tokens, and off-path exploration.Source: §9 · p. 12
  • Weakening PI also fails, so neither overly specific nor overly weak PI signals correctness.The conclusion identifies the target—not its density—as the limitation.Source: §9 · p. 12
  • The resulting student is flatter rather than sharper and is no better at reasoning.The paper locates the limitation in a teacher anchored to one reference trajectory rather than the task objective.Source: §9 · p. 12

A Datasets, Splits, and PI Construction

The study constructs training, in-domain evaluation, and transfer datasets across four domains, then supplies one of three forms of privileged information to the teacher. The forms range from a selected correct solution to answer-free hints and general skill cards.

A Datasets, Splits, and PI Construction

  • The study spans general question answering, mathematics, coding, and multi-turn agentic tool use, with in-domain training and evaluation plus held-out transfer datasets.MMLU-Pro, DAPO-Math, CodeForces, and Berkeley Function-Calling Leaderboard provide the respective in-domain datasets.Source: A Datasets, Splits, and PI Construction · p. 17, §A.1 · p. 17

A.1 Sources and splits

  • In-domain training generally uses a random 2,000-example subset and a stratified 300-example validation subset; the agentic domain uses its full training set.Transfer validation uses each transfer dataset in full.Source: §A.1 · p. 17

A.2 PI construction

  • The three PI forms are a selected correct solution, a short answer-free hint, and a cluster-level skill card that gives a reusable technique without a specific answer.Figure 6 illustrates the three forms on one mathematics question; each is supplied separately to the teacher.Source: §A.2 · pp. 17–18, Figure 6 · p. 18

B Hyperparameters (Implementation Details)

The appendix documents the primary training configuration and the prompts and templates used for problems, self-distillation, and privileged-information generation. It also illustrates how whole solutions, hints, and skill cards are constructed and presented.

B Hyperparameters (Implementation Details)

  • Primary runs use Qwen3-8B thinking and instruct variants across domains, with coding and agentic deviations and a separate Qwen3-32B size study.Runs use one node with eight NVIDIA B200 GPUs; the mixing coefficient is 0.5 and the teacher EMA rate is 0.001.Source: B Hyperparameters (Implementation… · p. 18

C Qualitative Examples (Full Solution vs Hints vs Skills)

  • The examples contrast a correct solution without its reasoning trace, a brief insight naming the governing idea, and a structured skill card that does not disclose a problem’s answer.The skill card describes a general technique for the problem cluster.Source: C Qualitative Examples (Full Solution… · pp. 18–20

D.1 Per-domain problem prompts

  • Domain-specific problem prompts wrap each problem and specify response formats, including answer choices, step-by-step reasoning, or a complete program.The coding prompt requests a Python program that matches the specification and passes tests.Source: D Prompts and Templates · p. 20, §D.1 · p. 20

D.2 Self-distillation templates

  • Self-distillation templates combine the problem, a solution section carrying PI, and optional feedback, then ask the student to solve using its own approach.The mathematics and multi-turn overrides label the injected PI as a correct solution.Source: §D.2 · pp. 21–22

D.3 Privileged-information generation prompts

  • A separate model generates compressed solutions and hints from correct solutions; hints state the core insight without giving the answer or a numerical result.The compressed-solution prompt requests key steps and the final answer in a concise, self-contained form.Source: §D.3 · pp. 22–23

D.4 Skill derivation prompt

  • The skill form is derived once per problem cluster as a reusable technique card, positioned between a one-line hint and a full solution.Its structured fields cover the general approach, tools, pitfalls, and cues for when the skill applies.Source: §D.4 · p. 23

E PI Bias Implementation Details

The PI Bias Score compares how strongly teacher and student prompts support a target continuation after the student’s response prefix. The implementation evaluates four target types at assistant-token positions and scores each target in full.

E PI Bias Implementation Details

  • The PI Bias Score is the teacher on-path score minus the student on-path score; positive values mean privileged information raises the tendency to continue toward the target.The teacher prompt contains PI, while the student prompt does not.Source: E PI Bias Implementation Details · p. 24
  • Four targets are scored: the in-context correct solution, a different correct solution, an incorrect solution, and an unrelated correct solution from another question.For hint-style PI without a full solution, the correct-solution target uses a reasoning trace removed.Source: E PI Bias Implementation Details · p. 24
  • Each target is scored in full with a 2000-token lookahead at every assistant-token position; prompts over 8000 tokens are skipped.A closing marker is inserted when the student is inside a reasoning block.Source: E PI Bias Implementation Details · p. 24
  • The on-path score averages target-token log-probabilities when the target is appended after the student prefix, using teacher or student prompt context.The teacher and student scores use their respective checkpoints, or the base model for both before the first checkpoint; the passes are computed separately and merged by position.Source: E PI Bias Implementation Details · p. 24

Related papers

Loading 2608.04794v1…