PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Distillation with Verbal Regularization

Jiazheng Li, Yawei Li, Junru Lu, Zhenyi Shen, Di Yin, Lin Gui, Xing Sun, David Rügamer, Xinpeng Wang, Yulan He

arXiv:2610.16354v1cs.CL

TL;DR

Privileged information can bias OPSD supervision toward shortcut reasoning and reduced entropy. VRSD uses teacher-only prompts to regulate that influence, and improves Pass@1 over OPSD in 11 of 12 model-task settings, by 3.9 percentage points on average.

  • Problem

    Privileged information can reinforce shortcut continuations and suppress reconsideration, concentrating teacher distributions and potentially limiting exploration of alternative reasoning paths.

    Source: §1 · p. 1, §4.1 · p. 4

  • Method

    VRSD uses teacher-only system prompts to encourage verification and reconsideration, modeling their regulation of privileged-information effects as affine attenuation of log-probability shifts.

    Source: §1 · p. 2

  • Results

    VRSD improves Pass@1 over OPSD in 11 of 12 model-task settings, with an average Pass@1 gain of approximately 3.9 percentage points.

    Source: §1 · p. 2, §5.2 · p. 7

  • Takeaways & Limitations

    The findings identify teacher conditioning as a practical means of regulating privileged supervision while retaining solution guidance and encouraging verification and reconsideration.

    Source: §6 · p. 10

    VRSD relies on manually curated, task-specific prompts that may not suit every problem or remain equally useful as student capabilities evolve during training.

    Source: §6 · p. 10

Abstract

from arXiv · show

On-policy self-distillation (OPSD) trains a large language model using supervision from a teacher instantiated from the same model as the student. The teacher is conditioned on privileged information (PI), such as verified solutions. The student, which does not observe PI, learns from the teacher's next-token distributions along its own sampled responses. However, PI can bias supervision toward shortcut reasoning with limited verification or reconsideration, accompanied by a sharp drop in the entropy of the output distribution. To address this challenge, we introduce self-distillation with verbal regularization (VRSD), which uses system prompts to regulate how the teacher uses PI, thereby encouraging rethinking and exploration to counteract premature commitment. Theoretically, we model prompting as attenuating the influence of PI on the teacher distribution and derive conditions for entropy preservation. Across three models and four tasks, VRSD improves Pass@1 over OPSD in 11 of 12 model-task settings, by 3.9 percentage points on average, while generally maintaining higher token entropy.

1 INTRODUCTION

VRSD uses teacher-only prompts to regulate how privileged information shapes OPSD supervision, addressing shortcut reasoning and reduced entropy. It improves Pass@1 over OPSD in 11 of 12 model-task settings.

1 INTRODUCTION

  • PI can reinforce shortcut continuations and suppress reconsideration, concentrating the teacher distribution and potentially limiting the student’s exploration of alternative reasoning paths.The challenge is to benefit from PI guidance while limiting premature commitment.Source: §1 · pp. 1–2
  • VRSD changes only the self-teacher’s conditioning, using system prompts to encourage verification, consideration of alternatives, and reconsideration before committing to an answer.The prompts regulate how strongly PI shapes token-level supervision without changing the architecture, training objective, or PI itself.Source: §1 · p. 2
  • VRSD models prompting as affine attenuation of PI-induced log-probability shifts and derives conditions for prompting to counteract teacher concentration and preserve entropy.Figure 1 illustrates PI favoring immediate commitment while the prompt restores probability to reconsideration; under the affine model, entropy preservation holds under mild conditions.Source: §1 · p. 2
  • VRSD improves Pass@1 over OPSD in 11 of 12 model-task settings and improves average Pass@1 and Worst@16 for all three models.Analyses also indicate less output-distribution over-concentration and greater reliance on reasoning chains for final answers than vanilla OPSD.Source: §1 · p. 2

2 RELATED WORK

Related work describes distillation approaches that train on student-generated responses and OPSD methods that use privileged information. VRSD retains complete reference solutions but regulates their influence through teacher-only prompts.

2 RELATED WORK

  • On-policy distillation matches teacher next-token distributions on the student’s own responses, reducing the distribution mismatch of fixed-response distillation and providing denser supervision than GRPO-like methods.Multi-teacher OPD has also emerged as a way to consolidate complementary teacher capabilities.Source: §2 · pp. 2–3
  • OPSD uses varied privileged contexts, while related approaches filter shortcuts, discourage matching the privileged teacher, replace solutions with structured guidance, or adapt how much reasoning is revealed.The cited approaches include Purified OPSD, AntiSD (Shen et al., 2026a), PS-OPSD (Zhao et al., 2026b), and ATESD (Han et al., 2026).Source: §2 · p. 3
  • VRSD differs by retaining complete reference solutions and using teacher-only system prompts to regulate their influence on the self-teacher.OPCD instead distills system-prompt-specified behaviors without instance-specific reference solutions (Ye et al., 2026).Source: §2 · p. 3

3 BACKGROUND OF ON-POLICY SELF-DISTILLATION

OPSD uses one model in two roles: the student samples a response without privileged information, while the self-teacher produces distributions conditioned on that information at the same prefixes. The student is trained toward those teacher distributions using generalized Jensen-Shannon divergence.

3 BACKGROUND OF ON-POLICY SELF-DISTILLATION

  • An LLM assigns response probabilities through a factorization over successive token predictions.The supplied passage introduces the response-probability factorization.Source: §3 · p. 3
  • In OPSD, the student samples a response without privileged information, and the same model produces teacher distributions with that information at the student’s prefixes.The student and self-teacher share model parameters and prefixes but differ in access to privileged information.Source: §3 · p. 3
  • At each prefix, OPSD transfers the privileged-information-conditioned teacher distribution toward the student using normalized generalized Jensen-Shannon divergence.The method defines a convex mixture of student and teacher distributions; the stop-gradient operator keeps the teacher distribution fixed during differentiation.Source: §3 · p. 3

4 METHODOLOGY

Section 4 models how teacher-only prompting attenuates privileged-information (PI) effects and analyzes when that attenuation preserves or increases teacher entropy. It connects PI-driven premature commitment to reduced student exploration and derives a covariance condition for entropy increase.

4.1 REGULARIZING PREMATURE TEACHER COMMITMENT

  • PI can favor immediate commitment while suppressing reconsideration, concentrating the teacher distribution; matching that distribution can make alternatives less likely in later student rollouts.The distillation gradient gives larger coefficients to tokens with larger positive PMI scores, connecting PI-favored continuations to student preference transfer.Source: §4.1 · p. 4
  • VRSD uses training-only teacher prompts to encourage verification and alternative reasoning; the student still generates without PI or the prompt, and the JSD objective is unchanged.The prompt adds no learned module or regularization loss and is absent from the student’s inference-time input.Source: §4.1 · p. 4
  • The affine model represents prompted PMI as a scaled version of the original PMI, with γt controlling token-preference scaling and κt ensuring normalization.The estimated attenuation factor γt is below 1, and the reported confidence intervals indicate statistically significant attenuation.Source: §4.1 · p. 4
  • Figure 2’s confidence intervals fall below the no-attenuation baseline, indicating that teacher prompts attenuate PI-induced log-probability shifts on average.The estimates average per-prefix values within problems and then across problems; the intervals are 95% bootstrap confidence intervals.Source: §4.1 · p. 4, Figure 2 · p. 5

4.2 WHEN PMI ATTENUATION INCREASES TEACHER ENTROPY

  • When the bare teacher’s PMI scores and log-probabilities have positive covariance, sufficiently mild attenuation increases teacher entropy under the affine model.Theorem 4.1 expresses the entropy change along the attenuation path through covariance; Corollary 4.2 gives the local sufficient condition when γt is close to 1.Source: §4.2 · p. 5
  • At prefixes with substantial PMI scores, mean covariance ranges from 0.43 to 0.58, and is positive at 79%–91% of prefixes.These measurements support the covariance condition for entropy increase at the examined prefixes.Source: §4.2 · p. 5

5 EXPERIMENTS

Across four automatically verified tasks, the experiments assess VRSD’s performance, student entropy, prompt design, learned reasoning patterns, and dependence of answers on reasoning.

5.1 EXPERIMENTAL SETUP

  • Experiments use four automatically verified tasks and verified-solution PI; single prompts share a composite routine, while multi prompts distribute aspect-specific guidance across rollouts.The multi-prompt rollouts are split as evenly as possible among aspect-specific prompts.Source: §5.1 · p. 6

5.2 TASK PERFORMANCE

  • VRSD (multi) improves average Pass@1 over OPSD by approximately 3.9 percentage points and wins in 11 of 12 model-task settings.On Qwen3-4B SQL, OPSD Pass@1 is 69.8% versus 71.3% for the untrained model, while VRSD (multi) reaches 81.0%.Source: Table 1 · p. 7, §5.2 · p. 7

5.3 STUDENT ENTROPY

  • In 19 of 24 comparisons, VRSD has higher student next-token entropy than OPSD; on Qwen3-4B SQL, H(p) is 0.03 for OPSD and 0.54 for VRSD (multi).For that SQL setting, VRSD (multi) also raises Pass@1 from 69.8% to 81.0%.Source: §5.3 · p. 7, Figure 3 · p. 8

5.4 ABLATING THE VRSD (MULTI) DESIGN

  • VRSD (multi) beats concatenation and each individual prompt on all four tasks, exceeding the strongest dk by 0.5 to 4.7 percentage points.VRSD (single) also exceeds concatenation and OPSD on all four tasks; multi further improves grading and SQL, with similar results on chemistry and Python.Source: Figure 4 · p. 8, §5.4 · p. 8

5.5 LEARNING THE REASONING PATTERNS ENCOURAGED BY TEACHER PROMPTS

  • Both VRSD designs increase every reported reasoning pattern on average relative to OPSD, with increases in most comparisons.On chemistry, the reflective prompt raises conclusion checking by 73.6 percentage points over OPSD, versus 5.7 for multi; multi more strongly increases discussing and ruling out options.Source: §5.5 · pp. 8–9, Table 2 · p. 9

5.6 EVALUATING REASONING FAITHFULNESS THROUGH CAUSAL INTERVENTIONS

  • Across tasks, RG averages 5.8 percentage points for VRSD (single) and 5.0 for VRSD (multi), versus 3.7 for OPSD; MR is lower in seven of eight comparisons.Retaining the full reasoning improves accuracy for both VRSD variants on all four tasks.Source: §5.6 · pp. 9–10, Table 3 · p. 9

6 CONCLUSION

The paper concludes that teacher conditioning can regulate privileged supervision while preserving solution guidance and encouraging verification and reconsideration.

6 CONCLUSION

  • VRSD regulates how the self-teacher uses privileged solutions, and the analysis links prompting to attenuated solution-induced log-probability shifts and conditions that counter teacher concentration.The experiments report improved accuracy and answer consistency, generally higher student entropy, and evidence of improved reasoning faithfulness.Source: §6 · p. 10
  • The approach uses the model’s existing instruction-following capability to preserve alternatives to premature commitment and improve its self-provided learning signal.Source: §6 · p. 10
  • The current approach relies on manually curated, task-specific prompts that may not suit every problem or remain useful as student capabilities evolve.The authors propose automating prompt selection and adaptation, and directly estimating and adjusting PI-induced log-probability shifts as future directions.Source: §6 · p. 10

AI USE STATEMENT

The authors disclose AI assistance for code debugging and prose readability while retaining responsibility for the manuscript.

AI USE STATEMENT

  • The authors used AI tools to debug code and improve the readability of author-written text.Source: AI USE STATEMENT · p. 10
  • The authors state that conceptual development, theoretical analysis, experimental prompts, and interpretation of results were carried out independently.Source: AI USE STATEMENT · p. 10
  • The authors reviewed and verified the manuscript and take responsibility for its integrity and accuracy.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The ethics statement describes benchmark-based evaluations without human subjects or sensitive personal data; Appendix A.1 estimates the prompting-related attenuation coefficient.

ETHICS STATEMENT

  • Evaluations use publicly released standardized academic benchmarks and involve no human subjects, human annotations, or personally identifiable or sensitive data.Source: ETHICS STATEMENT · p. 10
  • Code generation and execution evaluations run in isolated, secure sandboxes, and the authors foresee no immediate ethical concerns or direct societal harms.Source: ETHICS STATEMENT · p. 10

A.1 ESTIMATION ALGORITHM

  • The estimation procedure compares PI-induced and prompted-teacher shifts against a shared PI-free reference, then averages fitted attenuation slopes across trajectory positions and inputs.A confidence interval with an upper endpoint below 1 is evidence that the mean affine-response coefficient is below 1.Source: §A.1 · p. 15

A.2 SETUP AND RESULTS

The analysis estimates how teacher prompts attenuate privileged-information effects and examines when that attenuation preserves or increases teacher entropy. Across model-task settings, estimated attenuation coefficients support the proposed average attenuation effect.

A.2 SETUP AND RESULTS

  • The initial-policy estimates pair failed rollouts with verified solutions and measure full-vocabulary teacher distributions at sampled prefixes.For each problem, prefix estimates are averaged before aggregation across problems, with 95% bootstrap CIs resampling problems.Source: §A.2 · p. 16
  • 95% CIs for bγ lie entirely within (0, 1) in every model-task setting; estimates are 0.656–0.736 for grading and text-to-SQL versus 0.845–0.954 for LiveCodeBench and chemistry.The estimates support average PMI attenuation under teacher prompting before task-specific training.Source: §A.2 · p. 16, Table 4 · p. 16

B.1 PROMPTED TEACHER DISTRIBUTION

  • The prompted teacher distribution is derived by exponentiating the attenuated PI-induced log-probability shifts and substituting the PI-conditioned distribution.The derivation expresses the prompted distribution using the PI-free policy and the shifts induced by privileged information.Source: §B.1 · p. 17

B.2 TEACHER ENTROPY UNDER PMI ATTENUATION

  • The entropy analysis defines a normalized path of teacher distributions as the attenuation parameter varies.It writes the log probability along this path using the unprompted teacher distribution, the PI-induced shift, and a normalizer.Source: §B.2 · p. 17

B.3 ENTROPY INCREASE UNDER MODERATE ATTENUATION

  • When the covariance condition holds, sufficiently small attenuation increases teacher entropy under the affine model.The proof uses a positive covariance and a local smoothness argument around no attenuation.Source: §B.3 · p. 18

C.1 EXPERIMENTAL SETUP

  • The covariance experiment pairs PI-free rollouts with verified solutions and groups prefixes by sampled-token PMI magnitude.The sampled-token score sets the group, while covariance is computed over the full vocabulary under the PI-conditioned teacher distribution.Source: §C.1 · p. 18

C.2 RESULTS AND DISCUSSION

The analysis relates sampled-token PMI to full-vocabulary covariance under the PI-conditioned teacher. The positive covariance condition is concentrated among prefixes with large sampled-token PMI magnitude.

C.2 RESULTS AND DISCUSSION

  • Approximately 5–14% of evaluated positions have |ut(yt)| > 1, and most sampled tokens in this group have negative PMI.The group captures strong distillation signals despite its small share of prefixes.Source: §C.2 · p. 18, Table 5 · p. 19
  • Figure 5 bins sampled tokens by PI-free and PI-conditioned probabilities, colors bins by mean PMI, and marks unchanged probability with a dashed diagonal.Its bins show sampled-token patterns of PI reinforcement and suppression.Source: §C.2 · p. 18, Figure 5 · p. 19
  • 0.43–0.58 mean covariance occurs in the |ut(yt)| > 1 group across all 12 model-task settings, with positive covariance in 79–91% of its prefixes.Under the affine attenuation model, this positive alignment implies that sufficiently small attenuation increases teacher entropy.Source: Table 5 · p. 19, §C.2 · p. 19

D TRAINING CONFIGURATION

These appendices document training and evaluation settings, entropy measurements, prompt-set reporting, and a control without privileged information. The control indicates that prompts regulate use of privileged solutions rather than replace their supervisory signal.

D TRAINING CONFIGURATION

  • Training uses eight rollouts per prompt, batch size 32, 2048-token prompt length, temperature 1, no KL regularization, and early stopping after four validations without improvement.Evaluation uses 16 samples per problem at temperature 0.6 and top-p 0.95; Python uses a separate OPSD recipe and longer outputs.Source: D TRAINING CONFIGURATION · pp. 19–20

E STUDENT ENTROPY VALUES

F TEACHER ENTROPY VALUES

  • Teacher prompts raise entropy on every task; single prompts yield 1.6–2.0 times the OPSD teacher entropy and aspect prompts 3.3–3.4 times on grading and chemistry.The two Python prompts asking for a minimal program are exceptions to the general increase.Source: F TEACHER ENTROPY VALUES · p. 21, Table 8 · p. 21

G PROMPT-SET ABLATION: ALL METRICS

H CONTROL: THE TEACHER PROMPT WITHOUT PRIVILEGED INFORMATION

  • Without privileged information, the prompted teacher improves over the untrained model by only 1–3 points and remains 7–23 points below VRSD across tasks.It is 10–12 points below OPSD on three tasks, supporting the conclusion that the prompt does not replace the supervisory signal from c.Source: H CONTROL: THE TEACHER PROMPT WITHOUT… · p. 22, Figure 6 · p. 22

I TRAINING PROMPTS

  • Task prompts direct verification behaviors, including schema and join checks for SQL, boundary checks for code, and evidence-based rubric checks for grading.The single prompt asks for reflection before commitment; aspect prompts target specific task procedures.Source: I TRAINING PROMPTS · pp. 22–24

J.1 SETUP

  • Pattern prevalence is measured with deterministic rules, without model-based judges, on student rollouts and teacher continuations of half-written answers.Student evaluation uses 16 test rollouts per problem; teacher continuations use the question alone, with c, or with c plus a prompt.Source: §J.1 · p. 24

J.2 RESULTS: WHAT THE PROMPT CHANGES IN THE TEACHER’S TEXT

Teacher continuations show that privileged solutions can improve answer correctness without supplying reasoning steps. Task-specific prompts restore structured reasoning behaviors, though some adversarial prompts reduce continuation accuracy.

J.2 RESULTS: WHAT THE PROMPT CHANGES IN THE TEACHER’S TEXT

  • With privileged solutions, the teacher’s continuation accuracy rises across chemistry, grading, and SQL, while qualitative reasoning behaviors remain largely unchanged.The solution supplies the answer rather than the reasoning.Source: §J.2 · p. 24, Table 11 · p. 25
  • Given an unconstrained prefix in SQL, the OPSD teacher emits the privileged query verbatim; on Python it drops intermediate rationales and jumps to implementation.These examples show shortcut continuations that bypass intermediate reasoning steps.Source: §J.2 · p. 25
  • Prompts restore intermediate reasoning: chemistry elimination prompts reject incorrect options in over a third of rollouts, while SQL and Python verification prompts elicit checks across most partial drafts.The chemistry example shows both teachers reach the correct answer, but the prompted teacher explicitly rules out alternatives.Source: §J.2 · pp. 25–26, Figure 7 · p. 26
  • Chemistry d3 and grading d4 adversarial prompts sharply reduce continuation accuracy despite correct solutions in context; no Python prompt falls below OPSD accuracy.The authors connect these exceptions to student Worst@16 collapse in Table 9.Source: §J.2 · p. 26

Related papers

Loading 2610.16354v1…