PaperlayerView exact revision on arXiv

Source-linked AI summary

A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning

Mayug Maniparambil, Ayush Rai, Ekansh Verma, Noel E. O'Connor, Fergal Reid

arXiv:2610.13744v1cs.AIcs.CLcs.LG

TL;DR

Reference-conditioned self-distillation offers dense token-level signals, but the paper tests whether those signals actually localize errors or correct erroneous choices. It finds that scoring alone does not reliably do so, while critique-conditioned scoring uses generated error critiques to produce a more localized and corrective signal.

  • Problem

    Verifiable-reward RL gives only a correct-or-incorrect outcome and may provide no group-relative advantage when all sampled rollouts fail, while dense self-distillation signals still need to localize and correct errors to provide useful credit.

    Source: §1 · p. 1

  • Method

    The paper tests reference-conditioned teacher signals at annotated error spans and introduces critique-conditioned self-distillation, which generates an error critique before conditioning the scoring teacher on it.

    Source: §1 · p. 2, §3.3 · p. 3

  • Results

    Reference-conditioned signals are weakly localized, do not reliably suppress erroneous tokens, and do not reliably distinguish correct from incorrect same-problem rollouts, whereas critique-conditioned signals localize errors and lower probability on the student's continuation at flagged errors.

    Source: §1 · p. 2, §7 · p. 6

  • Takeaways & Limitations

    The findings suggest that error credit requires an explicit search over the reasoning trace, with generation identifying an error and scoring converting it into dense token-level supervision.

    Source: §1 · p. 2, §9 · p. 8

    Training evidence is limited to a 320-puzzle set and 100 steps, and critique-conditioned self-distillation has not been run at full RL scale.

    Source: §9 · p. 8

Abstract

from arXiv · show

On-policy self-distillation promises to replace the sparse outcome signal of verifiable-reward RL with dense token-level credit: a teacher conditioned on the reference solution scores the student's own rollout, and the resulting distributional shift is distilled back into the policy. We investigate whether this scoring signal actually contains corrective credit. Using gold error spans in student rollouts, we test the two signals consumed by self-distillation objectives, the full-distribution KL and the token log-ratio, for two properties: whether they localize errors and whether they push probability away from erroneous student tokens. They exhibit neither of these properties nor can they reliably distinguish correct from incorrect rollouts within the same problem. We argue that this failure is structural: locating an error in a reasoning trace is itself a search problem, while a scoring teacher is never explicitly asked to perform that search. The same model, with the same privileged information, localizes errors substantially better when allowed to generate a critique. This forms the motivation for \emph{critique-conditioned self-distillation}: first generate a critique of the current rollout, then condition the scoring teacher on that critique. The resulting signal localizes errors, is corrective within error spans, and shifts probability toward tokens that initiate backtracking such as ``Wait'', ``But'', and ``Actually''. We further show that the critique is useful both at test time, where resuming a wrong rollout with the critique in context converts math problems the model cannot solve by sampling, and in cold-start GRPO, where it provides an advantage signal on prompts for which verifiable rewards alone provide none.

1 Introduction

The paper tests whether reference-conditioned on-policy self-distillation provides corrective credit, not merely a dense signal. It finds that scoring does not reliably identify or correct errors, and motivates adding an explicit critique-generation step before scoring.

1 Introduction

  • On-policy self-distillation scores a student's rollout with a teacher given privileged information, yielding token-level discrepancies even when the final answer is wrong.A dense signal is not necessarily corrective: it should be larger where reasoning fails and push probability away from erroneous choices.Source: §1 · p. 1
  • Reference-conditioned scoring is weakly concentrated at annotated errors, does not reliably suppress erroneous tokens, and fails to distinguish correct from incorrect rollouts of the same problem.The authors argue that the limitation lies in scoring itself, not in how a particular loss consumes its signal.Source: §1 · p. 2
  • Critique-conditioned self-distillation first generates a localized critique, then conditions the scoring teacher on it so generation searches for errors and scoring supplies dense token-level supervision.The same model with the same reference can localize errors when allowed to generate a critique.Source: §1 · p. 2
  • The paper's contributions include evaluating reference-conditioned signals at gold error spans, testing critique-based localization, and reporting initial test-time inference and training results.The paper frames its three contributions as showing the privileged signal's limitations, locating errors through critique generation, and using critiques for supervision.Source: §1 · p. 2

2 Related Work

The paper positions its work between privileged self-distillation and methods for locating reasoning errors. It distinguishes its contribution by directly testing whether the resulting signal carries corrective credit at gold error spans.

2 Related Work

  • Prior work uses privileged information to provide dense token-level supervision over a student's own rollout [Zhao et al., 2026a, Hübotter et al., 2026, Yang et al., 2026, Pan et al., 2026].Recent studies identify context-induced bias and weak or stylistic distributional shifts [Harne et al., 2026, Kaur et al., 2026, Pan et al., 2026].Source: §2 · p. 2
  • A separate line of work treats error localization as requiring generation or explicit reasoning, motivating generative verifiers, process supervision, and critique-based reward models [Tyen et al., 2024, Zhang et al., 2025a, Ankner et al., 2024, Khalifa et al., 2025].This paper differs by measuring the signal directly at gold error spans and testing whether it carries corrective credit.Source: §2 · p. 2

3 Preliminaries

The preliminaries connect sparse group-relative rewards to the limits of reference-conditioned token signals and define a probe for testing their error-localization and correction properties. They then introduce critique-conditioned scoring, which uses a generated critique as privileged context.

3.1 Group-relative RL and the all-wrong group

  • When every sampled rollout is wrong, group-normalized advantages vanish, leaving the hardest prompts without a gradient from the group-relative objective.These all-wrong groups are described as problems the model cannot yet solve within G tries.Source: §3.1 · p. 2

3.2 On-policy self-distillation

  • On-policy self-distillation evaluates the student's rollout with the same network conditioned on privileged information, interpreting the teacher–student discrepancy at each position as token-level credit.The full-distribution variant uses per-token KL, while RLSD uses the written-token log-ratio to modulate the group advantage; the verifier sets update direction and the teacher modulates magnitude.Source: §3.2 · pp. 2–3

3.3 A general form of the probe

  • The probe compares teacher scoring with and without privileged context on the same rollout, using KL for distributional change and a token log-ratio whose negative values indicate correction.The KL readout tests localization, while the log-ratio tests whether context lowers the probability of the student's erroneous token.Source: §3.3 · p. 3
  • Matched-placebo comparisons subtract signals from another problem's reference or critique to control for shifts caused by a solution-shaped preamble alone.The critique condition uses a critique of another problem's rollout as its placebo.Source: §3.3 · p. 3
  • A generated critique quotes an erroneous sentence and explains the error; critique-conditioned self-distillation replaces the reference with that critique so generation searches and scoring supplies a dense signal.The paper separately ablates the quoted sentence and its explanation.Source: §3.3 · p. 3

4 Setup

The setup evaluates token-level signals on logic and competition-mathematics rollouts, using model-generated or oracle critiques and judged gold error spans. Its error-span Z readout compares the first-error region against clean regions within each rollout and centers out a clean-sentence control.

4.1 Models, data, and labels

  • The signal analysis uses Qwen3 and Olmo-3 rollouts on Knights & Knaves logic puzzles and two DeepMath difficulty bands, with Qwen3-8B also used as a scale-control teacher.Teachers are the same models as the students, evaluated with privileged context.Source: §4.1 · p. 3
  • Gold error spans are sentence quotes with reasons, accepted independently by two judges; critics include an oracle, self-critics, and a larger Qwen3 critic, all given the teacher's reference.The study also analyzes human-annotated ProcessBench error spans.Source: §4.1 · pp. 3–4

4.2 The error-span Z readout

  • The error-span Z readout normalizes a short window after the first gold error against length-matched clean windows from the same rollout.Only the first error is used because later reasoning is often wrong in ways the gold spans do not mark; localization KL is log-transformed and scores are calibrated against random clean sentences.Source: §4.2 · p. 4

5 The Privileged Signal Is Empty Where It Matters

Reference-conditioned scoring provides little useful credit at student errors: its signals neither localize errors reliably nor distinguish correct from incorrect rollouts within the same problem. The pattern persists across tasks and teacher scales, with some settings even increasing probability on erroneous tokens.

5 The Privileged Signal Is Empty Where It Matters

  • On K&K, error and clean traces are nearly indistinguishable; on DeepMath, the reference causes only a small boundary shift, much weaker than critique conditioning.Figure 1 aligns content KL and student-token log-probability changes to error and clean sentence boundaries.Source: §5 · p. 4, Figure 1 · p. 5
  • Across students, tasks, and both signals, reference-conditioned scores stay near the clean baseline, and no setting meets the corrective criterion.Table 1 defines positive error-span Z as localized and negative student-token log-probability change as corrective.Source: §5 · p. 4, Table 1 · p. 5
  • On L7-10, the Qwen teacher assigns more probability to the erroneous student token under reference conditioning, and an 8B teacher does not rescue the signal.Critique-conditioned signals become stronger across settings with the larger teacher.Source: §5 · p. 5
  • All twelve within-problem AUC confidence intervals include chance, so neither reference-conditioned signal reliably ranks correct rollouts above incorrect siblings across three tasks and both teacher scales.Pooled deviations from chance also appear under content-free controls.Source: §5 · p. 5, Figure 2 · p. 6
  • Table 2 measures generated critiques by error localization within five sentences, verbatim quote matching, and required-format compliance.The passage specifies the table’s measures but does not report their values.Source: Table 2 · p. 6

6 Dense Feedback Is a Search Problem

The paper frames error localization as a search problem that scoring alone does not perform. With the same reference, a model can generate critiques that locate errors, supplying anchors for critique-conditioned scoring.

6 Dense Feedback Is a Search Problem

  • The proposed explanation is that finding where and why a rollout fails requires comparing it with the reference, a search that a scoring pass is not asked to perform.Critiques request quoted errors with explanations, making their predictions directly localizable and usable as anchors for teacher scoring.Source: §6 · p. 6
  • The strongest critic localizes a gold error on well over half of rollouts across tasks; smaller critics succeed less often but preserve the same scale ordering.Common failures are practical: paraphrasing instead of quoting the rollout or violating the required format.Source: §6 · p. 6

7 Critique-Conditioned Self-Distillation Delivers

Conditioning the scoring teacher on a generated critique separates error and control traces and makes its shifts corrective at the flagged errors. The critique’s quote and reason contribute differently: the quote localizes the error, while the reason directs correction.

7 Critique-Conditioned Self-Distillation Delivers

  • With critique conditioning, error and control traces separate, and the teacher shifts its distribution after the flagged error sentence, lowering probability on the student’s continuation there across tasks.The same teacher, alignment, and estimand are used as in the reference-conditioned comparison.Source: §7 · p. 6, Figure 3 · p. 7
  • Oracle critiques satisfy the corrective criterion on both signals: they localize errors and pull probability away from erroneous continuations, at several times the reference-conditioned magnitude.The critic sees only the reference already available to the scoring teacher.Source: §7 · p. 6
  • Self-generated critiques yield positive localization and negative correction on all three tasks, with stronger effects at larger model scales.The self-critique training arm recovers most of the oracle-critique arm’s gain over GRPO.Source: §7 · p. 7
  • Critique conditioning raises reconsideration markers such as “Wait,” “But,” and “However” to two to three times the control rate, unlike reference conditioning.The comparison examines high-content tokens in the post-error window.Source: §7 · p. 7, Table 3 · p. 7
  • A critique’s quote primarily localizes the error, while its reason drives correction; removing the reason can shrink the corrective shift or even reverse it.Removing the quote weakens localization across tasks.Source: §7 · pp. 7–8, Table 4 · p. 7
  • At test time, critique content helps the model solve math problems it could not solve by sampling when it resumes its own wrong rollouts from a cut.The reported table covers sample accuracy on all problems and on those unsolved in eight samples.Source: Table 5 · p. 8

8 Using the Signal at Test Time

The paper tests whether a critique can help a model recover from its own wrong math rollout. Resuming from the critique-flagged error sentence solves problems the model had not solved by sampling more often than a cue-only control.

8 Using the Signal at Test Time

  • With the critique in context, the model solves problems it never solved by sampling about twice as often as the control.The test uses wrong Olmo-3-32B-Think rollouts on L7-10 and resumes from the first sentence flagged by the critique.Source: §8 · p. 8
  • The comparison cuts each rollout at its first critique-flagged sentence and resumes it with eight samples, against a fixed-depth cut with the correction cue alone.The study uses 55 rollouts from 39 problems, including 19 problems unsolved in eight samples.Source: §8 · p. 8
  • Because the critic had the reference solution in context, the gain measures a privileged critique at inference, not unaided self-correction.Source: §8 · p. 8

9 Conclusion and Limitations

The paper concludes that reference-conditioned self-distillation is dense but does not provide reliable corrective credit, while critique generation lets the teacher locate errors before scoring. Initial training evidence includes a four-puzzle cold-start pilot, but training remains small-scale and all critiques use a reference.

9 Conclusion and Limitations

  • Reference-conditioned self-distillation provides dense supervision but does not reliably localize errors or distinguish correct from incorrect rollouts.The authors argue that error localization requires an explicit search over the reasoning trace, separated from the scoring pass.Source: §9 · p. 8
  • Training evidence is limited to a 320-puzzle set and 100 steps, with no full-scale RL run; critique generation adds 6% of step time with a student critic and about a fifth with an API critic.Every critique in the study uses the reference solution in context, so the results do not demonstrate unaided self-correction.Source: §9 · p. 8

A Training with the Critique-Conditioned Signal

  • The training experiments compare critique-conditioned training with GRPO and reference-conditioned self-distillation over 100 steps, using both an oracle critic and the student as critic.The appendix also reports the compute cost of the critique pass.Source: A Training with the… · p. 13

A.1 Cold-start pilot on four puzzles

  • On four puzzles the model never solved in 64 samples, the critique-conditioned cold-start arm solved two puzzles that GRPO never solved.GRPO solved one puzzle by chance and never solved the other three; both arms used one seed and ran 30 steps of 32 rollouts.Source: Figure 4 · p. 13, §A.1 · p. 13

A.2 Training against GRPO and reference-conditioned self-distillation

Critique-conditioned training adds substantial out-of-distribution accuracy over GRPO and reference-conditioned self-distillation, while its critique pass uses a small share of training time. The section also illustrates how quote location and reason content serve different roles, and details the study’s critique and error-labeling procedures.

A.2 Training against GRPO and reference-conditioned self-distillation

  • The critique term adds 5.5 OOD points over GRPO and a further 3.9 over RLCSD; self-critique recovers most of the oracle gain.Self-critique reaches 0.593, compared with 0.606 for the oracle and 0.567 for RLCSD; its quotes match a rollout sentence 82% of the time versus 96% for the oracle.Source: §A.2 · p. 13, Table 6 · p. 14

A.3 Compute cost of the critique pass

  • The critique pass takes 6% of step time when the student writes short critiques averaging 175 tokens.Only 16–18 of 128 rollouts per step are critiqued; those critique tokens are 0.4% of rollout tokens generated per step.Source: §A.3 · p. 14, Table 7 · p. 14

B Critique Examples

  • In the examples, the oracle critique quotes the gold error and explains its correction, while the self-critique can give a correct reason without locating the error.For the residue example, the self-critique’s quote misses a derivation step even though its reason gives the correct residue.Source: B Critique Examples · p. 15

C Extended Related Work

  • The paper distinguishes critique-based error search from prior approaches that alter privileged context or consume a scored signal, and frames its method as combining generated critiques with token-level distillation.The authors note that token-level likelihoods from scoring a forced trace do not localize errors, whereas generating a critique does.Source: C Extended Related Work · p. 16

D Setup Details

  • The study samples student rollouts, labels wrong non-truncated boxed-answer rollouts, and compares conditioning blocks against matched, fixed-derangement placebos.Gold spans are grader-flagged error sentences; strict gold requires acceptance by both judges, while lenient gold requires acceptance by the Claude jury.Source: D Setup Details · pp. 16–17

D.1 Critique prompt

  • Critics are instructed to quote the student’s words, identify the earliest error, report only certain errors, and give a concise critique in a fixed format.The prompt allows up to five errors and asks for at most 120 words total.Source: §D.1 · p. 18

D.2 Gold-span grader prompt

  • The gold-span grader lists certain reasoning errors in order, quotes each student sentence verbatim, and records a reason, severity, and confidence.It should flag the earliest actual error, not merely the final answer, and omit later steps that only inherit an earlier mistake.Source: §D.2 · pp. 18–19

D.3 Jury prompt

  • Two models independently judge each flagged sentence, with position-matched control sentences used to measure jury leniency.A flag is accepted only when the marked reasoning contains a genuine error, rather than a downstream consequence or legitimate alternative reading.Source: §D.3 · p. 19

E Error-Span Z: Full Results

Across the expanded settings, reference-conditioned signals move with clean regions rather than selectively reacting at errors, whereas critique-conditioned signals react at error sentences. The same pattern appears across teacher scales and another model family; ablations separate the contributions of critique quotes and reasons.

E Error-Span Z: Full Results

  • The error-span results report conditioning and teacher comparisons on Qwen3-1.7B rollouts, including an 8B scale-control teacher and undifferenced negative controls.The corresponding Olmo-3 student tables are in Appendix I; the reported intervals are cluster bootstraps by problem.Source: E Error-Span Z: Full Results · p. 19, Table 8 · p. 19, Table 9 · p. 20

F Full Figures

  • With reference conditioning, error and control curves move together across settings; critique conditioning raises the signal at error sentences and nowhere else.The critique-conditioned pattern is stronger with the 8B teacher and weaker with the student’s own critiques.Source: F Full Figures · p. 20, Figure 5 · p. 21, Figure 6 · p. 22

G Correctness Prediction: Full Results

  • The reference-conditioned trace-mean signal does not reliably rank correct above wrong sibling rollouts within a problem; every reported interval includes chance.The within-problem test covers 34, 28, and 49 problems on K&K, DeepMath 5-7, and DeepMath 7-10, respectively.Source: Table 10 · p. 20, G Correctness Prediction: Full Results · p. 20

H Critique Ablations: Full Results

  • Removing critique quotes lowers localization, while removing reasons lowers correction; the same ablation pattern appears on Olmo-3-7B-Think rollouts.The paired per-rollout comparison measures each complete critique against its own ablated version.Source: H Critique Ablations: Full Results · pp. 20–21

I Replication on Olmo-3-7B-Think and Olmo-3-32B-Think

  • On both Olmo-3 models, reference-conditioned scoring does not react at errors, while critique-conditioned scoring localizes and corrects on every task.The 32B model’s self-critiques localize on all three tasks and correct on DeepMath 7-10; absolute rates are comparable only within each model because their rollouts differ in length.Source: I Replication on Olmo-3-7B-Think and… · p. 22

J Validation on Human-Annotated Error Spans

On human-labelled ProcessBench error steps, reference conditioning remains flat while critique conditioning reacts at errors. Correction depends on whether the critique points inside the human-labelled error span.

J Validation on Human-Annotated Error Spans

  • On human-labelled error steps, reference conditioning is neither localized nor corrective, while critique conditioning localizes errors for both evaluated models.Correction is clear for Qwen2.5-7B-Instruct and initially weak for Qwen2.5-Math-7B-Instruct; removing quoted tokens makes the signal corrective for both.Source: J Validation on Human-Annotated Error… · pp. 22–23, Table 17 · p. 26
  • Correction appears when the critique points inside the human-labelled error span and disappears when it points elsewhere.The oracle critique is generated without access to the human label.Source: J Validation on Human-Annotated Error… · pp. 22–23
Loading 2610.13744v1…