Recovering Reliable Supervision via Critique-Guided On-Policy Distillation

• OPD trains on student-generated sequences, but reliable corrective supervision depends on the teacher recovering from student prefixes rather than following their errors.Offline distillation has an occupancy mismatch because students encounter their own prefixes at inference, which may be underrepresented in teacher-generated training sequences.Source: §1 · p. 1
• Prior approaches reduce supervision or replace sequences to avoid unreliable targets, but both increase occupancy mismatch; this work seeks more reliable supervision with as little added mismatch as possible.Reduction leaves later inference states untrained, while replacement trains on sequences students rarely generate.Source: §1 · pp. 1–2
• On incorrect rollouts, teacher recoverability declines sharply—even on solvable problems—and up to half of failed continuations end in the student’s incorrect answer, despite retained error recognition.The teacher may fail to continue correctly while still recognizing errors.Source: §1 · p. 2
• CriticOPD has the teacher critique errors, the student rewrite from the last error, and the teacher score the rewrite with the critique while the student learns without it.The critique is privileged information during training; supervised tokens remain student-generated.Source: §1 · p. 2

2 Measuring the Reliability of Teacher Supervision

• Recoverability is expected completion accuracy from a student prefix; sampled rollouts were cut at paragraph boundaries and stratified by teacher success on the original problem.The estimates use teacher and student continuations from each cut point.Source: §2 · p. 2

2.1 Recoverability Declines Along Incorrect Rollouts

• ΛT stays ≥0.93 on correct rollouts, falls 0.82→0.29 on incorrect rollouts from high-initial-success problems, and falls 0.14→0.05 when initial teacher success is low.The contrast with correct rollouts suggests the decline is associated with erroneous prefixes, while low teacher capability is another source of unreliable supervision.Source: Figure 3 · p. 3, §2.1 · p. 3

2.2 The Teacher Follows the Error but Still Recognizes It

• 1%→35% of teacher continuations end with the student’s incorrect answer as prefixes grow, while valid error flags stay between 73% and 81% across recoverability bins.The incorrect-answer share rises from 8% to 50% of failed continuations; valid flags are 73% at ΛT < 0.2 and 74% at ΛT ≥0.8.Source: Figure 4 · p. 3, §2.2 · p. 3

2.1 Recoverability Declines Along Incorrect Rollouts

• 53% of tokens in incorrect rollouts come from prefixes with ΛT < 0.2, and approximately one third of all training tokens receive supervision from such prefixes.Incorrect rollouts account for 61% of training tokens; only 21% of incorrect-rollout tokens correspond to prefixes with ΛT ≥0.8.Source: §2.1 · p. 3

3.1 Critique and Rewrite

• For incorrect or unfinished rollouts, the teacher identifies errors, the student retains the prefix through the last identified error, and then rewrites the suffix using the critique.The student generates every token in the revised training sequence; correct rollouts and rollouts without a valid critique use standard OPD.Source: §3.1 · p. 4, Figure 5 · p. 5
• At 75% of the continuation, teacher recoverability is 0.14 on the original rollout and 0.67 on the rewrite; student success rises from 0.03 to 0.16.Figure 6 plots teacher and student recoverability against position from the cut point, with dashed original rollouts and solid rewrites.Source: §3.1 · p. 4, Figure 6 · p. 5

3.2 Training Objective

• For correct rollouts and those without a valid critique, CriticOPD uses standard OPD at student-generated prefixes with a sampled-token advantage.The objective minimizes reverse KL for these rollouts.Source: §3.2 · p. 4
• For valid critiques, the teacher scores the rewrite with the critique in context while the student uses its critique-free prefix, and only the rewritten suffix receives supervision.The suffix is sampled with the critique, but the student's log probabilities are computed without it.Source: §3.2 · p. 4
• CriticOPD combines the standard OPD and feedback-distillation losses, assigning each rollout to exactly one term.Because the suffix is sampled with the critique but the student's log probabilities omit it, the update is a surrogate rather than an unbiased gradient of Equation (2).Source: §3.2 · p. 4

4.1 Setup

• The experiments distill non-thinking Qwen3-1.7B and Qwen3-0.6B students from a Qwen3-4B teacher on DAPO-Math-17k, evaluating six math benchmarks.Methods share a training recipe; the reported results use three seeds and selected checkpoints.Source: §4.1 · p. 5

4.2 RQ1: Does critique-guided rewriting improve OPD?

• 4.57 points (50.08 →54.65) on Qwen3-1.7B and 4.06 points (34.80 →38.86) on Qwen3-0.6B: CriticOPD improves OPD overall.It achieves the best student accuracy on five of six benchmarks at both scales.Source: §4.2 · p. 6, Table 1 · p. 7
• CriticOPD’s advantage is largest on competition problems, with Qwen3-1.7B gains of +6.01, +7.36, and +6.94 on AIME 2024, AIME 2025, and AMC 2023.It also outperforms FastOPD and RelayOPD at both scales; outside mathematics, its average is comparable to OPD, with modest code-benchmark declines.Source: §4.2 · p. 6

4.3 RQ2: Which components of CriticOPD matter?

• Removing the critique lowers the Qwen3-1.7B average by 5.90 points, while removing the reference answer lowers it by 0.85 and generic feedback by 1.46.Teacher-written suffixes lower the average by 0.66; scoring student suffixes without feedback lowers it by 1.19.Source: §4.3 · p. 6, Table 2 · p. 7

4.4 RQ3: Where do the gains of CriticOPD come from?

• CriticOPD combines the highest teacher reliability with a low occupancy mismatch and the highest average accuracy.Its occupancy mismatch is 0.16, above OPD’s 0.10 but below FastOPD’s 0.21 and RelayOPD’s 0.35; its rewrite runs from the cut point to the response end.Source: §4.4 · pp. 6–7, Figure 7 · p. 6
• 1.47 per token: under CriticOPD scoring, a correcting continuation leads one continuing the error, compared with 0.24 under OPD scoring.The first 32 tokens account for 87% of the increase in this advantage.Source: §4.4 · pp. 7–8, Figure 8 · p. 8

The student learns to recover from its own

• On incorrect OlympiadBench prefixes, CriticOPD’s student leads OPD by 0.023 in success rate on average, while the students differ by at most 0.015 on empty prefixes and 0.000 on average on correct prefixes.The gain reaches about 0.04 at the last two cut points; part of it comes with longer responses, but the critique still helps at the same response length.Source: The student learns to recover from its… · p. 8, Figure 8 · p. 8

On-policy distillation and its supervision

• Prior approaches to declining teacher reliability truncate, select, or reweight supervision, whereas CriticOPD rewrites the rollout after its last error.The cited approaches include stopping or truncating supervision and choosing reliable contexts.Source: §5 · p. 8

Teacher intervention and trajectory repair

• Teacher-intervention methods use interleaved generation, revisions, takeovers, or regeneration, while CriticOPD has the teacher write only a critique that is removed before training.Every supervised token therefore comes from the student.Source: §5 · p. 8

Privileged context and language feedback

• Context distillation and language-feedback methods provide related training precedents, but CriticOPD uses a teacher critique for each incorrect rollout and needs neither feedback nor an extra turn at inference.Source: §5 · p. 8

6 Conclusion

• On-policy distillation depends on teacher recoverability, which fails after student errors even as the teacher continues to recognize them.Source: §6 · p. 8
• CriticOPD uses the teacher’s critique to restore recoverability and improves OPD on mathematical reasoning benchmarks at two student scales.Source: §6 · p. 8
• The critique changes the context for teacher scoring, while the student writes every supervised token and receives no feedback at inference.Source: §6 · p. 8

Limitations

• Reference-answer dependence limits direct application to tasks with verifiable answers; other tasks would need a reward model or a critic that judges correctness independently.Source: Limitations · p. 9
• Experiments cover non-thinking Qwen3 students at two sizes and one OLMo-3 student configuration, all on mathematical problems; thinking-mode students, code training, and larger models remain untested.Source: Limitations · p. 9
• Each critiqued rollout requires an extra teacher and student generation, although the selected step is reached earlier and total training compute is 0.42 times OPD’s.Source: Limitations · p. 9
• The teacher’s listed errors are judged real 73% to 81% of the time, and the authors note that a more accurate critic or multiple critiques could reduce noise.The diagnosis also relies on a model judge that may disagree with human annotators.Source: Limitations · p. 9

B.4 Rewrite Analysis

• The rewrite analysis selected incorrect rollouts where teacher success began at least 0.5 and later fell below 0.5, then compared teacher and student continuations with and without critique or a generic hint.It also sampled continuations at several positions in the critique-guided rewrite and discarded original rollout.Source: §B.4 · p. 14

B.5 Diagnosis with the Qwen3-0.6B Student

• Teacher recoverability fell from 0.89 to 0.38 across cut points on 106 incorrect rollouts from teacher-solvable problems, while incorrect-answer continuations rose from 0.00 to 0.15.The diagnosis used the Qwen3-0.6B student and reports the same pattern as with Qwen3-1.7B.Source: Figure 9 · p. 14, §B.5 · p. 14, Figure 10 · p. 14
• After the cut point, teacher success fell from 0.45 to 0.14 on the original rollout but stayed between 0.70 and 0.75 on the rewrite; student success on the rewrite rose from 0.06 to 0.12.Between 77% and 84% of the teacher-listed errors were real across groups.Source: Figure 11 · p. 14, §B.5 · p. 14

B.6 Occupancy Mismatch

• A 16,384-token training limit leaves 9.3% of inference tokens unsupervised, while shorter limits shift supervision earlier and raise total-variation distance from OPD's 0.10 to 0.84 at 1,024 tokens.The position comparison measures token locations, not prefix content.Source: §B.6 · p. 15
• CriticOPD's hidden-state distance is 0.05, matching OPD and below FastOPD (0.09), SKD (0.15), RelayOPD (0.19), SFT and KD (0.29), and TRD (0.50).The distance is a classifier bound on total variation between method and inference state distributions.Source: §B.6 · p. 15

C.1 Training

• OPD optimizes a sampled reverse-KL objective with policy gradients and no KL penalty; one epoch over DAPO-Math-17k takes 139 steps.Table 3 lists shared hyperparameters for the methods.Source: §C.1 · p. 15, Table 3 · p. 15

C.2 Software and Hardware

• The implementation uses Python 3.12, PyTorch 2.11 with CUDA 13.0, Transformers 5.15, Ray 2.57, and vLLM 0.21; training runs generally use four A100 GPUs split between student and teacher.Qwen3-0.6B runs also use H100, H200, or B200 GPUs.Source: §C.2 · pp. 15–16

C.3 Configuration of CriticOPD

• CriticOPD discards incomplete critique items and uses the remaining response budget for the rewrite; training runs for 40 steps on Qwen3-1.7B and 80 on Qwen3-0.6B.Rollouts at the response limit are also critiqued using a prompt that marks the solution unfinished.Source: §C.3 · pp. 16–17

C.4 Baselines

• The baselines include teacher-generated SFT and KD, teacher-rewritten TRD, speculative-decoding SKD, teacher-takeover RelayOPD, and OPSD-T, which scores uncritiqued rollouts with the reference answer.Critique-GRPO instead critiques groups of seven incorrect rollouts and adds a refinement to the group when available.Source: §C.4 · p. 17

C.5 Evaluation

• Evaluation covers AIME 2024, AIME 2025, AMC 2023, MATH500, OlympiadBench, and Minerva Math, reporting the three-seed mean and sample standard deviation with Avg as their macro average.The teacher and initial students are evaluated once.Source: §C.5 · p. 17

C.6 Checkpoint Selection

• Checkpoint selection uses a validation set that does not overlap training or test benchmarks, chooses the highest-scoring candidate checkpoint, and applies the selected step across seeds.The validation score averages MATH-val and Comp-val accuracies; test results do not enter selection.Source: §C.6 · pp. 17–18

D Computational Cost

• CriticOPD reaches its selected step after 40 steps at 17.2 GPU-hours, 0.42 times OPD's cost at step 80, while using 0.96 times OPD's GPU-hours over the first 40 steps.Its step time grows from about 170 seconds in the first 10 steps to about 870 seconds over steps 31–40, versus OPD's 160 and 560 seconds.Source: D Computational Cost · p. 18

E.1 Results with OLMo-3 Models

• CriticOPD raises the OLMo-3 OPD average from 54.69 to 57.23, with gains of +5.83 on AIME 2025 and +3.16 on AMC 2023.The comparison uses the OLMo-3-7B-Instruct-SFT student and OLMo-3.1-32B-Instruct teacher; the critique does not end with the reference answer.Source: §E.1 · p. 18, Table 4 · p. 19

E.2 Results Outside Mathematics

• The non-math evaluation covers MMLU-Pro, MMLU, ARC-Challenge, OpenBookQA, HumanEval+, and MBPP+, using four samples per problem and a 32,768-token budget.MMLU and MMLU-Pro use 2,000-question subject-proportional subsets; code uses EvalPlus pass@1 on extended tests.Source: §E.2 · p. 19, Table 5 · p. 19

E.3 Per-Benchmark Ablation Results

• The per-benchmark ablation table reports all variants at step 40 except OPD, which is reported at step 80.The supplied passage states the table's scope but not its cell values.Source: §E.3 · p. 19

E.4 Sensitivity to Hyperparameters

• The average rises from 51.56 with one listed error to 53.31 with three and 53.72 with all errors; critique limits of 2,048 and 4,096 tokens give similar averages of 53.72 and 53.31.Limits of 512 and 1,024 tokens truncate 79% and 27% of critiques and yield averages of 51.72 and 52.78, respectively.Source: §E.4 · p. 19

E.5 Training Dynamics

• Training curves track reference-answer-reaching sequences, not student-only accuracy, and show longer CriticOPD training sequences than OPD at both scales.The counted sequences include teacher-sampled tokens in SKD, teacher takeovers in RelayOPD, refinements in Critique-GRPO, and post-critique rewrites in CriticOPD.Source: §E.5 · pp. 19–20, Figure 16 · p. 20
• The plotted training losses cover offline methods, while Critique-GRPO entropy is omitted because its logged value is not a per-token mean.Qwen3-0.6B SFT, KD, and TRD curves average seeds 43 and 44 because seed 42 logs are missing.Source: §E.5 · p. 20

E.6 Generation Length

• At 16k and 32k tokens CriticOPD exceeds OPD, but at 4k and 8k it falls below OPD; its responses average 10.2k tokens versus 5.3k for OPD.The comparison uses Qwen3-1.7B responses evaluated under budgets from 4k to 32k tokens.Source: §E.6 · p. 20, Figure 17 · p. 21
• At the 32k budget, length-matched variants score 2.39 points lower without critique and 1.78 points lower with generic feedback than CriticOPD.FastOPD, which produces responses of similar length, is 2.10 points lower; the comparison controls for response length on AIME.Source: §E.6 · pp. 20–21, Table 7 · p. 21