PaperlayerView exact revision on arXiv

Source-linked AI summary

ReTeach: Building a Self-Teacher through Multi-Round Reflection and Retry

Yafeng Tang, Hao Li, Hongsheng Yu, Qiang Fu

arXiv:2610.11529v1cs.AI

TL;DR

Sparse reward supervision and teacher-dependent distillation leave open how a model can construct useful supervision from its own attempts. ReTeach builds a self-teacher through iterative reflection and retry, then distills its context-conditioned predictions. It achieves the highest average accuracy on five of six benchmarks and ranks second on the remaining one.

  • Problem

    Sparse response-level rewards provide no information about which intermediate steps drove an outcome, while on-policy distillation depends on a sufficiently capable teacher.

    Source: §1 · p. 1

  • Method

    ReTeach iteratively reflects on failed student attempts and retries until success or a fixed budget, then applies outcome-aware on-policy distillation to the student's own rollouts.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Five of six benchmarks rank ReTeach highest, with the remaining benchmark ranking it second, including compared methods that use privileged information.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The results suggest that iterative deliberation can be internalized into a more efficient single-pass policy.

    Source: §6 · p. 9

Abstract

from arXiv · show

Self-distillation can improve reasoning without a separately trained, more capable teacher, but its effectiveness depends on how the self-teacher gains an advantage over the student. Conditioning the teacher on reference answers or solutions can provide such an advantage, but this information may be unavailable. Reflection offers a way to derive explicit error diagnoses and revision guidance from self-generated attempts, yet existing reflection-based methods often combine it with reference information, rich task feedback, or persistent memory. We introduce ReTeach, a Reflective self-distillation framework that constructs its self-Teacher through multi-round reflection and retry using only self-generated attempts and outcome-level verification. Starting from an unsuccessful student rollout, the teacher alternates explicit reflection with renewed attempts until success or the retry budget is exhausted, without reference answers or solutions, external diagnostic feedback, or cross-example memory. Each failed retry informs subsequent reflection, while successful correction provides outcome-level evidence for the potential utility of the resulting teacher context. An outcome-aware selection and weighting strategy distinguishes initially correct, reflection-corrected, and unresolved examples, assigning separate weights to their category-normalized distillation losses. Through on-policy distillation, the student matches the teacher's context-conditioned token-level predictive distributions at prefixes of its own rollouts, transferring the benefits of iterative correction while retaining single-pass inference. Across six benchmarks spanning mathematical reasoning, science question answering, and tool use, ReTeach improves average accuracy over GRPO by 1.39 percentage points.

1 Introduction

ReTeach addresses the limits of sparse reward supervision and self-distillation methods that depend on stronger teachers or privileged information. It builds a self-teacher through iterative reflection and retry, then weights distillation according to correction outcomes.

1 Introduction

  • RLVR supplies only a binary response-level reward, while on-policy distillation depends on a sufficiently capable teacher.The paper frames this as a barrier to dense supervision when a stronger teacher is unavailable.Source: §1 · p. 1
  • Prior self-distillation methods use reference answers, solutions, task feedback, or persistent memory, which may be unavailable or outside the desired setting.OPSD conditions on reference chain-of-thought, RLSD on ground-truth answers, and reflection approaches may use cached solutions or environment feedback.Source: §1 · p. 2
  • ReTeach iterates explicit reflection and renewed attempts from an unsuccessful student rollout until success or the retry budget is exhausted.Failures inform later reflection, while successful correction provides outcome-level evidence about the potential utility of the resulting teacher context, without reference solutions or external diagnostic feedback.Source: §1 · p. 2
  • Outcome-aware weighting separates initially correct, reflection-corrected, and unresolved rollouts into category-normalized distillation losses.This balances preserving existing behavior with learning from reflective contexts while accounting for the less certain utility of unresolved examples.Source: §1 · p. 2
  • ReTeach achieves the highest average accuracy on five of six benchmarks and ranks second on the remaining one, including against methods using privileged information.Across mathematical reasoning, science question answering, and tool use, it averages 65.62%, 66.71%, and 62.72%, respectively, versus GRPO's 64.32%, 65.43%, and 60.85%.Source: §1 · p. 2

2 Background

The background motivates denser supervision from reasoning traces and token-level distillation, then identifies privileged context as the remaining dependency of on-policy self-distillation.

Step-Level Chain-of-Thought Reasoning

  • Chain-of-thought exposes intermediate reasoning steps that can support step-level supervision, but process reward models or token-level proxies require auxiliary models or annotation.The text also describes chain-of-thought as improving accuracy on mathematical and multistep reasoning tasks.Source: §2 · p. 3

Token-Level Knowledge Distillation

  • On-policy distillation matches next-token distributions on student-generated rollouts, avoiding the train–inference mismatch associated with offline teacher-trace imitation.The teacher–student log-ratio acts as a per-token advantage.Source: §2 · p. 3

On-Policy Self-Distillation

  • On-policy self-distillation uses one policy as prompt-only student and context-conditioned teacher, but existing approaches still rely on external privileged information.ReTeach instead seeks to correct the model through iterative self-reflection.Source: §2 · p. 3

3 Methodology

ReTeach builds a teacher context from failed student attempts through repeated reflection and retry, then transfers it through on-policy distillation with outcome-aware weighting.

3.1 Building a Teacher through Multi-Round Reflect-and-Retry

  • A failed student response starts repeated reflection and retry, with each attempt informing later reflection, until success or the retry budget is exhausted.Teacher context uses self-generated attempts without external reference solutions or corrective feedback.Source: §3.1 · pp. 3–4

3.2 Stable On-Policy Distillation from the Deliberative Teacher

  • On-policy distillation compares teacher and student token distributions along the same student rollout, with the teacher additionally conditioned on deliberative content.The objective minimizes token-level Jensen–Shannon divergence.Source: §3.2 · p. 4
  • An exponential moving average updates the teacher after each student optimization step, allowing it to track student improvements while maintaining stable distillation targets.The teacher is initialized from the student; the smoothing parameter controls the degree of temporal smoothing.Source: §3.2 · p. 5
  • Rollouts are divided into initially correct, reflection-corrected, and unresolved categories; verified correction indicates potential supervision utility, while unresolved contexts are less certain.Corrected examples are distilled along the original student rollout, not the successful retry.Source: §3.2 · p. 5
  • Category-specific OPD losses are averaged within each nonempty group and combined with prescribed weights; absent groups contribute no loss or gradient, without renormalizing remaining weights.The category-level means are combined in the overall distillation objective.Source: §3.2 · p. 5

4 Experiment

ReTeach is evaluated against post-training baselines on mathematical reasoning, science question answering, and tool use. It leads overall, while ablations examine unresolved-rollout weighting, explicit reflection, teacher updates, and retry rounds.

4.1 Experiment setup

  • Experiments use Qwen3-8B and compare ReTeach with GRPO, OPSD, RLSD, CEPO, and RESD across held-out science and tool-use tasks and cross-dataset math benchmarks.The setup uses SciKnowEval Physics and Materials, ToolAlpaca, and competition-level math benchmarks; training and evaluation follow prior protocols.Source: §4.1 · pp. 5–6

4.2 Main Results

  • 65.50% total average puts ReTeach 1.39 percentage points above GRPO and 1.35 points above CEPO; it ranks first on five of six benchmarks.It ranks second on SciKnowEval-Material and has the highest avg@12 on all three math benchmarks.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 Ablation Studies and Discussions

  • Unresolved-rollout weighting is task-dependent: it gives no consistent improvement on Math or SciKnowEval, while Tool-Use benefits at wC = 0.2 but degrades at wC = 0.5.The ablation fixes wA = wB = 0.5 and varies wC; Figure 3 reports results across three seeds.Source: §4.3 · p. 7, Figure 3 · p. 7
  • Removing explicit reflection while retaining retries lowers SciKnowEval-Physics accuracy from 64.24% to 62.91%, a 1.33-point drop.Figure 4 compares reflection and EMA designs on avg@16.Source: §4.3 · p. 7, Figure 4 · p. 7
  • EMA provides stable, adaptive targets: its gradient norm stays lowest at 0.08–0.14 and entropy rises from 0.21 to 0.24 nats, unlike tracking’s instability or frozen-teacher adaptation limits.Figure 5 tracks teacher entropy and gradient norm; tracking’s gradient norm peaks at 25.27 while frozen-teacher values remain near 0.19.Source: §4.3 · pp. 7–8, Figure 5 · p. 8
  • On SciKnowEval-Physics, increasing K from 1 to 2 raises avg@16 from 63.54% to 64.24%, while K = 3 reaches 64.38% with greater training inference time.The third round adds only 0.14 points over K = 2 and reduces seed-to-seed variability; K = 2 is the default.Source: §4.3 · pp. 8–9, Figure 6 · p. 8

5 Related Work

Related work builds self-teachers from privileged answers, verified rollouts, or feedback, and uses reflection for iterative refinement or targeted credit assignment. ReTeach is positioned as online multi-round reflection and retry without reference answers, external diagnostic feedback, or cross-example memory.

Self-Teacher Construction

Reflection-Based Self-Improvement

  • Reflection-based methods range from inference-time refinement and trajectory repair to step-level feedback, error-localized distillation, and feedback-rich or memory-based systems.ReTeach instead constructs contexts online through multi-round reflection and retry, with failed attempts informing later reflection and successful correction indicating potential supervision utility.Source: §5 · p. 9

6 Conclusion

ReTeach constructs a stronger self-teacher through repeated reflection on failed responses, and experiments show consistent effectiveness across mathematics, science question answering, and tool use. The authors report that its iterative deliberation can be internalized into a more efficient single-pass policy.

6 Conclusion

  • The teacher revisits failed student responses through multiple rounds of explicit reflection and retry rather than relying on externally supplied privileged information.The resulting deliberative context provides token-level targets on rollouts sampled from the current student policy.Source: §6 · p. 9
  • Outcome-aware classwise weighting balances learning signals from initially correct, successfully refined, and unresolved problems, while an EMA update maintains a stable, evolving teacher.Source: §6 · p. 9
  • ReTeach achieves the best performance on five of six benchmarks and ranks second on the remaining benchmark, including against methods using privileged information.The experiments span mathematical reasoning, science question answering, and tool use.Source: §6 · p. 9
  • The findings suggest capabilities elicited through iterative deliberation can be internalized into a more efficient single-pass policy when only outcome-level verification is available.Source: §6 · p. 9

A Experiment details

Experiments cover mathematical reasoning, science question answering, and tool use with task-specific datasets and evaluation protocols. Training-time verification uses binary outcome rewards rather than reference solutions or chains of thought.

A Experiment details

  • Training-time verification uses an RLVR outcome verifier that marks attempts correct or incorrect without exposing reference chains of thought or ground-truth answers to the teacher.Source: A Experiment details · p. 13
  • Mathematics uses up to 30,000 OpenThoughts problem–solution pairs and evaluates cross-dataset on AIME 2024, AIME 2025, and February HMMT 2025.Source: A Experiment details · p. 13
  • Science uses SciKnowEval undergraduate Physics and Materials subsets, while tool use uses ToolAlpaca, with held-out within-dataset splits following SDPO.Source: A Experiment details · p. 13
  • Unresolved examples receive zero weight on Math and Science, but Tool-Use uses wC = 0.2; these weights were chosen from an ablation, not a separate validation split.Tool-Use degraded at wC = 0.5, while unresolved rollouts did not improve Math or SciKnowEval.Source: A Experiment details · p. 13
  • Training uses Qwen3-8B across Math, Tool-Use, and Science with a shared configuration, while the teacher thinks and the student is trained with thinking disabled.GRPO-style baselines use eight rollouts per question; remaining hyperparameters match ReTeach.Source: A Experiment details · p. 13
  • Within-dataset Science and Tool-Use report avg@16 with thinking disabled; cross-dataset Math reports avg@12 on AIME24, AIME25, and HMMT25 with inference thinking enabled.Source: A Experiment details · p. 13

B Data Samples

The appendix presents training-question samples from mathematics, science, and tool use, alongside configuration-table captions. The samples are paired with verifiable outcomes rather than reference solutions or human-written chains of thought.

B Data Samples

  • The examples span competition-level Math, Physics, Materials, and Tool-Use data sources, and training uses verifiable outcomes without reference solutions or human-written chains of thought.Source: B Data Samples · p. 13
  • The Math sample asks which condition relating two quadratic inequalities is necessary or sufficient for their solution sets to be identical.Source: B Data Samples · p. 13
  • The Physics sample is a multiple-choice gas-volume problem, while the Materials sample asks for a crystal unit-cell volume from lattice parameters.Source: B Data Samples · pp. 13, 15
  • The appendix identifies separate tables for ReTeach's training configuration and all methods' evaluation configuration.Source: Table 2 · p. 14, Table 3 · p. 14
  • The ToolAlpaca sample provides a binary-to-decimal tool specification and asks for a tool call to convert 101001110.Source: B Data Samples · p. 15

C Prompt Templates

ReTeach separates student and teacher prompt roles by reasoning mode, while both use self-contained answer formats. Teacher templates require explicit reflection on failed attempts before producing a new solution or tool transcript.

C Prompt Templates

  • The student answers with thinking disabled, while the teacher has thinking enabled and, when correcting a failure, writes a structured reflection before solving.Neither role receives a reference solution.Source: C Prompt Templates · p. 15
  • For Math, Physics, and Materials, the teacher reflection names each failure type, re-derives the governing formula, rechecks flawed calculations, and states a concrete correction.Source: C Prompt Templates · pp. 15–16
  • The tool-use student must place the tool-call transcript inside boxed output, with each action paired with a single-line JSON argument object.Source: C Prompt Templates · p. 16
  • For tool use, the teacher checks missing or extra calls, argument values, and action names against prior verified-wrong tool calls before generating a new transcript.Source: C Prompt Templates · p. 16
  • Teacher solutions must stand alone and cannot refer to failed history or reflection content, preventing context-dependent references from entering distillation targets.Source: C Prompt Templates · p. 16

D Generation Samples

The math sample traces how ReTeach turns an incorrect student count into a corrected teacher solution. The reflection identifies omitted pairs involving 1, and the example reports that no external chain-of-thought or gold answer is introduced.

D Generation Samples

  • The teacher response places its native thinking and prompted reflection before the retried solution and final answer.The example describes the native <think> block, <reflect> block, retried solution, and final answer in sequence.Source: D Generation Samples · pp. 16–17
  • The student excludes valid factor pairs involving 1, yielding an incorrect count of three pairs per suit.The reflection identifies 1 × 4 and 1 × 9 as omitted pairs.Source: D Generation Samples · pp. 17, 19
  • The teacher attributes the undercount to missed pairs involving 1, then produces a derivation whose final answer matches ground truth.The example states that no external chain-of-thought or gold answer is introduced.Source: D Generation Samples · pp. 17, 19
  • At distillation time, ReTeach uses final-round thinking and reflection as a masked assistant prefix while the student trains on the solution and answer.If retry remains incorrect, only the solution and boxed answer are carried forward as a failed attempt.Source: D Generation Samples · p. 19
Loading 2610.11529v1…