PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning from Evolving Errors: Adaptive Iterative Repair for On-Policy Distillation

Rui Li, Liyang He, Zheng Zhang, Zhenya Huang, Linbo Zhu, Qi Liu

arXiv:2610.02700v1cs.LGcs.CL

TL;DR

Reference-conditioned OPSD offers token-level feedback but does not specify how to move from a student's current error toward a valid solution. AIR-OPD iteratively generates error-specific repair guidance and uses it for teacher supervision. It improves mathematical-reasoning averages over strong baselines while preserving near-base OOD performance.

  • Problem

    Reference-conditioned OPSD provides token-level feedback from a teacher given a full reference solution, but the solution identifies the target without specifying a transition from the student's current error.

    Source: §1 · pp. 1–2

  • Method

    AIR-OPD updates repair guidance for each failed on-policy retry and uses it as privileged teacher context to supervise an error-aligned response region, with outcome-aware stage weighting.

    Source: §1 · p. 2

  • Results

    Across Qwen3-4B and Qwen3-8B, AIR-OPD improves mathematical-reasoning accuracy by up to 3.6 percentage points over the strongest evaluated baseline while keeping average OOD performance near the base model.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Both self- and external-guidance AIR-OPD variants surpass evaluated mathematical-reasoning baselines while preserving base-model performance on OOD benchmarks.

    Source: §1 · p. 2, §8 · p. 9

    Stage weighting assumes later repair stages have less reliable supervision as failed responses and guidance accumulate in their context.

    Source: §4.3 · pp. 5–6

Abstract

from arXiv · show

On-policy self-distillation (OPSD) supplies dense token-level feedback on trajectories sampled from the student's own policy, a richer training signal than the outcome-level rewards of reinforcement learning. This feedback comes from a teacher conditioned on a full reference solution unavailable to the student. The reference solution specifies the target but not how to move from the student's current error toward it, creating a solution-conditioned shortcut risk. We introduce AIR-OPD, an adaptive iterative repair framework for on-policy distillation that provides error-to-repair supervision. Given a failed response, a guidance generator synthesizes repair guidance for the current error. The student samples an on-policy retry with this guidance. If the retry remains incorrect, the generator produces new repair guidance for the newly observed error. At each round, a fixed teacher receives the guidance as privileged context and supervises the student on an error-aligned region of its latest failed response. Outcome-aware stage weighting favors early repair stages and credits stages whose immediate retry passes verification. We train AIR-OPD on the DAPO-Math-17K dataset and evaluate on AIME24, AIME25, and HMMT25, alongside out-of-distribution tests on MMLU-Pro and GPQA. We examine two guidance sources, self-guidance from the current student policy and external guidance from a larger model. For both Qwen3-4B and Qwen3-8B, AIR-OPD attains the best mathematical-reasoning averages, improving over the strongest baseline by up to 3.6 points, while preserving base-model performance on the out-of-distribution benchmarks.

1 INTRODUCTION

AIR-OPD addresses the gap between knowing a solution and learning how to repair a student's current error. It updates repair guidance across failed retries and improves mathematical-reasoning averages while preserving near-base OOD performance.

1 INTRODUCTION

  • Reference-conditioned OPSD gives dense token-level feedback, but its full reference solution specifies the target without showing how to transition from the student's current error.The teacher's privileged reference may steer the student toward the reference's solution path rather than a generalizable error-correction mechanism.Source: §1 · pp. 1–2
  • AIR-OPD replaces a static reference-conditioned target with repair guidance synthesized for the student's latest error and refreshed after an incorrect retry.The student samples an on-policy retry with the guidance, which the teacher uses as privileged context.Source: §1 · p. 2
  • Training uses DAPO-Math-17K with GPT-5.5 reference solutions and evaluates mathematical reasoning, OOD transfer, and self- versus external-guidance settings.The evaluation includes AIME24, AIME25, HMMT25, MMLU-Pro, and GPQA on Qwen3-4B and Qwen3-8B.Source: §1 · p. 2
  • Its distillation objective supervises an error-aligned region of each failed response and weights repair stages by depth and immediate retry outcome.The guidance generator can use the reference solution, while the student receives the generated guidance rather than the full reference.Source: §1 · p. 2
  • Up to 3.6 percentage points over the strongest evaluated baseline, AIR-OPD improves average mathematical-reasoning accuracy while keeping average OOD performance near the base model.The reported pattern holds across Qwen3-4B and Qwen3-8B with self- and external guidance.Source: §1 · p. 2

2 RELATED WORK

Related work spans outcome-reward reinforcement learning, on-policy distillation, and iterative refinement. AIR-OPD differs by updating repair guidance across failed retries and using it for token-level supervision.

2 RELATED WORK

  • Hint-based RL uses auxiliary guidance to make hard problems more tractable and recover outcome-based reward signals, while AIR-OPD uses repair guidance as privileged context for token-level supervision.The related approaches include fixed hint hierarchies and learned hints conditioned on current failures.Source: §2 · pp. 2–3
  • On-policy distillation scores student-sampled responses, and OPSD uses asymmetric teacher and student contexts; prior failure-focused distillation methods construct corrected trajectories or privileged context in a single pass.AIR-OPD instead focuses on updating privileged context from one failed retry to the next.Source: §2 · p. 3
  • Inference-time refinement uses feedback to revise responses, while training-time methods aim to internalize improvement behavior through verified rationales, revision trajectories, or correction traces.AIR-OPD constructs error-aligned distillation targets on the student's own failed responses using newly synthesized repair guidance.Source: §2 · p. 3

3 PRELIMINARIES: ON-POLICY SELF-DISTILLATION

OPSD provides token-level teacher feedback on responses sampled from the student's own policy. Its teacher can use a reference solution unavailable to the student, but the setup does not explicitly teach transitions from errors to valid solutions.

3 PRELIMINARIES: ON-POLICY SELF-DISTILLATION

  • OPD trains on student-policy states by having a teacher score each token in a response sampled from the student's current policy.The teacher provides dense feedback on prefixes generated by the evolving student.Source: §3 · p. 3
  • OPSD initializes teacher and student from the same model, while the teacher receives the reference solution as privileged input unavailable to the student.The teacher is the frozen initial backbone; both branches score the same student-generated prefixes.Source: §3 · pp. 3–4
  • AIR-OPD's overview replaces the full-reference teacher context with synthesized repair guidance and confines supervision to an error-aligned repair region.A verified retry ends the trajectory, and stage losses are combined using outcome-aware weights.Source: Figure 1 · p. 4
  • The OPSD construction converts the reference into fine-grained token supervision but leaves the teacher to implicitly relate the full solution to the student's current reasoning.It does not explicitly teach how to transition from the current error state to a valid solution state.Source: §3 · p. 4

4 AIR-OPD: ADAPTIVE ITERATIVE REPAIR FOR ON-POLICY DISTILLATION

AIR-OPD makes repair guidance responsive to each failed on-policy attempt, distills over the response region carrying corrective signal, and weights stages by depth and retry outcome.

4 AIR-OPD: ADAPTIVE ITERATIVE REPAIR FOR ON-POLICY DISTILLATION

  • AIR-OPD makes explicit the extraction step left implicit in OPSD: a guidance generator uses the reference solution to write repair guidance for the current error.The three method components update guidance across rounds, select supervised tokens, and combine stage losses.Source: §4 · p. 4

4.1 MULTI-ROUND ADAPTIVE REPAIR

  • The method performs up to K repair rounds, appending each failed response and its repair instruction to the dialogue history before sampling a retry.If the retry is still incorrect, the generator re-localizes the error and produces updated guidance; the process stops at verification success or when the budget is exhausted.Source: §4.1 · pp. 4–5

4.2 ERROR-ALIGNED PRIVILEGED DISTILLATION

  • The frozen teacher receives newly generated guidance to score the latest failed response, while the student is scored under the shared dialogue history without that guidance.Supervision starts at the identified error and extends into the recoverable region, ending at a blank-line boundary that retains at least 1−ϵ of the suffix's corrective signal.Source: §4.2 · p. 5

4.3 OUTCOME-AWARE STAGE WEIGHTING

  • Outcome-aware stage weighting addresses variable trajectory lengths by allocating each supervised trajectory's weight according to stage depth and the immediate retry outcome.The weighting favors earlier stages because later contexts accumulate failed responses and guidance, and gives a bonus to stages whose retry passes verification.Source: §4.3 · pp. 5–6

5 EXPERIMENTAL SETUP

The experiments compare AIR-OPD with reproduced training baselines on two Qwen3 scales, evaluate mathematical reasoning and OOD transfer, and use a fixed OPSD training recipe.

5.1 MODELS, TRAINING DATA, AND BASELINES

  • The study trains Qwen3-4B and Qwen3-8B on DAPO-Math-17K, comparing self-guidance from the student with external guidance from Qwen3.5-27B and three trained baselines.Reference solutions are generated by GPT-5.5; only the 15,650 of 17,398 problems whose solutions verify against dataset answers are retained.Source: §5.1 · p. 6

5.2 EVALUATION BENCHMARKS AND METRICS

  • Mathematical reasoning is measured by unweighted Math Avg. across AIME24, AIME25, and HMMT25 using Avg@12; OOD Avg. averages MMLU-Pro and GPQA accuracy using Avg@4.Each math benchmark contains 30 problems, and evaluation adds no repair guidance to model inputs.Source: §5.2 · p. 6

5.3 IMPLEMENTATION DETAILS

  • Training uses a frozen initial backbone as teacher, rank-64 LoRA adapters, AdamW at 5 × 10−6, and an effective batch size of 32; AIR-OPD runs 100 steps with at most three repair rounds.Each round samples one on-policy rollout; the default stage weights are (γ, ρ) = (0.8, 1.5), and experiments use eight H100 GPUs on one node.Source: §5.3 · pp. 6–7

5.2 EVALUATION BENCHMARKS AND METRICS

  • Table 1 reports mathematical-reasoning Avg@12 and OOD Avg@4, noting that both AIR-OPD variants surpass baselines on Math Avg. while preserving base-model OOD performance.SELF-G and EXTERNAL-G use the current student policy and Qwen3.5-27B, respectively, to generate repair guidance.Source: Table 1 · p. 7

6 MAIN RESULTS

AIR-OPD improves mathematical reasoning over the strongest baseline while maintaining near-base OOD performance. Self-guidance supplies most of the math gain, with external guidance adding further improvement.

6 MAIN RESULTS

  • External-guidance AIR-OPD exceeds OPSD on Math Avg. by 3.6 points at 4B and 2.7 at 8B, while staying within 0.4 and 0.1 OOD points of base, respectively.AIR-OPD variants attain the best scores on every mathematical benchmark at both scales, with pronounced gains on HMMT25.Source: §6 · p. 7
  • Self-guidance improves over OPSD by 2.8 Math Avg. points at 4B and 2.1 at 8B; external guidance adds more, but its additional gain is smaller at 8B.The authors suggest this may be because the larger student generates better repair guidance itself.Source: §6 · p. 7
  • GRPO preserves OOD performance comparably, but its Math Avg. trails AIR-OPD by 2.9–4.4 points; selective error-aligned supervision may limit over-specialization.The proposed explanation contrasts AIR-OPD’s corrective signals on error-aligned regions with scalar RLVR feedback and full-response distillation.Source: §6 · p. 7

7 ABLATIONS AND ANALYSIS

The analysis tests which AIR-OPD components matter, how repair budget affects results, and how training dynamics compare with OPSD and GRPO. The reported gains involve error-aligned supervision, stage weighting, and continued repair, with AIR-OPD leading the baselines throughout training.

7.1 EFFECT OF SAMPLE SELECTION, SUPERVISION REGION, AND STAGE WEIGHTING

  • 62.5 is the Math Avg. for failure-only OPSD, 0.6 points below standard OPSD and 3.3 below AIR-OPD, so selection alone does not explain the gains.The table tests failure-only sample selection and ablations of the supervision region and stage weighting.Source: §7.1 · p. 8, Table 2 · p. 8
  • Widening supervision to the full response costs 1.1 points, while narrowing it to the error block costs 1.6, supporting supervision of both the identified error and downstream reasoning.The authors suspect full-response supervision dilutes the signal and error-block-only supervision omits reasoning that may also need correction.Source: §7.1 · p. 8
  • Either temporal factor or outcome bonus alone improves over uniform stage weighting, and combining both performs best.The authors conclude that weighting should favor earlier stages while also using the immediate retry outcome.Source: §7.1 · p. 8

7.2 EFFECT OF THE REPAIR BUDGET

  • Math Avg. rises through K = 3 and then flattens; increasing the budget beyond K = 3 yields diminishing returns.K = 4 adds only a small improvement, while K = 5 falls slightly below K = 4; the authors suspect accumulated noise in deep-round supervision.Source: Figure 2 · p. 8, §7.2 · p. 9

7.3 TRAINING DYNAMICS

  • Both AIR-OPD variants lead OPSD and GRPO on the three-benchmark average from the first checkpoint, and reach OPSD’s step-100 score by step 40.External guidance matches or exceeds self-guidance throughout; checkpoints are saved every 20 optimizer steps.Source: Figure 3 · p. 9, §7.3 · p. 9

7.2 EFFECT OF THE REPAIR BUDGET

  • At K = 1, AIR-OPD already surpasses OPSD, and its gain over failure-only OPSD is 1.1–1.5 points.The trend rises and then flattens as the budget grows; the supplied figure describes Math Avg. but does not provide its exact values.Source: §7.2 · pp. 8–9
  • At K = 3, the analysis keeps the practical default because it captures most observed gains with fewer supervised stages than larger budgets.Supervised stages rise from 22.2% of the samples OPSD supervises at K = 1 to 35.2–37.4% at K = 5, with shrinking increments.Source: Figure 2 · p. 8, §7.2 · p. 9

8 CONCLUSION

AIR-OPD replaces static reference-conditioned targets with repair guidance that evolves with the student’s errors, and distills each repair over an error-aligned region with outcome-aware stage weighting. Across two model scales, both guidance sources surpass strong RL and distillation baselines on mathematical reasoning while preserving base-model performance out of distribution.

8 CONCLUSION

  • AIR-OPD aligns privileged on-policy supervision with evolving student errors by synthesizing error-specific guidance and updating it after failed retries.The teacher distills each repair over an error-aligned region, while outcome-aware weighting allocates supervision across rounds.Source: §8 · p. 9

A COMPLETE TRAINING ALGORITHM

Algorithm 1 runs bounded on-policy repair rounds, forms a teacher-supervised region around each identified error, and combines stage losses. Guidance is regenerated after failed retries, subject to validation checks.

A COMPLETE TRAINING ALGORITHM

  • Each minibatch samples an initial response, then repeats guidance generation, error-aligned loss computation, dialogue updates, and retries until success, the repair budget, or a stop condition.Stage losses are combined using temporal and immediate-retry outcome weights.Source: A COMPLETE TRAINING ALGORITHM · p. 14

B.1 GUIDANCE GENERATOR INSTRUCTION

  • The generator uses the problem, privileged reference solution, final answer, and latest failed response; from later rounds it also receives previously seen guidance.Figure 4 depicts first-round diagnosis, while Figure 5 depicts guidance replacement after an incorrect retry.Source: §B.1 · p. 14, Figure 4 · p. 15, Figure 5 · p. 16

B.2 REPAIR DIALOGUE

  • The student receives the problem first, then repair guidance after each unsuccessful response and is asked to revise rather than repeat its attempt.From the second feedback turn, the dialogue states that the previous answer is still incorrect.Source: §B.2 · pp. 14–15

B.3 VALIDATION RULES

  • Validation requires a recognized output form, a nonempty error quote located in the latest response, and nonempty guidance that does not state the final answer.Surface-form checks do not guarantee detection of all semantic answer leakage; a failed check or missing repair ends the round without supervision and stops retries.Source: §B.3 · pp. 15–16

C.1 TRAINING SETTINGS

The training setup compares SFT, GRPO, OPSD, and AIR-OPD, using shared reproductions and specifying AIR-OPD’s repair and guidance-generation settings.

C.1 TRAINING SETTINGS

  • Table 3 compares training settings for SFT, GRPO, OPSD, and AIR-OPD; AIR-OPD samples one rollout per dialogue state, with at most K −1 retries.The table caption states that lengths are measured in tokens and the SFT limit applies to the full supervised sequence.Source: §C.1 · p. 16, Table 3 · p. 17
  • Baselines are reproduced on AIR-OPD’s training corpus and backbones; reference filtering leaves 15,650 of 17,398 DAPO-Math-17K problems as training rows.The GRPO reproduction applies no KL penalty, while SFT trains on verified GPT-5.5 reference solutions without on-policy sampling.Source: §C.1 · p. 17
  • AIR-OPD uses at most K = 3 repair rounds, stage weights (γ, ρ) = (0.8, 1.5), and endpoint tolerance ϵ = 0.1.Self-guidance and external guidance from Qwen3.5-27B share inference settings; trained methods also use AdamW, BF16, and FlashAttention-2.Source: §C.1 · p. 17

C.2 INFERENCE SETTINGS

The section specifies mathematical and out-of-distribution evaluation protocols, including sampling, context limits, and answer extraction. Mathematical scores are checkpointed and summarized with a three-benchmark macro-average.

C.2 INFERENCE SETTINGS

  • Mathematical evaluation uses 12 responses per problem at temperature 1.0, top-p 0.95, seed 2026, with thinking enabled and a 40,960-token context limit.Table 4 gives the inference settings shared by the base model and trained methods on AIME24, AIME25, and HMMT25, without repair guidance.Source: §C.2 · p. 17, Table 4 · p. 18
  • Checkpoints are evaluated every 20 steps through 100 steps, and Overall Math Avg. is the macro-average of the three mathematical benchmark scores.Source: §C.2 · p. 17
  • OOD evaluation uses a fixed 1,000-question MMLU-Pro subset and the 198-question GPQA-Diamond split, with four samples per question for each method.MMLU-Pro uses the chat template and accepts an explicit final-answer statement; GPQA omits the chat template, and both read the predicted option from the last boxed answer.Source: §C.2 · pp. 17–18

D ADDITIONAL RESULTS AND EXAMPLES

The worked examples show guidance targeting successive errors and retries that may fail before succeeding. They also illustrate why the outcome-aware weighting rewards verified repairs rather than penalizing every failed retry.

D.1 WORKED REPAIR EXAMPLES

  • A retry can remain incorrect despite valid guidance, so a failed retry is not penalized and only the stage whose guidance leads to verified repair receives the outcome bonus.The examples include a retry that fixes an anchored slip but returns the same wrong answer, and another that adopts a corrected equation before slipping again.Source: §D.1 · p. 18
  • The first worked trajectory corrects the product from 29,170 to 29,070, then targets an omitted mixed-color case before the retry reaches the verified answer 441.The second guidance adds 8,100 favorable outcomes for the case where both picks contain one red and one blue.Source: §D.1 · pp. 18–19
  • In the coin-probability example, the first retry derives the corrected equation but mishandles denominator clearing; later guidance recommends multiplying by 18, after which the retry succeeds.The initial error involved simplifying exponent terms incorrectly when dividing by common factors.Source: §D.1 · pp. 19–20

Related papers

Loading 2610.02700v1…