Source-linked AI summary
Failure Is the Mother of Success: Recovering Supervision with Grounding Hindsight
Sheldon Yu, Tong Yu, Nikki Lijing Kuang, Zhendong Chu, Zhenwei Tang, Sungchul Kim, Lina Yao, Julian McAuley, Junda Wu
TL;DR
When every rollout fails, GRPO receives no group-relative learning signal, leaving hard problems underused. RSHD validates corrective notes through fresh replays and distills admitted successes into the plain-prompt policy. It consistently outperforms GRPO across model scales and mathematics and science domains, with gains carrying out of domain.
Problem
Method
RSHD creates corrective notes from failed attempts without reference-answer access, admits notes whose fresh replays outperform the original group, and distills successful replays into the plain-prompt policy.
Source: §1 · p. 2Results
RSHD consistently outperforms GRPO across model families and sizes and mathematics and science domains, with gains on harder problems and out-of-domain benchmarks; it reaches 84.5 on MATH-500 against 76.9 for GRPO at 4B.
Source: §1 · p. 2, §6.1 · p. 9, §7 · p. 11Takeaways & Limitations
Verified lessons from failed rollouts can provide useful supervision, including for all-fail groups, without adding inference-time guidance.
Source: §1 · p. 2, §7 · p. 11On all-fail groups, a note can pass the replay test by chance, so admission does not prove population uplift and selected targets can still be incorrect.
Source: §5 · p. 8
Abstract
from arXiv · showhide
When every rollout for a problem fails, GRPO receives neither a successful exemplar nor a nonzero group-relative advantage, so it learns nothing from exactly the problems it most needs to learn. The failed reasoning can nevertheless suggest what to try next. We present Replay-Selected Hindsight Distillation (RSHD), which turns failed attempts into an additional, tested source of supervision during GRPO training. The policy writes a corrective note about each failed attempt without seeing a reference answer or solution, then re-solves the problem with the note in context. A note is admitted only if its replays succeed more often than the original group did, after screening for notes that state the answer, and an admitted replay is distilled into the plain-prompt policy. Because admission is decided by fresh successes rather than by the failed rollout itself, the procedure can recover supervision from groups in which every attempt failed, and it applies equally to failures in mixed groups. Only the policy is trained, the only judge is the outcome verifier, and deployment requires no notes or extra inference. Across model families and sizes and two training domains, RSHD consistently improves over GRPO on in-domain benchmarks, and the gains carry over to out-of-domain evaluation sets.
1 INTRODUCTION
All-fail groups give GRPO no task-reward gradient, even though failed trajectories may reveal mistakes; this is common on hard problems. RSHD uses verified, failure-conditioned replays to turn some of those failures into supervision.
1 INTRODUCTION
- When a problem has a 0.1 solve probability, an eight-sample group is all-fail more than 40% of the time; roughly half of hardest-level MATH groups carry no gradient.All-fail groups have zero group-relative advantages, despite potentially informative failed reasoning.Source: §1 · p. 1
- Failure-based supervision must avoid reference-answer leakage and unverified self-feedback, which can reinforce incorrect diagnoses or mistaken tokens.The proposed approach instead verifies whether a note helps through fresh exploration before transferring behavior.Source: §1 · p. 2
- RSHD writes a corrective note from each failed attempt without reference-answer access, then retains it only when fresh replays outperform the original group.Successful replays are distilled into the plain-prompt policy, allowing supervision from all-fail groups.Source: §1 · p. 2
- In the math run, one note in nine states the answer outright and three in five fail the replay test, motivating both screening and outcome-based admission.The already-sampled GRPO group supplies the baseline for the replay test.Source: §1 · p. 2
- RSHD improves over GRPO across model families and sizes and two training domains, with gains on harder problems and out-of-domain evaluation.The paper also reports shorter solutions and fewer budget overruns.Source: §1 · p. 2
2 RELATED WORK
Prior approaches vary retries through sampling or distill behavior from privileged context, but often keep the sampled trajectory fixed. RSHD tests note-conditioned behavior through fresh outcomes and transfers selected continuations separately from GRPO.
2 RELATED WORK
- GRPO and related methods rely on sampling-based exploration; higher temperature, wider nucleus sampling, and entropy-preserving updates diversify retries without using why earlier attempts failed.When every rollout earns the same reward, group-relative advantages vanish and the prompt contributes nothing.Source: §2 · pp. 2–3
- Prior hindsight methods infer goals or guidance from model text, making leakage and trust concerns salient; RSHD instead admits a note by comparing fresh replays with the sampled group.The writer sees neither the reference answer nor a solution, and the original group retains its advantages.Source: §2 · p. 3
- Prior privileged-context distillation uses reference answers, peer completions, trajectory summaries, or hindsight skills, typically keeping the sampled trajectory fixed and admitting context on sight.RSHD changes both the test and the target: it selects a fresh continuation by outcome before weighted plain-prompt imitation.Source: §2 · p. 3
- RSHD’s selected replay transfer is separate from GRPO’s reward normalization and uses a continuation sampled under a note, rather than standard student-on-policy distillation.The overview shows failed attempts producing notes, replay-based admission, and plain-prompt imitation added to the GRPO loss.Source: §2 · p. 3, Figure 2 · p. 3
3 PROBLEM FORMULATION
The formulation models verifier-scored completions and group-relative updates, then identifies why all-fail groups yield no GRPO gradient. It also defines the privileged-context distillation setup used to transfer fresh continuations to the plain-prompt policy.
3 PROBLEM FORMULATION
- The verifier returns a binary reward by checking a completion’s final answer against the reference, while the policy does not receive the reference answer.The finite-horizon verifier MDP represents the problem, generated prefix, and remaining completion budget as state.Source: §3 · p. 4
- GRPO shares one rollout-level statistic across a completion’s tokens, so failed completions reveal neither where reasoning went wrong nor how to correct it.When all group rewards match, the advantages vanish and the group supplies no task-reward gradient.Source: §3 · p. 4
- The distillation objective weights teacher–student token scores; in RSHD, the teacher scores a fresh continuation under privileged context and only the plain-prompt student is updated.Teacher scores are stop-gradient, and the student conditions on the original problem without the privileged context.Source: §3 · p. 4
4 METHOD: REPLAY-SELECTED HINDSIGHT DISTILLATION
RSHD uses failed attempts to generate corrective notes, tests those notes through fresh replays, and distills selected replay behavior into the plain-prompt policy alongside GRPO. Its objective credits replay tokens according to the note’s positive, clipped effect on their probability, while note admission screens for answer leakage and requires replays to outperform the original group.
4 METHOD: REPLAY-SELECTED HINDSIGHT DISTILLATION
- RSHD admits a note only when fresh replays with it outperform the original group, then distills an admitted replay into the plain-prompt policy alongside unchanged GRPO advantages.The chosen replay may be unsuccessful; admission enriches correctness but does not guarantee it.Source: §4 · p. 5, §4.1 · p. 5, §4.2 · p. 6
4.1 THE REPLAY-SELECTED HINDSIGHT DISTILLATION (RSHD) OBJECTIVE
- The auxiliary objective credits a replay rather than the failed rollout, weighting replay tokens by their detached, clipped positive probability lift under the note.This concentrates credit on tokens the note changed and gives no weight where it did not.Source: §4.1 · p. 5
- When all original attempts fail, an admitted replay with positive weight supplies a nonzero gradient; with no admitted note, training falls back exactly to GRPO.The note appears only in the teacher prompt, so deployment uses the plain prompt without extra inference.Source: §4.1 · p. 5
4.2 REASONING EXPLORATION AND NOTE ADMISSION
- For failed attempts in mixed or all-fail groups, the policy writes a corrective note without seeing the reference answer or a solution; fully solved groups are skipped.Notes must parse and not state the reference answer before they are replay-tested.Source: §4.2 · pp. 5–6
- A note that fails its first replay test can be revised once using its tally and up to three failed-replay excerpts, then screened and tested again.The admission test compares fresh replay outcomes against the original group.Source: §4.2 · p. 6
- Admission enriches rather than guarantees correctness: for an all-fail group, at least one replay must succeed, but the uniformly sampled admitted replay may still be wrong.The answer-leak screen is applied before admission because answer-revealing notes could pass the replay test trivially.Source: §4.2 · p. 6
5 FAILURE-CONDITIONED EXPLORATION AND TRANSFER
All-fail groups provide no task-reward gradient, while local repairs may remain trapped in an unsuccessful strategy. RSHD uses failure-derived contexts to test fresh replays and transfer selected behavior to the plain-prompt policy.
5 FAILURE-CONDITIONED EXPLORATION AND TRANSFER
- All-fail groups have zero group-relative advantages and no task-reward gradient, although KL regularization and other problems can still supply gradients.This is a finite-sample loss of reward signal.Source: §5 · p. 6
- A repair kernel with escape bound ϵ cannot succeed in cases where the successful strategy lies outside its reachable repair basin.Prefix- or strategy-preserving repair has ϵ = 0 in the stated case; restarting with failure-derived context can explore another strategy.Source: §5 · pp. 6–7
- After a failure, the policy writes context and generates fresh attempts from the original problem; escape rate measures strategy change, not success.These replays are on-policy for the snapshot on the augmented prompt, not for the deployed plain-prompt policy.Source: §5 · p. 7
- A context is admitted when at least one of k independent replays succeeds, and an accepted replay is selected uniformly.For replay success probability ps > 0, admission probability is 1 − (1 − ps)^k, and the selected-replay success rate is at least ps.Source: §5 · p. 7
- Admission does not establish that a note improves on the original success rate: a neutral note can pass by chance, and selected targets may still be incorrect.Population uplift remains an empirical premise.Source: §5 · p. 8
- RSHD leaves GRPO advantages unchanged and adds an auxiliary that imitates a selected replay without its note; absent an admitted replay, the update falls back to GRPO.A conditional non-degradation result requires the auxiliary policy to stay close to a mixture of GRPO trajectories and higher-value replay targets; the stated loss does not certify that closeness.Source: §5 · p. 8
- A two-strategy witness finds plain-policy success never decreases on a local step and increases strictly in expectation, conditional on admission.The witness depends on credit going to the successful root decision and that logit being unaffected by other trajectories; with shared parameters or multiple decisions, positive guided success alone does not guarantee improvement.Source: §5 · p. 8
- A two-strategy illustration shows the strategy-switch note learning fastest at H = 5 and H = 8, while unguided retry reaches 0.92 against 0.94 at H = 8.The note effects are fixed by the model, so the figure illustrates helpful notes rather than showing that real notes help.Source: Figure 3 · p. 7, §5 · p. 8
6 EXPERIMENTS
RSHD improves over GRPO across the reported benchmarks and backbones, with gains on hard problems and out-of-domain tests. The experiments also examine training dynamics and whether gains concentrate on problems where GRPO lacks signal.
6 EXPERIMENTS
- The study trains Qwen3-1.7B, 4B, and 8B on MATH or numeric MMLU-Pro questions and evaluates accuracy over eight samples per problem.Science evaluation includes held-out pool questions, GPQA-Diamond, and TheoremQA; the latter two are disjoint from the training pool.Source: §6 · pp. 8–9
6.1 MAIN RESULTS
- 84.5 versus 76.9 on MATH-500 at 4B, and 80.6 versus 75.3 at 8B, with RSHD ahead of GRPO in both comparisons.The trained rows share the same GRPO rollouts and 150-step budget.Source: Table 1 · p. 9, §6.1 · p. 9
6 EXPERIMENTS
- The comparison includes OPSD and SEED; RSHD tests notes before use and distills note-conditioned continuations rather than re-scoring the original rollout.All trained rows share the GRPO backbone, rollouts, and step budget, differing only in the added loss.Source: §6 · p. 9
6.1 MAIN RESULTS
- On science evaluations, at 4B RSHD scores 61.2 versus 48.7 on the held-out training-pool slice and 56.5 versus 40.1 on TheoremQA, using accuracy averaged over eight samples per problem.Source: §6.1 · p. 9, §6 · p. 9
- Figure 4 shows RSHD separating from GRPO by step 50 across backbones, with the widest gap on science, where GRPO plateaus early.Its panels use individually fitted y-ranges, so gaps should be compared within, not across, panels.Source: Figure 4 · p. 10, §6.1 · p. 10
- Science-trained RSHD models also improve on MATH-500: 81.3 versus 74.2 at 4B and 77.5 versus 70.7 at 8B.These models were not trained on MATH.Source: §6 · p. 9, §6.1 · p. 10
6.2 ABLATIONS
- On Qwen3-4B, the RSHD–GRPO margin rises with MATH difficulty, from equal 94.5 accuracy at Level 1 to a 10.8-point gain at Level 5.The difficulty labels predate this work and are not used in training or note admission.Source: §6.2 · p. 10, Figure 5 · p. 10
- On Gemma-4-E4B’s 66 never-solved problems, RSHD scores 15.0 versus GRPO’s 6.4; on the other 434 problems, the models are within half a point.The paired-bootstrap gap on the never-solved subset is +8.5 points, with a 95% interval of [+4.5, +12.9].Source: §6.2 · p. 11
7 CONCLUSION
The conclusion presents RSHD as a way to recover supervision when all GRPO rollouts fail, by validating corrective notes through replays before distillation. It reports consistent gains across model scales and mathematics and science domains, including larger gains on harder problems and no extra inference-time overhead.
7 CONCLUSION
- RSHD improves over GRPO across model scales and both mathematics and science domains, with the gain concentrated on problems where the released Gemma-4-E4B solves none of eight samples.Table 2’s caption states that the gain on MATH-500 lies entirely in that never-solved subset.Source: §7 · p. 11, Table 2 · p. 11
- RSHD turns failed rollouts into corrective notes, validates them with fresh replays, and distills only guidance that improves outcomes.Source: §7 · p. 11
USE OF LARGE LANGUAGE MODELS
- The paper reports using generative AI to implement analysis code, provide feedback on experimental design and result interpretation, and help draft the paper and figures.Source: USE OF LARGE LANGUAGE MODELS · p. 11
A PROMPTS
The appendix specifies prompts for writing corrective notes, solving with notes, revising notes after failed replays, and evaluating free-form answers. It also distinguishes training prompts from an answer-matching prompt used only in evaluation.
A PROMPTS
- The solver receives a hindsight note only in replay or teacher passes; the student and deployed model use the plain prompt.The science solve prompt adds an instruction to put numeric answers in the box without units.Source: A PROMPTS · p. 15
- The note writer sees an incorrect attempt but no reference answer or solution, and is asked to name the core mistake and advise what to do instead.Notes are constrained to instructions for the solver rather than descriptions of what happened.Source: A PROMPTS · pp. 15–16
- After a note’s replays fail to beat the group’s no-note rate, the revision prompt uses up to three failed replays to revise its advice.The prompt reports successes out of total new attempts and includes excerpts from failures.Source: A PROMPTS · p. 16
- For two free-form test sets, an answer-matching prompt lets the evaluated model judge rejected answers, and is never used in training.The prompt asks whether candidate and reference answers mean the same thing despite wording or formatting differences.Source: A PROMPTS · p. 16
B IMPLEMENTATION DETAILS
RSHD adds note search and replay distillation to GRPO training while retaining TRL’s rollout, synchronization, and loss machinery. The section also specifies the replay screens and evaluation protocol.
B IMPLEMENTATION DETAILS
- RSHD adds note search after batch scoring and broadcasts each kept note and one replay per failed rollout to its owning process.TRL handles generation, weight synchronization, distributed training, checkpoints, and the GRPO loss; the shared vLLM server uses current policy weights.Source: B IMPLEMENTATION DETAILS · p. 17
- For kept notes, replay tokens receive clipped positive teacher–student log-probability-gap weights in an auxiliary loss added to GRPO with coefficient λ.The note-prompt teacher pass is gradient-free; the plain-prompt student pass supplies gradients, and batches without kept notes use plain GRPO.Source: B IMPLEMENTATION DETAILS · p. 17
- The GRPO-only baselines differ from RSHD only in starred registered settings, and no model besides the policy is fit.Runs use seven training GPUs and a vLLM server on the eighth.Source: Table 3 · p. 17
- Three notes in four are discarded across 150 Qwen3-4B MATH training steps, and the two screens require no replays.Each step contains 14 problems and 112 rollouts.Source: Table 4 · p. 18
- The note writer uses a 2,048-token budget, replays use rollout sampling settings, and notes are revised once if the keep rule initially fails.Notes are discarded if no JSON note is produced; admission compares replay success with the no-note rate across all group rollouts.Source: B IMPLEMENTATION DETAILS · p. 18
- Evaluation uses eight samples per problem and exact set-specific verifiers, with model-matched scores also reported for rejected free-form science answers.The evaluation uses temperature 0.6, top-p 0.95, and a 4,096-token limit.Source: B IMPLEMENTATION DETAILS · p. 18
C NOTATION SUMMARY
The notation defines verifier rewards and group-level failure, then names the policy, trajectory, note, replay, and distillation quantities used in the formal analysis.
C NOTATION SUMMARY
- R(x, y) is the binary verifier reward for completion y, and F is the set of rollout indices with zero reward.The group’s no-note success rate is the average verifier reward across its n rollouts.Source: C NOTATION SUMMARY · p. 19
- πθ denotes the trained policy, while πθold is the frozen snapshot that sampled the group.The notation also names the GRPO group as the n rollouts for a prompt.Source: C NOTATION SUMMARY · p. 19
- A note s is inserted into the solve prompt as x ⊕ s, and the resulting note-conditioned continuations are the replays.A note is kept when its replay success rate exceeds the group’s no-note success rate; one replay is chosen uniformly for distillation.Source: C NOTATION SUMMARY · p. 19
- The distillation term uses a stop-gradient teacher and clipped positive per-token log-probability lift, then combines with the GRPO loss using coefficient λ.The notation defines the credited replay-token set and the GRPO and RSHD losses.Source: C NOTATION SUMMARY · p. 19
D FORMAL SETUP, PROOFS, AND BOUNDARY CASES
The appendix formalizes the verifier-MDP analysis and its boundary cases. It distinguishes guarantees about hinted-policy success from what the training loss establishes for the plain-prompt student.
D FORMAL SETUP, PROOFS, AND BOUNDARY CASES
- On an all-fail group, the reward mean and standard deviation are zero, so every GRPO advantage and task-reward surrogate term vanish.This claim excludes KL regularization and gradients from other prompts.Source: D FORMAL SETUP, PROOFS, AND BOUNDARY… · p. 19
- If each repair retry succeeds with conditional probability at most ϵ, then m retries succeed with probability at most mϵ, without requiring retry independence.The bound concerns repair kernels that preserve the constrained strategy; unconstrained restarts fall outside that hypothesis.Source: D FORMAL SETUP, PROOFS, AND BOUNDARY… · p. 19
- For a fixed note with replay success probability p_s, conditioning on admission yields replay success probability p_s/[1−(1−p_s)^k], at least p_s.The result follows because admission selects batches containing at least one successful replay.Source: D FORMAL SETUP, PROOFS, AND BOUNDARY… · p. 19
- Admission enriches accepted replays for success but does not certify note correctness; a uniformly chosen accepted replay is not guaranteed correct when k > 1.Admission can occur even when the note’s success probability equals the plain-retry rate, if the original group happens to fail.Source: D FORMAL SETUP, PROOFS, AND BOUNDARY… · p. 20
- A better hinted policy does not by itself establish nonnegative advantages for a plain-prompt student.The finite-horizon performance-difference identity supplies the condition underlying non-degradation.Source: D FORMAL SETUP, PROOFS, AND BOUNDARY… · p. 20
- Distributional distillation can establish the transfer premise only if it matches mixture conditionals on mixture prefixes; the selected-token loss alone does not certify the needed KL bound.The theorem concerns one problem and one local update, not an entire shared-parameter run.Source: D FORMAL SETUP, PROOFS, AND BOUNDARY… · p. 20
- A selected positive replay token raises its policy logit under the auxiliary gradient, while a selected zero-weight token leaves that logit unchanged.The proposition combines this update with positive probability of selecting a successful replay.Source: D FORMAL SETUP, PROOFS, AND BOUNDARY… · p. 20