Prospective Hindsight: Self-Calibrating Reinforcement Learning via Prediction-Reality Gaps

1 Introduction

• Confident mistakes can sustain flawed trajectories and downstream errors, making alignment between an agent’s prospective belief and outcomes important beyond final accuracy.The introduction contrasts confidently wrong agents with uncertain failures that invite fallback, retry, or human input.Source: §1 · p. 2
• Retrospective methods omit action-time beliefs from the gradient, giving identical updates to rollouts with the same outcome despite different confidence or surprise.The paper names this structural obstruction the hindsight trap and links it to accuracy gains without resolving overconfidence.Source: §1 · p. 2
• PH compares a prospective prediction with retrospective evaluation and upweights rollouts where the agent’s self-model is most inaccurate.Its shared policy parameters let the predictor co-evolve, while surprise-weighted optimization is presented as improving calibration without a separate calibration loss.Source: §1 · p. 2
• The paper connects prediction–outcome disagreement to a privileged-information gap and reports PH gains in task performance and calibration across single-turn and multi-turn tasks.The introduction also reports compatibility with GRPO, on-policy distillation, their combination, and consistent gains across model scales.Source: §1 · pp. 2–3

2.1 Agentic Reinforcement Learning

• The setup models an agent taking actions from observed states and receiving verifiable step-level outcomes, while an evaluator may also observe privileged context.The context can be a ground-truth reference in single-step tasks or the next environmental state in multi-turn tasks.Source: §2.1 · p. 3

2.2 Retrospective Training Methods

• GRPO, on-policy distillation, and combined methods construct updates from post-action rewards or context, leaving the agent’s action-time prospective belief out of the learning signal.The base loss is abstracted as a function of retrospective quantities.Source: §2.2 · p. 3

2.3 The Self-Evaluation Blind Spot

• The self-evaluation blind spot gives identical updates to same-outcome rollouts regardless of whether the agent’s belief was calibrated, miscalibrated, or uncertain.This treats lucky successes like confident successes and overconfident failures like aware failures, although the latter can be more dangerous because the agent’s model endorses the rejected action.Source: §2.3 · p. 3

2.4 Why Prediction–Reality Gaps Are Informative

• Outcome variance given action-time information decomposes into irreducible aleatoric noise and a non-negative privileged-information gap that no action-time predictor can resolve.The gap motivates using disagreement between an action-time self-evaluation and the realized outcome as a learning signal rather than treating it as noise.Source: §2.4 · p. 4

3 The Prospective Hindsight Method

• PH combines an action-time self-evaluator, a four-cell rollout taxonomy, and a base-loss reweighting driven by binary surprise.The method presents the same rule as having an implicit-calibration interpretation.Source: §3 · p. 4

3.1 Prospective Self-Evaluator

• The policy also produces a prospective belief by answering whether its action will succeed using only the state and action, without privileged context.The belief is elicited by prompting the policy; multiple samples are aggregated by majority vote, with M extra forward passes per rollout step.Source: §3.1 · p. 4

3.2 A Calibration Taxonomy of Rollouts

• The taxonomy marks confident successes and aware failures as calibrated, while overconfident failures and underconfident successes are miscalibrated.The table indexes cells by prospective belief and realized outcome; neutral predictions are deferred to the appendix.Source: §3.2 · p. 4, Table 1 · p. 4

3.3 The Prospective Hindsight Update Rule

• A rollout is surprising when prospective belief disagrees with outcome; PH stop-gradient-upweights those base-loss terms by 1+α while leaving calibrated rollouts unchanged.The update is agnostic to whether the base method is GRPO, OPD, or their combination, and α=0 recovers the base method.Source: §3.3 · p. 5

3.4 PH as Calibration through Reweighting

• PH’s surprise residual restricts the base loss to miscalibrated cells, but its calibration interpretation is not uniform: at convergence the residual vanishes and PH shares the base method’s fixed points.The text frames alignment with the negative miscalibration gradient as motivation, not a pointwise guarantee; Figure 2 shows PH weights declining as verifier pass rate rises.Source: §3.4 · pp. 5–6, Figure 2 · p. 6

4.1 Experimental Setup

• Experiments use Science Q&A and Tool Use with SDPO, and a GPT-4.1-simulated Personal Agent task with GRPO, OPD, and Combined; main runs use OLMo-3-7B-Instruct as policy and self-evaluator.The prospective predictor uses M=1 sample for single-turn and M=3 for multi-turn.Source: §4.1 · p. 6

4.2 Single-Turn Verifiable Tasks; 4.3 Multi-Turn Personal Agent

• PH preserves or slightly improves single-turn SDPO and improves all three multi-turn baselines, with the largest trajectory-averaged score gain on OPD (+15.3).The multi-turn comparison uses the teacher-side personalization-evaluator score.Source: Table 3 · p. 7, §4.2 · p. 7, §4.3 · p. 8

4.2 Single-Turn Verifiable Tasks

• On Science Q&A, late-training OFR falls from 34.2% with SDPO to 20.9% at α=0.5; on Tool Use, α=1.0 reaches mean@16=58.2/61.7 with OFR 33.6%.On Tool Use at α=1.0, the surprise rate is 33.5%; cross-scale tests report comparable or larger improvements on all three metrics.Source: §4.2 · p. 7, Figure 4 · p. 8
• Random reweighting barely changes Science Q&A OFR (33.3% versus SDPO’s 34.2%), while failure-only reweighting reaches 25.0%, short of PH’s 20.9%.Failure-only captures much of the OFR reduction but trails PH on surprise rate, consistent with underconfident successes being among the cells PH amplifies.Source: §4.2 · p. 7
• On Science Q&A, overconfident failures shrink and confident successes grow over training, while mean PH weight decays as verifier pass rate rises.The paper characterizes PH as a data-controlled curriculum that needs no learning-rate schedule for the surprise weight.Source: §4.2 · p. 8
• Prospective accuracy rises from about 30% to about 80% with PH, versus an SDPO plateau near 73%, although the predictor receives no direct gradient.The α sweep reports simultaneous improvement in capability and calibration, with α=0.5 achieving the lowest mean OFR at matched capability.Source: §4.2 · p. 8

4.3 Multi-Turn Personal Agent

• Single-turn rollouts are dominated by overconfident failures (34.1%), whereas multi-turn underconfident successes rise from 13.8% for RL to 34.5% for Combined, with overconfident failures staying at 4–5%.The two settings therefore expose different dominant miscalibration modes.Source: §4.3 · p. 8, Figure 5 · p. 8
• At α=1.0, OFR falls and prospective accuracy rises across multi-turn blocks, but α=2.0 lowers RL OFR to 2.5% while predictor collapse leaves prospective accuracy at 84.0%.At α=2.0 the surprise rate is 16.0%, indicating that the lower OFR accompanies a degraded self-model rather than better belief–outcome agreement.Source: §4.3 · pp. 8–9

Retrospective training signals in Agentic RL

• Agentic RL methods including process rewards, OPD, and OpenClaw-RL derive learning signals from post-action quantities; PH instead reweights base-method gradients using the gap between prospective prediction and retrospective evaluation [45] [4].PH leaves the base rollout, advantage, and regularization structure intact.Source: §5 · p. 9

Calibration of language-model agents

• Calibration research includes inference-time measurement and uncertainty methods, while calibration-augmented RL adds confidence objectives or edits teacher confidence tokens; these approaches report capability–calibration trade-offs [43].PH uses prospective–outcome disagreement to reweight gradients, with calibration described as a byproduct of task-optimization steps.Source: §5 · p. 9

6 Conclusion

• The conclusion frames PH as a first step toward pipelines that optimize task performance and self-calibration without a separate calibration objective.The proposed scope beyond the studied binary verifiers, per-turn PRMs, GRPO, and on-policy distillation is presented as an expected extension.Source: §6 · p. 9

A Extended Related Work and Discussion

• Retrospective RL signals—including process scores and hindsight-teacher distributions—are post-action, while PH adds per-rollout reweighting without changing the base signal.PH is described as orthogonal to the base method and applicable jointly with GRPO, OPD, and the Combined recipe.Source: A Extended Related Work and Discussion · p. 16
• Unlike calibration approaches that add confidence supervision or alter teacher targets, PH leaves the base method untouched and is reported to avoid their capability–calibration trade-off.Source: A Extended Related Work and Discussion · pp. 16–17
• PH preserves capability while substantially cutting OFR on single-turn tasks and lifts trajectory-averaged and best-checkpoint scores across all three multi-turn base methods.Source: A Extended Related Work and Discussion · p. 17
• PH casts action-time inputs as standard information and post-action context as privileged information, using their evaluation gap to prioritize rollouts where action-time information is insufficient.The paper identifies the surprise indicator as a per-rollout realization of the privileged-information gap.Source: A Extended Related Work and Discussion · p. 17
• PH differs from intrinsic-motivation methods by prioritizing outcome-space prediction errors for training, rather than state-space novelty for exploration or action selection.Its predictor shares policy parameters, and its stop-gradient surprise reweights existing on-policy gradients.Source: A Extended Related Work and Discussion · p. 17
• Unlike inference-time self-correction, PH shapes training and does not change deployment behavior or require confidence reasoning at inference.Source: A Extended Related Work and Discussion · p. 17

• Both terms in the variance decomposition are non-negative; the privileged-information term vanishes exactly when the privileged conditional mean is almost surely a function of the standard inputs.Source: §B.1 · p. 17
• The agent-RL setting maps (st, at) to standard input, yt to target, and ct to privileged signal; PH makes their previously unmeasured gap an explicit training signal.GRPO uses ct through its scalar reward, while OPD-style methods condition the hindsight teacher on ct.Source: §B.1 · p. 18
• The conditional mutual information I(yt; ct | st, at) is zero precisely in the degenerate case where I⋆(st, at) is zero almost everywhere.Source: §B.1 · p. 18

Setup and notation

• The analysis uses a shared autoregressive model for the policy and prospective self-evaluator, with a parameter-independent verifier and non-negative base loss after an additive shift.It restricts the analysis to evaluator outputs z in {−1, +1}; the z = 0 class contributes neutrally.Source: Setup and notation · p. 18

Surprise-weighted decomposition

• The surprise residual is non-negative and vanishes only when surprise and base loss are not simultaneously positive almost surely.Residual minimization can proceed by reducing miscalibration or reducing base loss on miscalibrated rollouts.Source: Surprise-weighted decomposition · pp. 18–19

Exact residual identity

• The residual exactly factors into the conditional mean base loss on surprising rollouts and the miscalibration rate; if that mean loss stays above c0, residual descent bounds the rate from below by c0.The bound is not uniform across the optimization landscape.Source: Exact residual identity · p. 19
• Early in training both loss on surprising rollouts and miscalibration may be large; they decrease together mid-training and both approach zero at convergence, where PH shares the base method’s fixed points.The identity becomes uninformative once both factors are zero.Source: Exact residual identity · p. 19

Connection to the overconfident-failure rate (OFR)

• OFR is a one-sided conditional projection of the full miscalibration rate; it serves as a proxy when confidence remains stable and underconfident successes remain a small fraction of that rate.The stated monotonic relations additionally require that underconfident successes do not increase and positive-confidence predictions do not collapse.Source: Connection to the… · pp. 19–20

Gradient-flow analysis: when is the PH update calibration-aligned?

• When base loss is constant on surprising rollouts, PH differs from the base update by a calibration-aligned component and a base-loss descent component on miscalibrated rollouts.This is the δ(θ) → 0 limit.Source: Gradient-flow analysis: when is the PH… · p. 20
• PH’s update has a nonnegative leading calibration-aligned term, but the base-loss component can have indeterminate sign and the total direction is not exactly aligned with −∇θM(θ).The alignment interpretation is most informative when the relative remainder is small; the authors do not claim monotone descent on M(θ).Source: Gradient-flow analysis: when is the PH… · p. 21

Discussion

• In multi-turn settings, the analysis applies turn by turn and aggregates alignment over the trajectory, while stochastic PRM judging can enlarge the remainder bound.The finite-step neural-network conclusion is approximate under standard learning-rate, smoothness, and Lipschitz assumptions.Source: Discussion · p. 21
• At convergence, PH and the base method share fixed points as c(θ), M(θ), and R(θ) vanish; PH differs in the optimization trajectory before convergence.The paper relates M to the binary Brier-score specialization and says thresholded ECE coincides with it; PH does not directly minimize evaluator NLL.Source: Discussion · pp. 21–22

C.1 Base Method Advantage Formulations

• PH preserves each base method’s internal advantage structure, multiplying its resulting per-rollout or per-turn loss by a surprise-dependent weight.For GRPO, reward normalization remains based on the original rewards; for OPD, the surprise scalar is broadcast across action tokens.Source: §C.1 · p. 22

C.2 PH Algorithm Details

• The training step obtains prospective predictions, computes surprise from prediction and verifier outcome, and weights the base loss, leaving its optimizer and other internal operations unchanged.The appendix also notes α = 0 recovers the base method and M = 1 removes majority voting.Source: §C.2 · pp. 22–23

C.3 Discrete and Continuous PH Gates

• Discrete and continuous gates share weights in [1, 1 + α] that approach 1 as prediction and outcome agree; the continuous gate retains intermediate predictor and PRM-judge uncertainty.The weight curriculum therefore fades as the prediction–reality gap closes, and PH and the base method share fixed points.Source: §C.3 · pp. 23–24

C.4 Honest Uncertainty: The zt = 0 Class

• The prospective evaluator’s zt = 0 denotes honest uncertainty and receives weight 1, so uncertain rollouts are neither amplified nor dampened.Ties and unparseable majority-vote outputs also default to this neutral class; it is excluded from the four-cell taxonomy.Source: §C.4 · p. 24

D Additional Experimental Setup

• The experimental regimes differ in retrospective evaluation: single-turn tasks use a rule-based binary verifier, whereas multi-turn uses an OLMo-3-7B-Instruct PRM judge.The appendix identifies Science Q&A and Tool Use as the single-turn tasks and a personal-agent task as the multi-turn setting.Source: D Additional Experimental Setup · p. 24

D.1 Single-Turn Verifiable Tasks

• Single-turn experiments build on SDPO using the Chemistry subset of SCIKNOWEVAL and Tool Use, with rule-based verification; PH adds a pre-verification prediction call whose stop-gradient weight multiplies each rollout’s reverse-KL loss.Source: §D.1 · pp. 24–25

D.2 Multi-Turn Personal Agent

• The multi-turn personal-agent task uses OpenClaw-RL’s Combined Binary RL plus OPD method, with per-turn PRM scoring and PH predictions made before the judge scores each response.The task alternates Student and Teacher scenarios on GSM8K; evaluation reports capability scores and per-turn OFR, among other metrics.Source: §D.2 · pp. 25–27

D.3 Reweighting-Baseline Protocols

• Random reweighting matches PH’s per-step total weight but selects rollouts uniformly, while failure-only weighting upweights verifier failures rather than PH’s overconfident failures and underconfident successes.These controls isolate selection effects and whether prospective predictions add information beyond verifier outcomes.Source: §D.3 · p. 27

D.4 Compute and Reproducibility

• PH adds about 6% wall-clock overhead in single-turn runs and essentially zero in multi-turn runs, where prediction runs concurrently with PRM scoring.Reported runtimes are approximately 8 hours for Science Q&A, 6 hours for Tool Use, and 4 hours for multi-turn runs.Source: §D.4 · p. 28

E Additional Experiment Results and Analysis

• The additional appendix results expand the main experiments with full α-sweeps, finer-grained training dynamics, and per-turn breakdowns.Configurations match the experimental setup unless stated otherwise.Source: E Additional Experiment Results and… · p. 28

E.1 Single-Turn α-Sweep: SDPO + PH on Science Q&A

• On Science Q&A, α = 0.5 yields the largest absolute OFR drop (0.560), largest relative drop (72.9%), and highest final verifier pass rate (73.2%).Across α values from 0.1 to 2.0, OF shrinks while CS grows; all reported OFR trajectories descend monotonically, and mean PH weight decays toward 1.Source: §E.1 · p. 28

E.2 Multi-Turn α-Sweep: Per-Round Capability and Per-Turn Calibration α

• Later turns begin with higher overconfidence, and PH compresses this region most, bringing α=1.0 late-turn OFR to levels comparable to the baseline’s early-turn OFR.This effectively flattens the per-turn calibration profile, a pattern not surfaced by trajectory-averaged metrics.Source: §E.2 · p. 29, Figure 8 · p. 30
• At α=2.0, per-turn OFR is lowest overall, but capability oscillates and ends below α=1.0 as large weights amplify gradients on long, miscalibrated rollouts.The reported mechanism is up to 1 + α = 3 weighting on miscalibrated rollouts, which tend to be longer token sequences; this was not observed in single-turn training, where rollout lengths are bounded and similar, whereas multi-turn sessions vary up to Tmax=48 turns.Source: §E.2 · p. 29, Figure 8 · p. 30
• The authors recommend α=1.0 as the default and α=0.5 as a more conservative option when capability stability is paramount; no α outside {0.5, 1.0} beats the in-band defaults on both capability and calibration.The per-domain optimum may be α=0.5 for Science Q&A and Combined, or α=1.0 for Tool Use and RL.Source: §E.2 · pp. 29–30
• On Combined with the GPT-4.1 simulator, α=1.0 rises fastest and plateaus highest in capability, while α=0.5 tracks closely and α=2.0 has the greatest round-to-round variance.The plotted capability trajectory reproduces the headline of Table 3.Source: §E.2 · p. 28, Figure 8 · p. 30

F.2 Multi-Turn Tasks

• PH uses ternary scores in {−1, 0, +1}: the prospective self-evaluator predicts before the next state is observed, while the retrospective PRM scores the response after that state is revealed.The PRM serves as verifier R; the prospective prompt receives the latest user message and response, without access to the next state.Source: F Prompt Templates · p. 30, §F.2 · pp. 32–33

F Prompt Templates

• The appendix lists prompt templates for PH’s prospective self-evaluator and the multi-turn retrospective PRM, with scores parsed from a boxed field and majority-vote ties broken toward 0.The templates elicit scores z_t or r_t in {−1, 0, +1}.Source: F Prompt Templates · p. 30

F.1 Single-Step Tasks

• For single-step tasks, the prospective self-evaluator routes to a task-specific template by dataset type and applies a shared ternary scoring instruction to the problem and emitted action.Routing covers math, multiple-choice, tool-use, code, and a default fallback.Source: §F.1 · p. 30, F Prompt Templates · p. 30

E.2 Multi-Turn α-Sweep: Per-Round Capability and Per-Turn Calibration α

• Table 6 summarizes the full α-sweep using OFR, Mean Weight, and Verifier Pass Rate.It defines OFR as OF/(CS+OF) and reports Mean Weight and Verifier Pass Rate as early → late.Source: Table 6 · p. 36
• Early and late results correspond to the first and last five training steps, respectively.The caption specifies this window for the Early/Late reporting.Source: Table 6 · p. 36

G Limitations and Future Work

• High α (≥2.0) can spike gradient norms in long-horizon tasks, while the calibration-alignment analysis assumes the base loss is approximately constant on surprising rollouts.The authors describe the practical alignment as quantitative rather than absolute because per-rollout loss varies with log-probabilities.Source: G Limitations and Future Work · p. 37
• PH adds M forward passes per rollout step and may produce noisier signals early when the base model is extremely poorly calibrated.The reported overhead is approximately 6% in single-turn synchronous training and negligible in asynchronous multi-turn settings.Source: G Limitations and Future Work · p. 37
• Empirical evaluation covers verifiable Science Q&A and Tool Use plus a math-tutoring personal agent; performance on creative or subjective tasks remains unexplored.The paper reports cross-scale transfer from 7B to 20B models.Source: G Limitations and Future Work · p. 37
• Future work includes evaluating the prospective evaluator independently and studying PH on agent trajectories longer than the experiments’ Tmax = 48 turns.The authors identify sparse, delayed feedback and changing per-turn calibration-cell composition as open questions at longer horizons.Source: G Limitations and Future Work · p. 37
• The authors propose exploring richer continuous-valued PRM signals and whether training-time PH composes constructively with inference-time self-correction.The current surprise indicator is binary or a half-distance in [0, 1]; composition with methods such as Reflexion and Self-Refine remains open.Source: G Limitations and Future Work · p. 37

H Broader Impact

• The paper presents improved reliability, human-AI trust, and reduced need for separate calibration training as potential benefits of PH.These are framed as potential societal impacts, including lower computational and human-labor barriers.Source: H Broader Impact · p. 38
• The authors flag risks that agents could avoid difficult tasks to appear calibrated, or that PH could amplify errors in biased or incorrect verifiers and PRMs.They also note potential dual-use concerns from greater agent autonomy and tool-use capability.Source: H Broader Impact · p. 38