Source-linked AI summary
Where the Model Changes Its Mind: Hindsight-Divergence Localization for Efficient Reinforcement Learning with Verifiable Rewards
Fanchao Chen, Hengyu Fu, Shivaram Venkataraman, Jiantao Jiao
TL;DR
Group-relative RLVR relies on multiple sampled trajectories, making rollout generation costly and leaving alternative decisions unexplored. HDL uses hindsight-induced token-likelihood changes to choose branch points and reuse prefixes. Across math, code, and agent tasks, it reduces rollout cost and improves performance relative to GRPO.
Problem
Group-relative RLVR requires multiple trajectories per prompt, making rollout generation costly, while complete-trajectory sampling does not target alternative continuations at critical decisions.
Source: §1 · p. 1Method
HDL uses hindsight-induced changes in token log-likelihood to select branch points, then fills groups with continuations that reuse root prefixes and update only newly generated suffixes.
Source: §1 · pp. 2–3, §3.2 · p. 4Results
Across three models and math, code, and agent tasks, HDL improves performance across all domains while reducing generated tokens by 35–61% and rollout wall-clock time by 18–45% relative to GRPO.
Source: §5 · p. 11Takeaways & Limitations
HDL allocates additional exploration and learning to decisions reconsidered after feedback while reusing prefixes and retaining group-relative training.
Source: §1 · p. 3, §5 · p. 11HDL depends on reliable interpretation of verifier feedback; Qwen3-1.7B outcome agreement is 76.0% on Math and 56.1% on Code, and its ScienceWorld advantage vanishes.
Source: A. Limitations · p. 14
Abstract
from arXiv · showhide
Group-relative methods for reinforcement learning with verifiable rewards (RLVR) learn from differences in rollout outcomes. Independently sampling complete trajectories is costly and does not explicitly explore the decision space at critical positions. Feedback on a completed trajectory can reveal which earlier choices the policy reconsiders, suggesting where to sample alternative continuations. We introduce Hindsight-Divergence Localization (HDL), which uses hindsight-induced changes in token log-likelihoods to select branch points. HDL generates a small number of complete root trajectories and fills each training group with continuations from the selected positions under the original task context. Each continuation reuses its root prefix and contributes policy updates only through its newly generated suffix, reducing generation cost while focusing additional exploration and learning on decisions after branching. Experiments with three models across math, code, and agent tasks show gains in both rollout efficiency and task performance. Compared with GRPO at matched group sizes and training steps, HDL yields up to a 2.5$\times$ reduction in generated tokens and a 1.8$\times$ speedup in rollout wall-clock time. Despite this reduced generation budget, HDL improves performance across all three domains, with gains of up to 12.5 points on agent tasks.
1. Introduction
Group-relative RLVR depends on multiple trajectories per prompt, making rollout generation costly; HDL uses feedback-induced changes in token likelihoods to select where to branch. Across math, code, and agent tasks, HDL improves performance while reducing generation cost relative to GRPO.
1. Introduction
- Group-relative methods require multiple trajectories for each prompt, making rollout generation a dominant training cost that grows with long reasoning traces and agentic tasks.Asynchronous systems improve throughput without changing complete-trajectory sampling as the group-relative sampling unit.Source: §1 · p. 1
- Prior hindsight comparisons suggest that feedback can reveal which earlier choices the model reconsiders, motivating branch-point selection by changes in token likelihood.The cited work on token-level supervision compares predictions with and without hindsight; HDL applies that comparison to branch selection.Source: §1 · p. 2
- HDL scores sampled tokens with and without a hindsight context containing verifier feedback and a reflection, then selects positions with the largest absolute log-likelihood changes.The coding example selects “for” after reflection identifies a missing x<2 guard.Source: §1 · pp. 2–3, Figure 1 · p. 2
- HDL fills groups with fresh continuations that reuse a root prefix, sample under the original task context, and apply policy updates only to newly generated suffixes.Hindsight informs branch selection, while the continuations’ verified outcomes provide feedback on alternatives from the same history.Source: §1 · p. 3
- HDL improves performance across math, code, and agent tasks, with gains reaching 12.5 points on agent tasks, while halving rollout generation budget relative to GRPO, cutting tokens by up to 61% and wall-clock time by up to 45%.Source: §1 · p. 3
2. Related work
Prior RLVR efficiency work changes rollout scheduling or selectively uses generated data, while branching methods reuse prefixes to explore intermediate states. HDL instead ranks branch points by hindsight-induced changes in sampled-token log-likelihoods.
Efficient RLVR
- HDL reduces generation through prefix reuse while preserving group size and the RL objective, contrasting with methods that improve rollout throughput or filter generated data.AReaL and Kimi k1.5 decouple or carry rollouts across updates; DAPO filters uniform-reward groups, and token-selective RL still generates complete trajectories.Source: §2 · p. 3
Branch-point selection
- Unlike uncertainty-guided branching or reflection-guided retries, HDL derives branch points from changes in sampled-token log-likelihood and samples continuations under the original rollout context.The comparison includes TreeRL, BPO, InfoTree, PivotRL, PivoARL, and R3L.Source: §2 · p. 3
On-policy self-distillation
- On-policy self-distillation methods use additional information to supervise sampled trajectories, with some methods targeting particular decisions or action spans.Examples include SDPO, RLSD, SRPO, SD-Search, HINT-SD, and HSD.Source: §2 · pp. 3–4
- HDL compares token likelihoods with and without hindsight to rank branch points, rather than using the difference for token-level supervision.This distinguishes its use of hindsight from the token-supervision approach described for related self-distillation methods.Source: §2 · p. 4
3. Hindsight-Divergence Localization
HDL uses hindsight-conditioned changes in token likelihood to select branch points, then forms group-relative training groups from complete roots and continuations. Reused prefixes reduce duplicated generation and continuation losses focus on newly sampled suffixes.
3. Hindsight-Divergence Localization
- HDL samples M<G complete roots, selects branch points using verifier feedback and hindsight reflections, and fills the remaining group slots with continuations.The roots and continuations together form one group for training.Source: §3 · p. 4
3.1. Hindsight-conditioned scoring
- Hindsight-conditioned scoring compares predictions for each recorded root token using the same policy and root prefix, with only the hindsight context differing.That context combines verifier feedback with a reflection interpreting the outcome in relation to earlier decisions.Source: §3.1 · p. 4
3.2. Branch-point selection
- The hindsight-divergence score is the absolute change in a sampled token’s log-likelihood, capturing either increased or decreased support after the outcome is known.HDL ranks candidate positions within each root by this score and selects the highest-scoring positions as branch points.Source: §3.2 · p. 4
3.3. Localized group construction
- At each selected position, HDL reuses the root prefix and samples a fresh suffix under the original task context to form a complete continuation.The number of roots and allocation of continuations are specified in Section 4.1, while alternative branching configurations are compared in Section 4.5.Source: §3.3 · p. 5
- Roots and continuations share the GRPO objective and group-relative rewards, but continuation policy loss applies only to newly sampled tokens.Excluding reused prefixes from continuation loss avoids counting them again and focuses learning on decisions explored after branching.Source: §3.3 · p. 5
4. Evaluation
Across Math, Code, and Agent tasks, HDL reduces rollout costs while improving task performance, with the largest reported gains on Agent tasks. Comparisons also favor HDL over alternative localization signals and branching configurations in the reported settings.
4.1. Experimental setup
- Evaluation covers verifiable-reward Math, Code, and Agent tasks; HDL uses two complete roots and allocates continuations to each root's two highest-scoring branch points.The group contains 16 trajectories, matching the comparison methods' group size; evaluation includes six math benchmarks, LiveCodeBench v5 and v6, and held-out ScienceWorld variations.Source: §4.1 · pp. 5–6
4.2. Rollout efficiency
- Across all three models and domains, HDL cuts generated tokens by 35–61% and rollout time by 18–45% relative to GRPO.HDL's rollout speedups over GRPO are 1.22×–1.83×; generated-token reductions versus DAPO are 43–75%.Source: §4.2 · p. 7, Figure 2 · p. 7, Figure 3 · p. 7
4.3. Downstream task performance
- HDL improves performance across Math and Code and gains up to 12.46 points over GRPO on Qwen3-8B ScienceWorld (71.96% vs 59.50%).On Qwen3-8B Math, HDL's average score is 54.16%, versus 53.17% for GRPO and 53.47% for DAPO; it also records the highest Code accuracy on both Qwen3 models.Source: §4.3 · p. 8, Table 1 · p. 8, Table 2 · p. 8
4.4. Comparison of localization signals
- On Qwen3-8B, HDL scores highest across all three domains, exceeding Entropy and Reflection on Agent by 5.67 and 6.54 points, respectively.Entropy averages 3.22 branch points per group versus HDL's 3.95, out of a maximum of four.Source: §4.4 · pp. 9–10, Table 4 · p. 9
- Entropy selects opening verbs at similar rates in successful and failed roots (80.8% and 83.7%), while HDL selects arguments more often in failed than successful roots (52.4% vs. 32.7%).Across training, Entropy's selected-position mean falls from 1.03 to 0.57 and its verb TVD from 0.278 to 0.246; HDL's divergence rises from 6.85 to 7.78 and its verb TVD from 0.329 to 0.348.Source: §4.4 · pp. 9–10, Figure 5 · p. 9, Figure 6 · p. 10
4.5. Comparison of branching configurations
- For Qwen3-8B Agent, the default 2 × 2 configuration scores 71.96%; 2 × 1 reduces rollout time by 15% with a 0.87-point score decrease, while 4 × 2 scores 4.09 points lower.The comparisons favor multiple branch points per root while retaining several continuations per position.Source: §4.5 · pp. 10–11, Table 5 · p. 10
5. Conclusion
HDL allocates rollout generation to decisions reconsidered after feedback, using hindsight divergence to choose branch points and prefix-reusing continuations. Across three domains, it reduces rollout costs while improving task performance.
5. Conclusion
- HDL reduces generated tokens by 35–61% and rollout wall-clock time by 18–45% relative to GRPO while improving performance across math, code, and agent tasks.Agent-task gains reach up to 12.5 percentage points.Source: §5 · p. 11
- Controlled comparisons show HDL achieves higher task scores than entropy-based and reflection-based localization methods.The authors state that these comparisons support hindsight divergence as a criterion for selecting branch points.Source: §5 · p. 11
- HDL selects decisions the model reconsiders after feedback, then fills groups with original-context continuations that reuse root prefixes.Additional exploration and learning concentrate on newly generated suffixes, reducing generation cost.Source: §5 · p. 11
A. Limitations
HDL depends on the policy interpreting verifier feedback reliably, and the reported results show a boundary across model scales. On Qwen3-1.7B, outcome agreement is lower and the ScienceWorld performance advantage disappears.
A. Limitations
- On Qwen3-1.7B, reflection outcome agreement is 76.0% on Math and 56.1% on Code, versus 99.7–99.9% on Qwen3-8B.Even single-word Math feedback (“Correct.”) is frequently hallucinated or misattributed by the smaller model.Source: A. Limitations · p. 14, Table 6 · p. 14
- On Qwen3-1.7B ScienceWorld, HDL scores 45.70% versus GRPO’s 45.66% and trails Entropy at 47.54%.The reported HDL performance advantage vanishes in this setting.Source: A. Limitations · p. 14, Table 7 · p. 14
- When the policy cannot reliably interpret verifier feedback, hindsight re-scoring can introduce noise into branch-point ranking.The method relies on the policy accurately assessing whether its root trajectory succeeded or failed.Source: A. Limitations · p. 14
B. Reflection prompts
The appendix contrasts HDL’s reflection prompt with a reflection baseline for choosing branch points. Both provide the task, attempt, and feedback, but request different outputs and selection instructions.
B.1. HDL reflection
- The HDL reflection prompt supplies the completed root and verifier feedback as inputs.It also includes the problem field.Source: §B.1 · p. 14
- HDL asks the policy to explain the attempt and why it succeeded or failed, and, if it failed, to state the right approach.The prompt says not to quote the attempt or mention positions, line numbers, or percentages.Source: §B.1 · p. 14
- HDL requires an OUTCOME label and a 60–120-token SUMMARY, with no text after the summary.The specified outcome is SUCCESS or FAILURE.Source: §B.1 · p. 14
B.2. Reflection baseline
- The Reflection baseline requests two branch points, presenting Math and Code roots as numbered steps and Agent roots as numbered turns.The lo and hi fields bound the allowed index range.Source: §B.2 · p. 15
- For failed Math or Code roots, the baseline selects the earliest step where the attempt goes wrong.The prompt supplies the problem, numbered steps, and verifier feedback, then requests a second, different step as an alternative.Source: §B.2 · p. 15
- For successful Math or Code roots, the baseline asks which step to branch on for a potentially better continuation.It requests two different steps within the allowed range and specifies a two-line STEP/STEP2 response.Source: §B.2 · p. 15
- For failed Agent episodes, the baseline selects the earliest turn where the episode goes wrong.It then requests a second, different turn as an alternative.Source: §B.2 · p. 15
- For successful Agent episodes, the baseline asks for a turn to branch on for a potentially better course of action.It requests two different turns within the allowed range.Source: §B.2 · p. 15