IGSD: Environment-Verified Hindsight Self-Distillation for Search Agents

• Trajectory-level outcome rewards can give informative and redundant or misleading queries the same credit, leaving intermediate search decisions without fine-grained supervision.Source: §1 · p. 1
• A hindsight query useful in a successful sibling may be redundant or misleading at the student’s failed state, so teacher preference alone does not establish retrieval benefit.Source: §1 · p. 1
• Model-internal likelihood-gap filters measure preference rather than whether a proposal brings needed evidence, motivating verification by search execution.Source: §1 · p. 2
• IGSD executes matched teacher and student query proposals from the student’s visited state and uses their environment-derived utility contrast as a positive-only soft distillation weight.The GRPO objective remains unchanged, and verification is confined to training.Source: §1 · p. 2
• Across seven single-hop and multi-hop QA benchmarks, IGSD achieves the highest macro-average exact-match accuracy, reaching 42.8% with 3B and 47.0% with 7B policies without inference-time verification.Source: §1 · p. 2
• Mechanistic analysis reports stronger observed alignment between executed gain and terminal outcomes than for the tested model-internal signals.Source: §1 · p. 2

• Search-agent reinforcement learning uses trajectory-level rewards, while IGSD adds paired environment-executed comparisons of candidate actions from the same local state and retains that objective.Source: §2 · p. 2
• Prior hindsight self-distillation methods include SD-Search, Search-E1, SDAR, and Skill-SD; unlike these approaches, IGSD executes teacher- and student-induced queries from the same state.SDAR uses a detached likelihood gap, an internal preference signal rather than retrieved-evidence utility.Source: §2 · p. 2
• IG-Search and IGPO use information gain as an RL signal, while CRAFT and LOTAPO estimate token- or process-level credit from sibling rollouts; IGSD uses executed gain only to weight auxiliary distillation.Source: §2 · pp. 2–3

3.1 PROBLEM SETUP

• IGSD derives local supervision for failed trajectories while retaining GRPO’s terminal reward and advantage estimator unchanged.The answer evaluator supplies a scalar terminal outcome score; IGSD uses answer correctness to label successful or failed trajectories.Source: §3.1 · p. 3

3.2 HINDSIGHT-GUIDED CANDIDATE CONSTRUCTION

• For a failed trajectory with a successful sibling, IGSD selects the last valid search action and tests teacher-proposed tokens that differ from sampled student tokens.The hindsight hint contains the sibling’s search queries and outcome score, but excludes its answer, reasoning, retrieved documents, and tool observations.Source: §3.2 · pp. 3–4

3.3 PAIRED ENVIRONMENT VERIFICATION

• Each candidate token is greedily completed into a query by the frozen student policy, then compared with its paired query under shared execution conditions.The two branches share the state, prefix, continuation policy, decoding procedure, and retriever; only the candidate token differs.Source: §3.3 · p. 4
• IGSD controls for query-dependent shifts in answer likelihood using shared counterfactual document sets, then contrasts the controlled utilities of the executed queries.A positive executed paired information gain means the teacher-induced query’s retrieved evidence provides more answer support than the student-induced query’s evidence.Source: §3.3 · p. 4

3.4 GAIN-GATED CANDIDATE-PAIR DISTILLATION

• Executed gain sets whether and how strongly a candidate pair contributes to auxiliary distillation; the Jensen–Shannon divergence transfers the verified teacher preference.Pairs with non-positive gain receive zero auxiliary weight, and rejected query positions contribute zero to the loss normalized over all query-token positions.Source: §3.4 · p. 5
• Verification is training-only; inference uses the standard search agent without a teacher, verification branches, or extra retrieval calls.The auxiliary objective supplements the actor objective rather than changing the GRPO objective.Source: §3 · p. 3, §3.4 · p. 5

4.1 EXPERIMENTAL SETUP

• The evaluation trains on NQ and HotpotQA and reports exact-match accuracy on seven benchmarks, using 3B and 7B policies and several inference-only, outcome-reward, and process-supervision baselines.The experiments ask whether verified hindsight improves policies, aligns with downstream utility, and which decision-chain components contribute.Source: §4.1 · pp. 5–6, §4 · p. 5

4.2 MAIN RESULTS (RQ1)

• IGSD reaches macro-averages of 0.428 at 3B, exceeding AutoRefine (0.400), and 0.470 at 7B, ahead of its ungated control (0.453), Search-R1 (0.445), and SD-Search (0.456).At 7B, IGSD leads on all four multi-hop benchmarks.Source: §4.2 · p. 6

4.3 MECHANISTIC EVIDENCE (RQ2)

• Executed paired gain tracks downstream utility: positive-gain teacher branches reach 0.378 vs. 0.199 EM, interval gaps reach +44.8 points, and winner ranking achieves AUC 0.708 vs. 0.552.The comparison fixes candidate tokens in turn and independently completes both branches with the same frozen student policy and decoding procedure.Source: §4.3 · pp. 6–8, Figure 3 · p. 7

4.4 ABLATION STUDIES (RQ3)

• Verifying every valid turn reaches 0.421 macro-average versus 0.428 for last-valid-turn verification, so the study focuses supervision on the final valid search turn.The authors say accumulated context there makes teacher–student disagreement more likely to expose a concrete query alternative.Source: §4.4 · p. 8
• 0.428 for full paired verification exceeds 0.397 without the sampled-token branch, 0.405 without counterfactual control, and 0.413 with the student argmax reference.The ablation supports using the sampled student action and a shared counterfactual control in the executed comparison.Source: §4.4 · p. 8, Table 3 · p. 8
• 0.428 per-pair executed-gain gating exceeds query-level averaging (0.419), the log-probability gate (0.409), and ungated unit weights (0.417).The authors interpret the differences as evidence that query averaging can wash out local action differences and internal preference is not an adequate substitute for executed utility.Source: §4.4 · pp. 8–9, Table 3 · p. 8
• 0.428 candidate-pair JSD exceeds top-50 JSD (0.419) and top-50 reverse-KL (0.420), whose broader supports include tokens not directly adjudicated by execution.Source: Table 3 · p. 8, §4.4 · p. 9

4.5 QUALITATIVE CASES

• A retained teacher query resolves Berlinda Tolbert and retrieves The Jeffersons (∆IG = +3.58), while a redundant Freud query is rejected (∆IG = −0.198).The examples contrast an unresolved entity in the student query with a relation already identified in the current evidence.Source: Table 4 · p. 9, §4.5 · p. 9

5 LIMITATIONS

• IGSD’s utility estimate relies on reference-answer supervision.The method does not directly extend to open-ended tasks lacking an explicit correctness criterion.Source: §5 · p. 9
• Direct application is limited to tasks with an explicit answer or reliable outcome evaluator.Source: §5 · p. 9
• Open-ended tasks without an explicit correctness criterion are not a direct extension of IGSD.Source: §5 · p. 9

6 CONCLUSION

• IGSD improves macro-average exact-match accuracy over evaluated baselines across seven QA benchmarks and both 3B and 7B policies.Executed gain also aligns more strongly with terminal utility than the tested model-internal signals, while GRPO and inference remain unchanged.Source: §6 · p. 10

AI USE STATEMENT

• The authors report that generative AI tools helped polish presentation and support an initial survey and work organization; they reviewed and verified the technical claims, citations, results, and final wording.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• The paper and appendix provide benchmark, data-split, retrieval, prompt, hyperparameter, evaluation, random-seed, and ablation details intended to make the setup and comparisons independently checkable.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 DATA STATISTICS

• The study uses seven QA benchmarks from the FlashRAG collection, combines NQ and HotpotQA training splits, and evaluates on test splits when available or development splits otherwise.Table 5 is identified as the dataset-statistics table; its caption does not provide the cell values.Source: §A.1 · p. 12, Table 5 · p. 12

A.2 COUNTERFACTUAL SAMPLING PROTOCOL

• Counterfactual document sets are sampled uniformly without replacement from the current batch’s retrieved passages after removing duplicates and branch-retrieved passages.Teacher and student branches share each sampled set; the default is C = 3 draws, serialized using the real-response format.Source: §A.2 · p. 12

A.3 VERIFICATION COVERAGE

• Auxiliary candidates are teacher–student token disagreements in a failed rollout with a successful sibling and valid last search turn; only valid matched executions with positive gain receive nonzero weight.Table 6 distinguishes token-level training-funnel rates from a separate terminal-audit diagnostic subset.Source: §A.3 · p. 12, Table 6 · p. 12

A.4 PROMPTS AND TOOL INTERFACE

• The base question prompt, privileged teacher instruction, and Hermes-style search schema enter at different trajectory stages; IGSD does not rewrite the dataset question.Table 7 summarizes the prompts and search-tool interface.Source: §A.4 · p. 13, Table 7 · p. 13

B SIGNAL DIAGNOSTICS

• 47 of 72 terminal-discordant pairs (65.3%, CI [54.2%, 76.1%]) select the winning branch under executed gain’s zero-threshold sign, the highest AUC among tested signals.Table 8 compares executed gain with model-internal token scores using the same terminal-winner labels.Source: B SIGNAL DIAGNOSTICS · p. 13, Table 8 · p. 13
• Positive executed gain contains 34 teacher-only correct and 6 student-only correct pairs, versus 19 and 13, respectively, in the non-positive group.Figure 5 shows the terminal-discordant pair composition; the positive-gain group has a larger share of teacher-winning outcomes.Source: B SIGNAL DIAGNOSTICS · p. 13, Figure 5 · p. 14
• Executed gain rises early, reaches a mid-training maximum, then decreases while remaining positive as agreement among the three teacher and student actions increases.Figure 6 tracks executed gain alongside three-way action agreement over training.Source: B SIGNAL DIAGNOSTICS · p. 14

C REPEATED-RUN RESULTS

• Table 9 reports results for four core configurations across three random seeds.The table presents means ± standard deviations for Qwen2.5-3B-Instruct.Source: Table 9 · p. 14, C REPEATED-RUN RESULTS · p. 14
• The repeated-run results use mean ± standard deviation to report variability across seeds.Source: Table 9 · p. 14
• The best mean in each Table 9 column is bolded.Source: Table 9 · p. 14

D EFFICIENCY AND SEARCH USAGE

• Table 10 separates training cost over 400 optimization steps from inference-time tool use.Both are wall-clock measurements under their corresponding experimental settings.Source: D EFFICIENCY AND SEARCH USAGE · p. 14, Table 10 · p. 15
• Table 10 reports inference-time search calls on the same 51,713-example greedy evaluation set.Calls per example are reported to three decimals.Source: Table 10 · p. 15
• Table 11 gives dataset-level average evaluation-time search calls per example for Search-R1, SD-Search, and IGSD.Source: D EFFICIENCY AND SEARCH USAGE · p. 14, Table 11 · p. 15

E.2 COMPLETE VARIANT COMPARISONS

• Table 12 compares supervision-scope, verification, gating, support, and counterfactual-draw variants with the 3B policy.The verification variants include removing the sampled-token reference or shared counterfactual control, while gate variants alter how executed gain weights supervision.Source: §E.2 · p. 15, Table 12 · p. 16

E.1 HINDSIGHT CONTEXT ABLATION

• The default “Queries + score” context is strongest at macro-average 0.428; “Queries only” scores 0.417, “Queries + score + docs” 0.413, “No hindsight” 0.400, and “Queries + score + answer” 0.385.The answer-provided context is an intentionally leaky stress test of privileged information.Source: §E.1 · p. 15

F HYPERPARAMETER SENSITIVITY

• The default configuration reaches the highest macro-average in each one-at-a-time hyperparameter sweep.Figure 8 varies gate sharpness, distillation coefficient, and counterfactual draws around the default.Source: F HYPERPARAMETER SENSITIVITY · p. 16, Figure 8 · p. 17

G ADDITIONAL CASE SUMMARIES

• Table 14 presents three positive-gain paired trajectories and one negative-gain example with independently completed terminal answers.The cases show both search branches, retrieved snippets, and resulting executed paired information gain.Source: G ADDITIONAL CASE SUMMARIES · p. 17, Table 14 · p. 18