PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Outcome Rewards: Constructing and Assigning Retrieval Credit for Search Agents

Wenyu Huang, Xinyu Hou, Pavlos Vougiouklis, Ruofei Lai, Jeff Z. Pan

arXiv:2610.10179v1cs.CLcs.AIcs.LG

TL;DR

Sparse final-answer rewards can leave retrieval progress unused during search-agent training. The paper compares intermediate retrieval signals and trajectory-level versus action-local credit assignment, finding that both improve on outcome-only training and that their effects depend on how credit enters the update.

  • Problem

    Final-answer rewards provide limited information about intermediate searches, and tied outcome scores can leave retrieval progress unused in group-relative training.

    Source: §1 · p. 1

  • Method

    The paper uses corpus-grounded questions with evidence and dependency annotations to compare retrieval signals incorporated as trajectory-wide bonuses or local credit on the search action that produced evidence.

    Source: §1 · p. 1, §2.1 · p. 3

  • Results

    Across seven QA benchmarks, scalar and local retrieval supervision improve over outcome-only training; Cov-local gains 3.09 average F1.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Retrieval-signal choice and credit assignment are coupled: prerequisite constraints help scalar incorporation, while crediting evidence as acquired works better under local incorporation.

    Source: §8 · p. 9

    The analysis leaves how retrieved evidence enters subsequent reasoning through intact search trajectories as a further question.

    Source: §7 · p. 9

Abstract

from arXiv · show

Search agents enable Large Language Models (LLMs) to iteratively retrieve and use information for complex multi-hop questions. Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising approach for post-training such agents, but its reliance on sparse, outcome-based supervision can make credit assignment difficult and limit learning efficiency. In this paper, we systematically investigate how intermediate supervision can improve reinforcement learning for search agents. We study a range of reward-shaping and credit-assignment strategies that provide learning signals from intermediate retrieval steps. Building on these insights, we develop a training framework that combines intermediate signals with final outcome rewards to improve learning from multi-step search trajectories. Experiments across multiple benchmarks under matched training conditions demonstrate improvements in aggregate search-agent performance and show that both the choice of intermediate signal and where its credit is assigned affect training behaviour. These findings show that reward design and credit assignment are important design dimensions for training effective search agents.

1 INTRODUCTION

The section frames sparse final-answer rewards as overlooking retrieval progress, then studies how intermediate evidence signals and their credit assignment affect search-agent learning. It reports gains from retrieval supervision and distinguishes evidence acquisition from evidence use.

1 INTRODUCTION

  • Final-answer rewards can treat partial evidence retrieval and no retrieval alike, leaving retrieval progress unused when all trajectories in a question group receive the same answer score.Intermediate supervision can expose that progress before the final answer is correct.Source: §1 · p. 1
  • The study compares what evidence earns credit and whether that credit changes a trajectory-wide advantage or the update on the search action that produced it.A fact may be rewarded as soon as it is retrieved or only after its prerequisite evidence is found.Source: §1 · p. 1
  • Corpus-grounded questions retain intermediate answers, supporting passages, and dependencies, enabling retrieval-credit comparisons on fixed questions and analysis of retrieval separately from answer correctness.The annotations identify which subquestions a retrieval helps answer and how those subquestions connect.Source: §1 · p. 2
  • Across seven QA benchmarks, scalar and local retrieval supervision improve over outcome-only training; Cov-local gains 3.09 average F1 and 4.81 on multi-hop QA.Local credit gives the largest gains.Source: §1 · p. 2
  • On annotated development questions, Cov-local retrieves more supporting evidence while coverage-conditioned answer F1 remains similar, suggesting its main change is evidence acquisition.The passage identifies evidence use as a separate question.Source: §1 · p. 2

2 CONSTRUCTING CORPUS-GROUNDED RETRIEVAL REWARDS

The section constructs corpus-grounded questions with intermediate evidence annotations, then defines retrieval signals that score the evidence acquired along a search trajectory.

2.1 TRAINING DATA AND EVIDENCE ANNOTATIONS

  • Questions combine corpus-grounded relations into chains or conjunctive branches, represented as subquestion nodes with dependencies and their supporting passages.Conjunctive nodes retain both supporting passages; a node may depend on earlier answers.Source: §2.1 · p. 3
  • Generated questions retain subquestions, answer mentions, supporting-passage locations, and dependency edges, so coverage requires retrieved chunks to match every required passage for a node.The graph determines the answer decomposition, while the descriptive reference identifies the starting entity without naming it.Source: §2.1 · p. 3
  • The shared training set contains 14,000 questions sampled across one- to four-hop and single-evidence, conjunctive, chain, wide, and fan-in structures.The categories describe annotated structures, not the number of searches an agent must execute.Source: §2.1 · p. 3
  • Retaining evidence and dependencies lets the same observations be scored with or without prerequisite constraints while holding questions and grounding fixed.A separate annotated set supports evidence-coverage analysis.Source: §2.1 · p. 3

2.2 RETRIEVAL SIGNALS

  • Cov counts covered subquestion nodes regardless of predecessor coverage, whereas Cov-Dep credits a covered node only after its prerequisites are credited.For a chain A→B→C, retrieving A and C gives Cov 2/3 but Cov-Dep 1/3; later retrieving B unlocks B and the already covered C.Source: §2.2 · pp. 3–4
  • AM checks for a reference-answer alias in retrieved text, while all three signals convert cumulative progress into nonnegative increments over the previous maximum.Repeated retrieval of already credited evidence produces a zero increment.Source: §2.2 · p. 4

3 ASSIGNING RETRIEVAL CREDIT TO POLICY UPDATES

The section compares adding retrieval progress to a trajectory-level outcome reward with assigning credit to the search action that produced the evidence. Both approaches modify advantages within a GRPO policy update.

3 ASSIGNING RETRIEVAL CREDIT TO POLICY UPDATES

  • Both approaches build on GRPO, where outcome rewards are standardised within each question’s sampled trajectory group to produce relative advantages.The clipped policy objective is retained; scalar changes the shared advantage while local adds an action-specific term.Source: §3 · p. 4
  • Scalar incorporation adds the scaled accumulated retrieval bonus to the outcome reward, then broadcasts the resulting advantage across trainable response tokens.Retrieval differences can change trajectory rankings and supply variance when answer scores are tied.Source: §3 · p. 4
  • Local incorporation retains the outcome advantage and adds signed retrieval credit to eligible executed-query tokens aligned with the search that produced the evidence.Tool responses are excluded from the policy loss, and the full tool-call length sets the event-mass scale.Source: §3 · p. 5
  • Controlled comparisons shuffle local event credit across rollouts or restrict it to tied-score versus differing-score groups; these restrictions also reduce the total local credit applied.The comparisons test event–action correspondence and whether local credit is useful only when outcome rewards are tied.Source: §3 · p. 5

4 EXPERIMENTAL DESIGN AND RESULTS

Across seven QA benchmarks, local evidence-coverage supervision achieves the strongest reported average F1, while the benefits of retrieval signals vary with credit incorporation and task group.

4 EXPERIMENTAL DESIGN AND RESULTS

  • All conditions train Qwen3-4B-Instruct-2507 on the same 14,000 questions with fixed retrieval and equal GRPO budgets; Avg is micro-average F1 over 3,197 examples.Source: §4 · pp. 5–6

4.1 THE BENEFIT OF A SIGNAL DEPENDS ON ITS INCORPORATION

  • 54.34 average F1 for Cov-local versus 51.25 for outcome-only training; Table 1 compares scalar and local incorporation across seven datasets.Local incorporation outperforms scalar incorporation for each retrieval signal, with the largest gain for evidence coverage.Source: §4.1 · p. 6, Table 1 · p. 6
  • Removing dependency closure raises local F1 from 52.96 to 54.34 but changes scalar F1 from 52.82 to 52.64.Scalar incorporation sums retrieval events before group normalisation, whereas local incorporation compares them at each search step.Source: §4.1 · p. 6

4.2 PERFORMANCE ACROSS TASKS

  • Cov-local and AM-local improve over outcome-only training on all seven benchmarks, with larger gains on multi-hop than single-hop QA; Cov-local leads on multi-hop, while AM-local slightly exceeds it on single-hop.Source: §4.2 · p. 6
  • Cov-Dep-local improves over its scalar counterpart on multi-hop QA but falls below it on single-hop QA.Source: §4.2 · p. 6

5 WHICH PROPERTIES OF RETRIEVAL CREDIT MATTER?

The section tests whether local retrieval credit works best when tied to its source search and applied across outcome groups, then examines what Cov-local changes in evidence acquisition.

5.1 PRESERVING ACTION CORRESPONDENCE IMPROVES LOCAL CREDIT

  • Aligned credit beats permuted credit for both signals, increasing average F1 by 2.12 for Cov and 3.12 for AM, with a larger advantage on multi-hop QA.Figure 3b compares aligned and permuted credit overall and within task groups.Source: Figure 3 · p. 7, §5.1 · p. 7

5.2 RESTRICTING LOCAL CREDIT TO OUTCOME TIES LIMITS ITS BENEFIT

  • Using all outcome groups gives the highest aggregate F1 for Cov and AM; restricting Cov loses 0.32 with no-flat or 2.27 with flat-only, while AM loses 1.44 or 2.19, respectively.Cov-flat-only remains above OO, while AM-flat-only performs similarly to OO.Source: Figure 3 · p. 7, §5.2 · pp. 7–8

5.1 PRESERVING ACTION CORRESPONDENCE IMPROVES LOCAL CREDIT

  • Against OO, Cov-permute reaches F1 52.22 versus 51.25, while AM-permute is 50.39; matching credit to its source search raises F1 further.The shuffle preserves signed event masses but changes which search receives each credit.Source: §5.1 · p. 7

5.3 EVIDENCE ACQUISITION

  • Cov-local reduced zero-coverage questions from 616 to 551 and raised complete-coverage questions from 142 to 167; mean Cov rose from 27.51% to 31.97%.The policies averaged 2.45 and 2.55 searches per question, respectively.Source: Figure 4 · p. 8, §5.3 · p. 8
  • Coverage-conditioned F1 curves follow similar patterns and do not consistently favor Cov-local, but the policies’ groups at each coverage level contain different question IDs.The curves therefore do not isolate evidence use on the same questions.Source: Figure 5 · p. 8, §5.3 · p. 8

6 RELATED WORK

Related work covers multi-turn search agents, intermediate supervision, and retrieval-credit assignment; this study examines how retrieval signals and their policy-update rules interact.

Multi-turn search agents

  • Prior work trains multi-turn search behavior with reinforcement learning and answer feedback, while one empirical study reports limited gains from intermediate retrieval rewards in the settings examined.The paper investigates how reward effectiveness depends on how retrieval credit enters the policy update.Source: §6 · p. 8

Intermediate supervision

  • Retrieval-supervision methods reward document relevance, decompose tool-use trajectories, assess search necessity, or combine forward solvability gains with hindsight estimates of search-step importance.This study compares answer matching with corpus-grounded evidence coverage, including a variant constrained by prerequisite relations.Source: §6 · p. 9

Credit assignment for retrieved evidence

  • Prior credit-assignment methods attribute or redistribute intermediate rewards in different ways; this study compares signed retrieval-event residuals across same-question rollouts and search steps.The residual can remain active at zero outcome advantage or reverse a nonzero advantage.Source: §6 · p. 9

7 LIMITATIONS

The limitations identify two unresolved questions: how acquired evidence is used during intact trajectories, and how reward construction interacts with update strength and generalizes across settings.

7 LIMITATIONS

  • The analysis examines evidence acquisition and retrieval credit, but does not trace evidence use through intact trajectories, including cases where added coverage leaves the answer unchanged.Such analysis could inform supervision connecting evidence acquisition with evidence use.Source: §7 · p. 9
  • The interaction between reward construction and update strength remains open; independent controls and matched-update comparisons could clarify contributions, while broader model and retriever tests would assess generalization.The proposed controls vary event timing, reward magnitude, and credit eligibility independently.Source: §7 · p. 9

8 CONCLUSION

The conclusion argues that retrieval-signal design and credit assignment should be considered together, with their effects depending on how the signal enters the policy update.

8 CONCLUSION

  • The same retrieval signal has different effects depending on its policy-update incorporation: dependency constraints help scalar incorporation, while crediting evidence as acquired works better under local incorporation.A more constrained evidence signal does not necessarily provide more useful supervision.Source: §8 · p. 9
  • Local retrieval credit is most effective when tied to its source search and used across tied and differing outcome groups; on development questions, it primarily changes evidence acquisition, leaving evidence use open.The conclusion treats evidence scoring and action-level credit assignment as coupled design choices.Source: §8 · p. 9

REPRODUCIBILITY STATEMENT

The reproducibility statement directs readers to the paper’s data-generation methods, training settings, grounding checks, and complete results.

REPRODUCIBILITY STATEMENT

  • Question generation is described in Section 2.1, and retrieval rewards and credit assignment are defined in Sections 2 and 3.Appendix A covers training settings and reward-scale checks; Appendix B covers data construction and grounding checks.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Appendix C provides complete final-checkpoint EM and F1 results and dataset-sensitivity analyses.Appendix D details the example in Figure 1a.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The authors state that they will release the code, trained models, and constructed dataset upon acceptance.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE STATEMENT

The appendices document AI use, shared training and interface settings, reward implementation, scalar-reward checks, and evaluation procedures.

AI USE STATEMENT

  • GPT-5.6 Luna generated descriptive root references for synthetic questions, and generative AI also assisted with experimental code and manuscript drafting.The authors say they reviewed AI-assisted work and checked results against saved evaluation outputs.Source: AI USE STATEMENT · p. 10

A.1 HARDWARE, OPTIMISATION AND RUNTIME

  • Training used Qwen3-4B-Instruct-2507 on two 80-GB NVIDIA A100 GPUs, with shared hyperparameters and a fixed E5-base-v2 retriever.Training took approximately 28–31 hours across seven measured conditions, including rollouts, updates, validation, and checkpoint saving.Source: §A.1 · p. 13, Table 2 · p. 13

A.2 PROMPT AND TOOL INTERFACE

  • The native tool interface executes only the first query in each call, keeping the retrieval budget per call fixed across conditions.Calls use a fixed KILT/E5 retriever, and runtime enforces a four-assistant-turn limit.Source: §A.2 · p. 14

A.3 REWARD AND CREDIT IMPLEMENTATION

  • Scalar variants add 0.25 times accumulated retrieval progress to terminal F1, while local variants assign retrieval credit to executed query tokens with λ = 0.5.Missing execution records, unidentifiable query spans, inconsistent event arrays, or truncated trajectories disable local credit, leaving the global outcome advantage.Source: §A.3 · p. 14

A.4 SCALAR REWARD SCALE

  • At α = 0.25, Cov-Dep reverses 0.57% of unequal-F1 pairs and AM reverses 1.49%, while scalar bonuses also provide updates in outcome-flat groups.The scale check uses saved trajectories; lowering α does not proportionally lower the standardised update except near the numerical tolerance.Source: §A.4 · pp. 15–16

A.5 EVALUATION AND TRAJECTORY MEASUREMENTS

  • The main evaluation weights dataset-level EM and F1 by sample counts across 3,197 questions, while development trajectory analysis uses 1,200 synthetic questions.The development analysis measures retrieved evidence from visible tool observations and separately groups each policy’s questions by final coverage.Source: §A.5 · p. 16
  • The development analysis retains missing answers and truncated trajectories in its denominator; 507 questions share annotated evidence with training, while 693 have fully unseen evidence.Search count measures tool use, not generation cost or wall-clock latency.Source: §A.5 · p. 16

B TRAINING DATA AND CORPUS CONSTRUCTION

The appendices describe how corpus-grounded questions and evidence annotations are constructed, then report retrieval-credit effects and task-group performance patterns.

B TRAINING DATA AND CORPUS CONSTRUCTION

  • The construction combines corpus-grounded relations into chain and conjunctive questions while retaining supporting passages, intermediate answers, and evidence dependencies.The August 2019 KILT Wikipedia corpus and August 2022 Wikidata dump inform relation selection; question structures and dependencies are assembled using graph operations and templates.Source: B TRAINING DATA AND CORPUS CONSTRUCTION · p. 16
  • The study deterministically sampled within each structural category, reserving 1,200 questions for the Section 5.3 evidence-coverage analysis and selecting 14,000 training questions from the remainder.Of 87,171 source questions, 86,681 were eligible; 168 failed the chunk-grounding check, 297 were labeled graph shortcuts, and 25 did not match a requested category.Source: B TRAINING DATA AND CORPUS CONSTRUCTION · p. 16
  • Cov and Cov-Dep match annotated evidence to retrieved chunks using page, paragraph, and answer-mention start position, with dependency closure applied separately by Cov-Dep.The check does not require the full answer mention or supporting sentence to fit in a chunk.Source: B TRAINING DATA AND CORPUS CONSTRUCTION · p. 17
  • The development split has no exact question-ID or normalized-question overlap with training, but only 693 questions have fully unseen gold evidence.The authors note that shared evidence limits the split’s value for measuring generalization.Source: B TRAINING DATA AND CORPUS CONSTRUCTION · p. 17

C ADDITIONAL RESULTS

C.1 MULTI-HOP AND SINGLE-HOP QA

  • Cov-local exceeds Cov-permute by 3.73 F1 on multi-hop QA and 0.39 F1 on single-hop QA; AM-local exceeds AM-permute by 4.75 and 1.35 F1, respectively.Local incorporation improves multi-hop QA for every signal, but Cov-Dep-local falls below its scalar counterpart on single-hop QA; the groups contain different benchmarks, so this does not isolate hop-count effects.Source: §C.1 · pp. 18, 20, Table 8 · p. 19
  • Both scalar and local incorporation improve EM and F1 over outcome-only training for all three signals, with Cov-local giving the largest gains.Figure 6 reports average EM and F1 differences from outcome-only training.Source: Figure 6 · p. 20

C.2 SENSITIVITY TO THE DATASET MIXTURE

Cov-local and AM-local remain above outcome-only training across dataset omissions. The examples show that evidence coverage can differ even when answer correctness does not, and that Cov may credit a retrieved fact before its prerequisites are found.

C.2 SENSITIVITY TO THE DATASET MIXTURE

  • Cov-local and AM-local remain above OO under every dataset omission in the sensitivity check.Table 9 reports full seven-dataset F1 contrasts of +3.09 and +2.26, respectively.Source: §C.2 · p. 20, Table 9 · p. 21

D COVERAGE EXAMPLE

  • 0.67 Cov versus 0.33 Cov-Dep: the search retrieves the dynasty fact but not the intervening father link.The covered set is {v1, v3}, while dependency closure admits only {v1}.Source: D COVERAGE EXAMPLE · p. 21

E A PAIRED EXAMPLE OF EVIDENCE ACQUISITION

  • Cov-local retrieves both annotated evidence nodes in two searches, while OO takes three; both policies answer correctly.OO finds the identifying passage on its third search; Cov-local finds it on its first and retrieves the career passage on its second, raising coverage from one node to two despite the answer already being present.Source: E A PAIRED EXAMPLE OF EVIDENCE… · pp. 21–22, Figure 8

Related papers

Loading 2610.10179v1…