PaperlayerView exact revision on arXiv

Source-linked AI summary

Grounded Credit Assignment in Reinforcement Learning for Multi-Turn Search Agents

Qirui Liu, Hao Chen, Yi Wen, Weijie Shi, Yao Zhao, Xianhe Chen, Hao Zhao, Xing Tang, Weihong Luo, Yuyu Luo

arXiv:2610.23148v1cs.LG

TL;DR

Outcome-reward RL assigns the same advantage to every token in a search rollout, conflating finding evidence with using it. GCA separates credit using verifiable sub-outcomes and bounded token-level residuals. Across seven QA benchmarks and two backbones, it improves accuracy and more strongly improves grounding of correct answers.

  • Problem

    Outcome-reward RL broadcasts one group-relative advantage across a rollout, although search agents must both acquire evidence and use it to answer.

    Source: §1 · p. 1

  • Method

    GCA uses verifiable retrieval-hit and answer-correctness sub-outcomes to route bounded, sign-preserving residuals to functional segments and decision-critical tokens.

    Source: §1 · p. 1, §4.1 · p. 3

  • Results

    Across seven QA benchmarks, GCA improves average EM over Search-R1 by 5.7 points on Qwen2.5-3B and 6.3 points on Qwen3-30B-A3B, while the 3B grounded share rises from 76.9% to 93.4%.

    Source: §1 · p. 2, §5.2 · p. 6

  • Takeaways & Limitations

    Across the evaluated benchmarks and backbones, GCA improves accuracy and, more strongly, the grounding of correct answers without extra models, rollouts, or training stages.

    Source: §6 · p. 8

    The retrieval-hit signal measures answer coverage rather than semantic support, can miss some answer forms, and has not been evaluated with gold-free replacement signals.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Outcome-reward reinforcement learning such as GRPO gives every token of a search-agent rollout the same advantage, although the rollout exercises two capabilities: finding evidence and using it. We show that accuracy improves without grounding: Search-R1 raises accuracy on all seven QA benchmarks, yet on Bamboogle the evidence-supported share of correct answers falls from 84.0% to 42.9%. We attribute this to GRPO's credit assignment: rollouts that retrieve answer-bearing evidence are favoured in aggregate only in proportion to their accuracy gain over the rest, a grounding gap that answering from memory or misusing evidence shrinks or reverses. To address this, we propose Grounded Credit Assignment (GCA), which uses verifiable sub-outcomes (evidence retrieval and answer correctness) to correct search and answer credit separately: a wrong answer is blamed more than the search that found the evidence, and a correct answer that bypasses retrievable evidence is credited less. Never negative on evidence-finding search, the correction does not shrink with the grounding gap; capped at 10% of each token's advantage, it never overrides the outcome signal. Without extra models, rollouts, or training stages, GCA improves average EM over Search-R1 by 5.7 (Qwen2.5-3B) and 6.3 points (Qwen3-30B-A3B) and raises this share from 76.9% to 93.4%.

1 Introduction

Outcome-only RL can raise search-agent accuracy while weakening evidence grounding because one rollout-level advantage conflates finding evidence with using it. Grounded Credit Assignment (GCA) separates these credits using verifiable sub-outcomes and bounded, token-level corrections.

1 Introduction

  • Search-R1 raises EM on all seven benchmarks, yet the evidence-supported share of correct answers falls from 88.3% to 71.8% on MuSiQue and from 84.0% to 42.9% on Bamboogle.The paper attributes this drift to retrieval receiving aggregate credit only insofar as it coincides with correct answers.Source: §1 · p. 1
  • GCA uses verifiable evidence-retrieval and answer-correctness sub-outcomes to correct credit separately for each functional segment.It shifts blame for wrong answers from evidence-finding search toward answering, and reduces credit for redundant searches and correct answers that bypass retrievable evidence.Source: §1 · pp. 1–2
  • GCA's residual is capped at 10% of each token's advantage, preserves its sign, and is never negative on valid search.The residual is concentrated on decision-critical tokens, without extra models, rollouts, or staged training.Source: §1 · p. 2
  • GCA improves average EM over Search-R1 by 5.7 points on Qwen2.5-3B and 6.3 points on Qwen3-30B-A3B, while raising the evidence-supported share from 76.9% to 93.4%.The token-level diagnosis finds 38–51% of answer-finding search tokens in later-incorrect rollouts and 33–40% of positively credited tokens redundant or ungrounded.Source: §1 · pp. 1–2

2 Related Work

Prior search-agent methods vary environments, rewards, rollout procedures, or credit granularity, while GCA retains the existing setup and changes token-level advantage distribution. Its functional-role grouping differs from grouping by time step or environment state.

2 Related Work

  • GCA keeps the search environment, reward, and rollout procedure fixed while changing how advantage is distributed over tokens.Search-R1 follow-ups alter one or more of those components but still assign one advantage to all tokens of a trajectory or turn.Source: §2 · p. 2
  • GCA groups credit by a segment's functional role, an axis orthogonal to methods that group by time step or environment state.The contrasted approaches include turn-level rewards or critics, step-level advantages, and a two-stage curriculum.Source: §2 · p. 2
  • GCA uses margin and entropy to distribute an already-diagnosed segment residual and requires no extra model.This contrasts with token-level relevance discrimination and restricting policy gradients to high-entropy tokens.Source: §2 · pp. 2–3

3 Credit Conflation in Outcome-Reward Search RL

The section formalizes how outcome-only group-relative advantages can conflate evidence acquisition and answer correctness. Its grounding-gap identity and rollout analysis explain when aggregate credit favors or penalizes retrieval.

3.1 Setup

  • Each rollout receives an exact-match-plus-format outcome reward and a group-relative advantage broadcast to all assistant tokens.The setup samples multiple multi-turn search rollouts per question; only assistant tokens are optimized.Source: §3.1 · p. 3
  • The setup records whether a tool response contains a gold alias, the first hit turn, and whether the final answer is correct.These signals define grounded success, evidence found but misused or budget-exhausted, no-hit incorrect, and unsupported-correct rollout types.Source: §3.1 · p. 3

3.2 The grounding-gap identity

  • Under binary correctness rewards, the aggregate advantage difference between hit and miss rollouts is proportional to the within-group grounding gap.The identity concerns aggregate scalar advantages, not the full search-token policy gradient.Source: §3.2 · p. 3
  • Parametric shortcuts and evidence misuse shrink the grounding gap and can make it negative, penalizing rollouts that found evidence.Search-R1's TOOLHIT falls from 25.8 to 21.9 on MuSiQue and from 35.2 to 19.2 on Bamboogle relative to the initial checkpoint.Source: §3.2 · p. 3
  • Even with a positive grounding gap, type-B rollouts give negative outcome credit to both the wrong answer and the search turns preceding the first hit.These rollouts contain 38–51% of answer-finding search tokens; conflict cells occur when a segment's outcome sign disagrees with its controlled sub-outcome.Source: §3.2 · p. 3

4 Grounded Credit Assignment

GCA leaves rewards, group statistics, and the optimizer unchanged while adding a bounded residual to token-level advantages. It routes residuals by segment function and sub-outcome, then applies token salience.

4 Grounded Credit Assignment

  • GCA corrects token-level advantages while leaving the reward, group statistics, and optimizer unchanged.Its overview diagnoses rollouts by retrieval hit and answer correctness, routes signed residuals to functional segments, and concentrates them on decision-critical tokens.Source: §4 · p. 3, Figure 2 · p. 4

4.1 Routing motivated by verifiable trajectory sub-outcomes

  • VSEARCH covers valid searches through the first evidence hit, RSEARCH covers later or no-hit searches and malformed calls, and ANS covers the final-answer turn.This first-hit segmentation is an operational proxy for acquisition-related versus potentially redundant search, not proof of each turn's causal contribution.Source: §4.1 · pp. 3–4
  • Routing residual signs follow each segment's proxy sub-outcome, with no residual for type C or type D when the corpus lacks the answer.The routing coefficients mark conflict cells where outcome and sub-outcome signs disagree; type D with no retrievable answer is masked from policy loss.Source: §4.1 · p. 4, Table 1 · p. 4
  • Residual magnitudes follow a bottleneck rule, and learning them under the same budget does not improve results.The segment that decided the outcome receives the largest magnitude, while useful search in a failed rollout retains a moderate positive residual.Source: §4.1 · p. 4

4.2 Bounded, sign-preserving residual

  • A gated residual is added to each token's advantage and replaces that advantage in the GRPO clipped surrogate or GSPO sequence-level objective.The gate excludes zero-variance groups, type C rollouts, structural-tag literals, and tokens without a nonzero routing coefficient.Source: §4.2 · p. 4

0. Atoken

GCA preserves outcome-derived token-advantage signs and the PPO clipping branch while reweighting tokens within a bounded range. Margin–entropy salience redistributes the residual toward useful uncertain choices or harmful confident choices.

0. Atoken

  • Each token keeps its outcome-derived advantage sign and PPO clipping branch, with its gradient reweighted by a factor in [1 − κ, 1 + κ].Groups with zero advantage throughout receive no signal.Source: §0 · pp. 4–5
  • With κ = 0.1, GCA changes token-credit strength by at most 10% without reversing outcome credit.Positive residuals can attenuate negative outcome credit, while negative residuals can attenuate positive outcome credit.Source: §0 · p. 5
  • The residual on evidence-search tokens in hit rollouts is non-negative and does not vanish as the grounding gap shrinks, though it remains capped per token.For unsupported correct answers, the negative residual attenuates but does not reverse positive outcome credit.Source: §0 · p. 5

4.3 Margin–entropy token salience

  • Margin–entropy salience favours uncertain but useful choices in positive cells and confident but harmful choices in negative cells.The examples are a pivotal query term and a confidently wrong year; positive and negative cells with equal λ receive equal budgets.Source: §4.3 · p. 5
  • Salience redistributes only a residual whose sign and bound are already fixed, and results are reported as insensitive to K and M.The configuration uses K = 3 competitors and M = 32 top-policy tokens.Source: §4.3 · p. 5

5 Experiments

Across seven QA benchmarks, GCA improves average EM over Search-R1 and increases evidence grounding. Ablations, routing tests, signal audits, and training traces examine the method’s components and behavior.

5.1 Setup

  • The evaluation uses seven QA benchmarks and reports EM, lenient accuracy, and grounding metrics; Search-R1 and GCA share the retrieval setup and prompts.The setup uses a 2018 Wikipedia dump, top-3 passages, and at most five turns; GCA changes only the token-level advantage, while using a higher KL coefficient than Search-R1.Source: §5.1 · p. 5

5.2 Main results

  • 5.70 points on Qwen2.5-3B and 6.30 points on Qwen3-30B-A3B: GCA improves average EM over Search-R1 under identical settings.Gains are largest on multi-hop benchmarks; GCA also raises the grounded share of correct answers from 76.9% to 93.4% on 3B and from 92.3% to 94.1% on 30B.Source: §5.2 · p. 6, Table 2 · p. 6

5.3 Ablation study

  • Merging the search segments costs 2.75 EM and 10.9 TOOLHIT points, lowers all seven benchmarks, and reduces the grounded share from 93.4% to 82.0%.Removing salience retains 4.65 of the 5.70 EM-point gain but loses 6.8 TOOLHIT points and 5.8 grounded-share points; the first-hit split is the most consequential tested ablation.Source: §5.3 · p. 7, Table 4 · p. 7

5.4 Learned routing and sensitivity

  • The learned router matches the fixed table in-domain but trails it slightly out-of-domain, so learning magnitudes brings no gain.On 3B, EM remains 40.8–43.7 across tested ranges of residual and salience parameters; on 30B, every tested configuration exceeds Search-R1, though EM is more sensitive to the λ scale.Source: §5.4 · p. 8, Table 5 · p. 8, Table 6 · p. 8

5.5 Reliability of the grounding signal

  • H is a near-perfect recall proxy for genuine evidence except for yes/no verdicts, which are inferred rather than copied.The matcher misses aliases in 3.61% of annotated supporting-document cases, falling to 0.040% when yes/no answers are excluded; misdiagnosis changes token credit by at most 10% without flipping its sign.Source: §5.5 · p. 8

5.6 Training dynamics

  • GCA raises SUPCOR and retrieval coverage without a corresponding increase in the reported hit-but-wrong rate, while mean outcome reward and actor entropy are higher.At the end of training, zero-variance groups are 70% vs. 75%, and actor entropy is 0.59 vs. 0.36; BFCL v3 overall improves from 68.62 to 69.14, mainly on multi-turn calls.Source: Figure 3 · p. 7, §5.6 · p. 8

6 Conclusion

GCA improves accuracy and the grounding of correct answers across seven QA benchmarks and two backbones, using bounded residuals routed by verifiable sub-outcomes. The conclusion expects this approach to transfer to agents with other verifiable intermediate outcomes, while the evaluation and design choices remain limited in scope.

6 Conclusion

  • Across seven QA benchmarks and two backbones, GCA improves accuracy and, more strongly, the grounding of correct answers without extra models, rollouts, or training stages.The authors expect the bounded-residual recipe to transfer to agents with verifiable intermediate outcomes such as evidence-entailment judges or executable tests.Source: §6 · p. 8

Limitations

  • GCA signals use gold-answer aliases and a lexical matcher, which measures answer coverage rather than semantic support and can match unsupported mentions.Applying GCA where no gold answer exists would require alternative signals, such as evidence-entailment judges or executable checks, which the authors have not evaluated.Source: Limitations · p. 9
  • The evaluation covers two Qwen-family backbones and one E5 retrieval environment over the 2018 Wikipedia dump; other model families and live web search remain untested.At 30B, only Search-R1 is retrained as a baseline, and seed variance is reported only for GCA.Source: Limitations · p. 9
  • One routing table is used across benchmarks and backbones, but not every cell is ablated, salience weights are fixed, and salience overhead is unmeasured.The salience score reuses old-policy logits.Source: Limitations · p. 9

A Notation

  • The notation distinguishes retrieval hit H, answer correctness C, first-hit turn k*, rollout type T, token segment S, and token-level advantage A.It also defines the within-group grounding gap ∆ as Pr[C | H=1] − Pr[C | H=0].Source: A Notation · p. 10, Table 7 · p. 10

B Algorithm summary

  • The one-pass algorithm samples a rollout group, computes retrieval-hit and answer-correctness signals, classifies rollouts, and segments tokens using the first-hit index.It then computes outcome advantages, applies eligible residuals, and takes a GRPO/GSPO step.Source: B Algorithm summary · p. 11

C Proofs

The proof expresses the aggregate advantage of retrieval-hit rollouts through the within-group grounding gap, and extends the result to general rewards via covariance. A separate bound ensures the residual preserves token-advantage signs.

C Proofs

  • For binary answer-correctness rewards, the aggregate advantage of retrieval-hit rollouts is proportional to the within-group grounding gap, with a positive prefactor.The proof derives Pr[C | H=1] − µ = (1 − pH)∆, and the general-reward form uses within-group covariance.Source: C Proofs · p. 11
  • For a general reward, the sum of advantages among retrieval-hit rollouts is n Cov(H, R)/σ; for R=C, Cov(H, R)=pH(1−pH)∆.Adding a small constant to σ or using the sample standard deviation changes only the positive prefactor.Source: C Proofs · p. 11
  • The residual bound keeps each token’s advantage sign unchanged because its magnitude is capped below one.The proof states that the token gradient is multiplied by 1 + ϵi,t.Source: C Proofs · p. 11

D.1 EDA observations on Search-R1 checkpoints

Search-R1 checkpoint profiling reveals redundant retrieval and growing misaligned token activity, motivating segment-specific residuals. The section defines how those residuals are routed, calibrated, and concentrated on tokens.

D.1 EDA observations on Search-R1 checkpoints

  • B1·RSEARCH rises from 25.18% to 39.37% across checkpoints while B1·VSEARCH falls from 27.97% to 24.59%, and type-D search and answer tokens also grow by STEP 900.Supported-correct A rollouts also show an overall rise in RSEARCH share from 26.81% to 32.91%.Source: §D.1 · pp. 11–12

D.2 Polarity and importance from per-segment diagnosis

  • Residual polarity follows each segment’s sub-outcome, while the largest magnitude targets the outcome bottleneck and valid search in failed-answer rollouts retains positive credit.These rules aim to suppress redundant search while protecting useful search.Source: §D.2 · p. 12, §D.1 · p. 12

D.3 Calibration constraints from observed failure modes

  • The residual is bounded by ηλmaxcmax ≤0.1; calibration also caps token salience at 0.5 and requires λ ratios to be set before η.Large negative residuals and uncapped token extremes produced entropy explosions and collapse in diagnostic runs.Source: §D.3 · p. 13

D.4 Sensitivity analysis

  • The s sweep measures sensitivity to the overall residual budget because scaling s strengthens every active residual without changing its polarity.Source: §D.4 · p. 14

E.1 Building blocks: sampled-token margin and Top-M entropy

  • Sampled-token margin and top-M entropy capture different uncertainty information: margin depends on the chosen token, while entropy measures distribution breadth.Together they recover information either signal alone discards.Source: §E.1 · p. 15

E.2 Positive salience c+

  • Positive salience weights low-margin or broad-distribution tokens, with a small interaction term boosting tokens where both signals are high.The reported coefficients are α=0.6, β=0.3, and γ=0.1.Source: §E.2 · pp. 15–16

E.3 Negative salience c−

  • Negative salience instead emphasizes above-median margins and peaked distributions, targeting confident commitments in undesirable trajectories.It mirrors the positive score with sign-flipped margin and complementary entropy.Source: §E.3 · p. 16

E.4 Why margin is necessary: empirical limitations of entropy alone

The section argues that margin and entropy provide complementary token-level signals and uses their additive composition to identify useful residual targets. Distributional and end-to-end sweeps support K=3 and M=32 as practical defaults.

E.4 Why margin is necessary: empirical limitations of entropy alone

  • At B1·ANS step 900, the high-entropy and low-margin tails are 16.2% and 19.9%, respectively, but overlap on only about 5% of tokens.Entropy contracts more sharply than margin on productive cells, while both remain informative on failure-mode cells.Source: §E.4 · pp. 17–18

E.5 Why the composition identifies the right tokens

  • An additive margin-plus-entropy score gives nonzero salience to tokens surfaced by either signal and boosts tokens where both signals fire.The composition avoids degenerating when one signal vanishes while keeping the component weights interpretable within a unit budget.Source: §E.4 · p. 18, §E.5 · p. 19

E.6 Robustness of K=3 and M=32

  • K=3 and M=32 are presented as practical defaults rather than load-bearing choices for identifying active-region tokens.The defaults balance competitor support and top-M probability-mass coverage.Source: §E.6 · p. 19, §E.6.1 · p. 19

E.6.1 Distributional stability of margin@K and entropy@M

  • Across 36 cells, the bottom-1% margin tail retains at least 0.75 overlap across the K sweep, while entropy-mean shifts are below 0.034 for M≥16.The reported analysis places M=32 on a stable plateau with M=64.Source: Table 14 · p. 19, §E.6.1 · p. 19

E.6.2 End-to-end ablation on K and M

  • Across K settings, EM ranges from 42.72 to 43.18; for M, the 16/32 plateau differs by at most 0.04 EM, while M=64 is 0.75 EM below M=32.The authors retain K=3, M=32 as the operating default.Source: Table 15 · p. 20, §E.6.2 · p. 21, Table 16 · p. 21

F Oracle-query retrievability

  • Oracle queries distinguish retrievable type-D rollouts, which receive an unsupported-correctness penalty, from corpus misses, which are masked.The conservative check treats an answer as retrievable if any rollout in its group has a hit; type-D rollouts account for 2–3% of training rollouts.Source: F Oracle-query retrievability · p. 21

G Reliability of the alias-based grounding signal

The alias-based grounding matcher has high recall on annotated HotpotQA supporting evidence for ordinary answers, but it does not establish precision on arbitrary retrieved passages. A single-seed, 100-step comparison without yes/no questions is only indicative.

G Reliability of the alias-based grounding signal

  • On annotated HotpotQA supporting documents, answer aliases are absent for 3.61% of questions, or 0.040% after excluding yes/no answers.The authors note that the remaining ordinary-answer absences are formatting artefacts; alias matching does not measure precision on arbitrary retrieved passages.Source: G Reliability of the alias-based… · p. 21
  • Removing yes/no questions improves in-domain NQ and HotpotQA EM by about 0.7 points after 100 steps, while out-of-domain performance is approximately unchanged.The authors qualify this as an indicative single-seed comparison.Source: G Reliability of the alias-based… · p. 21

H Ablation and learned-routing details

Ablations test the salience and segment-split components, while training and evaluation follow Search-R1 settings with token-level loss changes. A higher-KL Search-R1 control does not account for GCA’s gains.

H Ablation and learned-routing details

  • The w/o segment split ablation merges VSEARCH and RSEARCH into one search segment per type, fixing its polarity and magnitude before training from token-weighted signed and absolute residual means.Source: H Ablation and learned-routing details · p. 22

I Training and evaluation hyperparameters

  • Training uses the Search-R1 retrieval setup and recipe, modifying only token-level loss; GCA uses KL coefficient 0.005 versus 0.001 for reproduced Search-R1 runs.The setups use E5 over the 2018 Wikipedia dump, top-3 passages, and at most five turns.Source: I Training and evaluation… · p. 22

J Search-R1 KL control

  • Raising Search-R1’s KL coefficient to 5×10−3 does not improve its default-setting results, so higher KL alone does not explain GCA’s gains.The control keeps other settings unchanged.Source: J Search-R1 KL control · p. 22, Table 18 · p. 22

L BFCL v3 function-calling evaluation

On BFCL v3, GCA’s main improvement over Search-R1 comes from multi-turn function calling. The paper also compares GCA’s credit-assignment design and illustrates how its routing treats distinct rollout types.

M Design-space comparison

  • GCA uniquely combines token-level credit, separate search and answer credit, and single-pass end-to-end training without auxiliary models or extra rollouts.Across benchmarks, HITACC falls on NQ, TriviaQA, PopQA, and HotpotQA and rises on the other three; these changes alone do not indicate better or worse evidence use.Source: M Design-space comparison · p. 22, Table 19 · p. 23, Table 22 · p. 24

L BFCL v3 function-calling evaluation

  • 44.12% accuracy in multi-turn function calling with GCA, up from 41.12% with Search-R1 and 6.50 points above the base model.The evaluation follows the official BFCL v3 protocol across non_live, live, and multi_turn collections, reporting the official overall score.Source: L BFCL v3 function-calling evaluation · p. 22, Table 21 · p. 24

N.1 Type A: hit-correct (H=1, C=1)

  • A title-matched search retrieves the author of A Theft, and the agent correctly extracts Saul Bellow; both search and answer segments receive positive credit.This is the representative hit-correct Type A trajectory.Source: §N.1 · p. 25

N.2 Type B1: hit-wrong (H=1, C=0, valid wrong answer)

  • For the cue-ball question, retrieval finds the white-ball fact but the agent misreads evidence about a red object ball and answers incorrectly.The routing preserves the useful search while suppressing the wrong answer span.Source: §N.2 · p. 26

N.3 Type B2: hit-timeout (H=1, C=0, max-turn termination)

  • In the Casa Loma example, relevant evidence appears by the final search, but redundant searching exhausts the turn budget before an answer is emitted.The routing assigns a deliberately small negative residual to redundant search after a hit.Source: §N.3 · p. 26

N.4 Type C: miss-wrong (H=0, C=0)

  • When search returns articles about the wrong work Abandoned, the agent answers incorrectly; this miss-wrong trajectory receives no residual beyond GRPO..Source: §N.4 · p. 27

N.5 Type D: miss-correct (H=0, C=1, unsupported correctness)

  • Earth’s orbit direction is absent from retrieved passages, yet the agent answers correctly from memory; retrievable-but-missed evidence receives a soft negative residual, while non-retrievable cases are masked.Source: §N.5 · p. 27

Related papers

Loading 2610.23148v1…