PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Credit: Credit Assignment for Long-Horizon Agentic RL via Intra-group Rescaling

Fangcong Yin, Ming Yin, Tanvi Bhandarkar, Aashu Singh, Eryk Helenowski, Shlok Kumar Mishra, Saghar Hosseini, Karthik Abinav Sankararaman, Greg Durrett, Dinkar Juyal

arXiv:2610.28388v1cs.LG

TL;DR

Long-horizon RLVR agents face sparse outcome rewards, making credit assignment across many turns challenging. Self-Credit generates cited-turn credit from summaries and refines it across rollouts before reweighting advantages. It outperforms strong agentic RL baselines on BrowseComp-Plus, GAIA, and xBench-DeepSearch.

  • Problem

    Sparse outcome rewards in long-horizon agentic RL provide coarse learning signals across many turns, while additional dense supervision is expensive to define and collect.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    Self-Credit uses policy-generated summaries citing critical turns as credit, refines that credit across successful and failed rollouts, and reweights advantages for cited-turn tokens.

    Source: §1 · p. 2

  • Results

    Self-Credit beats the strongest agentic RL baselines by 8.5 points on Qwen3-8B and 5.6 points on Qwen3-32B on BrowseComp-Plus, and by 4.9 points on GAIA and 4.7 points on xBench-DeepSearch.

    Source: §1 · p. 2, §5 · p. 7

  • Takeaways & Limitations

    Analyses find self-generated citations identify actions related to gold answers, while intra-group rescaling—especially on failed rollouts—is important across model scales.

    Source: §1 · p. 2

    The rescaling analysis assumes some turns can be matched across rollouts and that different turns contribute independently to the final outcome.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

LLM agents are commonly trained with RLVR, but the sparse nature of typical RLVR rewards makes credit assignment challenging, especially for long-horizon tasks. We propose Self-Credit, an end-to-end agentic RL method that creates dense supervision signals from sparse outcome rewards using self-generated credit assignment via intra-group rescaling. Self-Credit builds on group-based RL with three components. First, during its rollouts, the policy model generates summaries with citations to earlier turns containing critical evidence, which serve as self-generated credit. Second, we refine this model-generated credit with intra-group rescaling, which compares cited actions across successful and failed rollouts for the same prompt. This approach uses group-level agreement and disagreement to remove undeserved credit and further emphasize credit assigned to critical actions. Third, advantage reweighting uses the rescaled credit weights to update the advantages of the cited turns. On BrowseComp-Plus, Self-Credit improves over the base models by 37.0 and 36.2 points on Qwen3 8B and 32B respectively, and outperforms the strongest agentic RL baselines by 8.5 and 5.6 points, respectively. Self-Credit also improves over the strongest baselines by 4.9 points on GAIA and 4.7 points on xBench-DeepSearch. Analyses show that self-generated credit provides useful signals for selecting critical actions, and that intra-group rescaling, especially on failed rollouts, is important across model scales.

1 Introduction

Long-horizon agents receive sparse outcome rewards, making turn-level credit assignment challenging. Self-Credit generates and refines turn-level credit, and outperforms strong agentic RL baselines across long-horizon tasks.

1 Introduction

  • RLVR commonly trains LLM agents with sparse rewards, while adding dense supervision is expensive to define and collect.Existing sparse-reward credit-assignment methods include external judges and learned critics.Source: §1 · p. 1
  • 8.5 points on Qwen3-8B and 5.6 points on Qwen3-32B; on deep research, 4.9 points on GAIA and 4.7 points on xBench-DeepSearch over the strongest baselines.On BrowseComp-Plus, Self-Credit also improves over the base agents by 37.0 and 36.2 percentage points, respectively.Source: Figure 1 · p. 1, §1 · p. 2
  • Self-Credit uses summaries citing critical earlier turns as self-generated credit, refines that credit across successful and failed rollouts, and reweights advantages of cited-turn tokens.The method creates dense supervision from a single sparse outcome reward.Source: §1 · p. 2
  • Ablations find intra-group rescaling, especially downweighting repeated turns shared across failed rollouts, important across model scales; self-generated citations identify actions related to gold answers.The rescaling rules are also applied selectively during training at higher selection quality than baselines.Source: §1 · p. 2

2 Preliminaries

The preliminaries formulate long-horizon agent interaction as a sequence of turns ending in a binary-rewarded answer. GRPO broadcasts a rollout advantage across its tokens, motivating turn-level credit assignment.

Long-horizon agentic tasks

  • A trajectory contains interaction turns with a thinking trace, action, and observation, followed by a final-answer turn receiving a binary reward.The agent state comprises the query and interaction history.Source: §2 · p. 2

Multi-turn RL with GRPO

  • GRPO generates multiple trajectories for the same query, assigns each a trajectory-level advantage, and uniformly applies it to every model-generated token.The rollout group shares a prompt while trajectories receive individual rewards.Source: §2 · p. 2

Credit assignment

  • Figure 2 depicts self-compaction with turn citations, intra-group rescaling across successful and failed rollouts, and advantage reweighting.Shared cited turns lose extra credit, success-only turns gain positive credit, and failure-only turns receive stronger negative credit.Source: Figure 2 · p. 3
  • Uniform rollout advantages provide coarse credit because all turns share the same signal, while longer trajectories require one outcome reward to supervise more turns.The paper focuses on non-uniform, turn-level advantages for credit assignment.Source: §2 · p. 3

3 Methods

Self-Credit creates turn-level learning signals from sparse outcome rewards by combining policy-generated summaries with citations, comparisons across rollouts, and advantage reweighting.

3 Methods

  • Self-Credit refines self-generated turn citations through within-prompt comparisons, then uses the resulting weights to reweight advantages.The method aims to create dense supervision from sparse outcome rewards.Source: §3 · p. 3

3.1 Self-Compaction with Turn Citations

  • Compaction replaces the interaction history with a compact summary when context reaches threshold C, and the agent continues from the query and summary.Compaction can recur as context grows, dividing a trajectory into segments.Source: §3.1 · pp. 3–4

3.2 Intra-group Rescaling

  • Matched turns cited across rollouts are rescaled according to outcomes: repeated success strengthens positive credit, repeated failure strengthens negative advantages, and mixed outcomes remove the success upweighting.Matching assumes some turns recur across rollouts; summary and final-answer turns are excluded from comparisons.Source: §3.2 · pp. 4–5

3.3 Advantage Reweighting

  • The citation set replaces uniform advantage broadcasting with turn-level weights, while successful-rollout summary and final-answer turns automatically receive credit.All generated tokens in a turn receive the same reweighted advantage.Source: §3.3 · p. 5

3.4 Annealing the Weights

  • The repeated-failure penalty weight is annealed: stronger early penalties discourage repeatedly unsuccessful retrieval actions, while weaker later penalties allow more exploration.Source: §3.4 · p. 5

4 Experimental Setup

The experiments cover long-horizon multi-hop question answering and deep research tasks, compare Self-Credit with zero-shot and agentic RL baselines, and evaluate answers using LLM-as-a-judge accuracy.

4.1 Setup

  • Evaluation covers BrowseComp-Plus, GAIA's text-only subset, and xBench-DeepSearch, with Qwen3-8B, Qwen3-32B, and Gemma4-E4B trained in thinking mode.Training uses 747 BrowseComp-Plus examples and 1K deep-research QA pairs; the BrowseComp-Plus test split has 83 examples.Source: §4.1 · pp. 5–6

4.2 Baselines

  • Baselines include zero-shot and GRPO controls, matched-action GiGPO, uniform-credit SUPO, and ECHO's selected-turn credit on successful rollouts.Source: §4.2 · p. 6

4.3 Evaluation Protocol

  • BrowseComp-Plus uses a static corpus and Qwen3-Embedding-8B retriever, while deep-research evaluation uses open-web search; both report LLM-as-a-judge accuracy.Decoding temperature is 0.7 for both task groups.Source: §4.3 · p. 6

5 Results

Self-Credit outperforms the evaluated RL baselines on BrowseComp-Plus at both Qwen3 scales and on both deep-research benchmarks. Its training dynamics also balance useful exploration with restrained tool use.

5 Results

  • Self-Credit beats the strongest BrowseComp-Plus baseline by 8.5 points at Qwen3-8B and 5.6 points at Qwen3-32B, and leads the strongest baselines by 4.9 points on GAIA and 4.7 on xBench-DeepSearch.The BrowseComp-Plus results are test accuracy (Pass@1); the deep-research results use Gemma4-E4B.Source: §5 · pp. 6–7, Table 2 · p. 7, Table 3 · p. 7
  • On BrowseComp-Plus, Self-Credit outperforms strong RL baselines across model scales; Table 2 reports test accuracy (Pass@1) and average turns, with results over three seeds.Source: Table 2 · p. 7
  • Across training, Self-Credit typically uses fewer turns than ECHO later on while maintaining high reward; GRPO and GiGPO use shorter trajectories and achieve lower reward.The paper concludes that simply minimizing or maximizing interaction length is insufficient for long-horizon tasks.Source: §5 · p. 7, Figure 3 · p. 7

6 Analysis

The analyses find that both self-generated citations and intra-group rescaling contribute to performance, with repeated turns in failed rollouts especially important. Self-Credit also yields stronger tool-use and citation-quality signals, while rescaling is applied selectively.

6.1 Ablations

  • Removing penalties for repeated turns cited by failed rollouts reduces accuracy from 47.0% to 34.9% for Qwen3-32B and from 40.6% to 26.5% for Qwen3-8B.Without these penalties, Qwen3-8B searches excessively and fails to commit to an answer; crediting all or random turns reduces its accuracy to 27.7% and 32.5%, respectively.Source: §6.1 · p. 8, Table 4 · p. 8
  • The ablations show that self-generated citations and intra-group rescaling work together, with rescaling repeated turns in failed rollouts especially important across model scales.Applying credit to arbitrary turns performs worse than using model-cited turns.Source: §6.1 · p. 8

6.2 Quality of Credit Assignment

  • Self-Credit achieves higher Gold Searched, Gold Opened, and Answer Recall scores than baselines at both model sizes.These metrics assess retrieval of gold documents and documents containing the gold answer or supporting evidence.Source: §6.2 · pp. 8–9, Table 5 · p. 8
  • Self-Credit citations score higher in citation-quality metrics than ECHO and the random- and all-turn baselines, especially for identifying critical search turns.The evaluation compares cited turns with gold-document Search or Open actions and turns judged important by Claude-4.6-Sonnet.Source: §6.2 · p. 9, Table 6 · p. 9
  • Intra-group rescaling is triggered selectively during training, and its precision is assessed by whether it credits gold-document-related actions while reducing credit for less relevant ones.Figure 4 reports firing rate and firing precision across three rules and different training stages.Source: §6.2 · pp. 9–10, Figure 4 · p. 9

7 Related Work

Prior work addresses long-horizon agent challenges through inference-time methods, context management, self-compaction, and varied approaches to credit assignment. Self-Credit combines self-compaction with group-level statistics already available in standard group-based RL.

Agentic RL for long-horizon tasks

  • Long-horizon interaction histories can exceed the context window, while sparse outcome rewards give limited information about individual actions.Prior approaches include inference-time methods, context management systems, and policy self-compaction learned through RL.Source: §7 · p. 10

Credit assignment under sparse outcome rewards

  • Prior sparse-reward credit-assignment methods use learned critics, process reward models, answer-conditioned hindsight, sampling, rollout comparisons at similar states, or policy confidence.Some methods require additional rollouts or hindsight information unavailable at intermediate steps.Source: §7 · p. 10
  • Self-Credit combines standard group-based RL signals with self-compaction and rescales them using group-level statistics.This positions the method alongside prior work that reuses sampled rollouts or leverages policy confidence.Source: §7 · p. 10

8 Conclusion

Self-Credit creates dense supervision for long-horizon agentic RL by using cited summaries and intra-group rescaling to refine turn-level credit. The appendix notes that GRPO’s token-level importance ratios do not provide turn-specific credit.

8 Conclusion

  • Self-Credit uses cited summaries and intra-group rescaling to create dense supervision from sparse outcome rewards, and outperforms strong agentic RL baselines on challenging search QA and deep research tasks across models.The policy model cites critical turns during compaction; the method compares cited actions across successful and failed rollouts and selectively strengthens advantages for cited turns.Source: §8 · p. 10

A.1 GRPO Formulation

  • The GRPO formulation presents an advantage equation and objective, with epsilon greater than zero specified as a numerical-stability constant.Source: §A.1 · p. 15
  • Token-level importance ratios and clipping can yield different gradient contributions but do not assign turn-specific credit based on each action’s contribution to the outcome.Source: §A.1 · p. 15

B Implementation Details

The implementation prompts the policy to cite decisive turns in summaries and uses task-specific action-observation matches to rescale credit across rollouts. It also assumes turns contribute independently to outcomes.

B.1 Summarization Instruction

  • The summarization instruction asks the model to cite only a few decisive assistant turns, usually 2 to 5, using turn numbers counted from the first assistant tool-calling turn.Tool responses and prior summary blocks are excluded from numbering; invalid or empty citation lines trigger a format error and give the rollout reward 0.Source: §B.1 · p. 15
  • A completed rollout receives an auxiliary summary of its final segment because compaction thresholds might otherwise leave short final segments uncited.This auxiliary summary neither replaces context nor enters N_j.Source: §B.1 · p. 15

B.2 Turn Matching

  • Turn matching compares the evidence retrieved by actions rather than tool-call text, which rarely repeats verbatim across rollouts.The method focuses on matching actions for the same prompt across rollouts.Source: §B.2 · p. 16
  • For BrowseComp-Plus, search turns match when retrieved-docid sets have Jaccard similarity of at least 0.6, while open_page turns match only for the same docid.Source: §B.2 · p. 16
  • For deep research, searches match by retrieved-docid Jaccard similarity of at least 0.6, opens by docid, and finds by adjacent lines in the same document.Docids are used because links and cursors are local to each rollout.Source: §B.2 · p. 16
  • The matching recipe can extend to other agentic tasks when a task-specific function matches turns by their observations; unlike GiGPO, Self-Credit matches observations obtained by cited actions.The passage gives embodied tasks such as ALFWorld as an example setting.Source: §B.2 · p. 16
  • The method approximates turn contributions as independent, although actions can depend on earlier turns, because trajectory-prefix comparisons would rarely match.Source: §B.2 · p. 16

C Training Configurations

The training configurations specify shared optimization, batching, sequence, and rollout settings, alongside baseline and Self-Credit-specific hyperparameters.

C.1 General Hyperparameters

  • Training uses a 1 × 10−6 learning rate, 256 trajectories per step across 16 prompts and 16 rollouts per prompt, and a 16,384-token compaction threshold.The maximum prompt and response lengths are 4,096 and 32,768 tokens; rollouts allow up to 100 assistant turns, and tool responses are truncated to 16K characters.Source: §C.1 · p. 17
  • BrowseComp-Plus training allows five parallel tool calls for 115 steps, while deep research allows one parallel tool call for 100 steps; baseline methods use official implementations.Source: §C.1 · p. 17, §C.2 · p. 17

C.2 Hyperparameters of Baselines

  • Baseline settings include ECHO turn-selection top-k of 5 with three recent turns retained verbatim, while GiGPO uses γ = 0.95 and groups by raw-observation similarity.Source: §C.2 · p. 17

C.3 Hyperparameters of Self-Credit

  • Self-Credit uses at most 10 summary rounds, β = 1.5, and w_s = 2.5, while w_f is 1.5 and anneals linearly to 1.0.Source: §C.3 · p. 17

D.1 Out-of-distribution Generalization

Self-Credit achieves the highest average accuracy in zero-shot transfer to four out-of-distribution search-augmented QA benchmarks after long-horizon training.

D.1 Out-of-distribution Generalization

  • 46.6% average accuracy for Self-Credit exceeds GRPO at 45.4%, ECHO at 44.5%, and SUPO at 44.3% on four out-of-distribution search-augmented QA benchmarks.The evaluation uses zero-shot Qwen3-32B policies trained on long-horizon tasks.Source: §D.1 · p. 17

E Details of Analysis

The analysis section defines metrics for tool-use quality, citation quality, and rescaling-rule behavior. The output example shows how the agent summarizes context and cites decisive earlier turns between context windows.

E Details of Analysis

  • Gold Searched and Gold Opened track questions with at least one gold document retrieved or opened, while Answer Recall averages the fraction of gold documents retrieved or opened.Evidence Recall applies the same fraction-based measure to evidence documents.Source: E Details of Analysis · pp. 17–18
  • Citation quality compares cited turns, excluding the final answer, with gold-document-finding turns and with turns judged decisive or newly informative by Claude Sonnet 4.6.The two measures are F1 scores; the judge-based metric uses the LLM-as-judge prompt from Xu et al. (2026).Source: E Details of Analysis · p. 18
  • Rule firing rate is the fraction of cited turns receiving a rescaling rule, and firing precision checks fired turns against whether they find gold documents.For repeated success, precision is the share of fired turns that find gold documents; for repeated failure and mixed outcome, it is the share that do not.Source: E Details of Analysis · p. 18

G Output Example

  • At each context-window end, the example agent summarizes the query and useful findings, cites decisive earlier turns, then restarts with the system prompt, query, and summary.The example is a Qwen3-8B model trained with Self-Credit on BrowseComp-Plus.Source: G Output Example · p. 18
  • The example BrowseComp-Plus query asks for a viral song matching biographical and release clues, and gives “Mando” as the ground-truth answer.Thinking traces and tool outputs are truncated to their first two sentences, while summaries and citation lines are shown in full.Source: G Output Example · pp. 18–19

Related papers

Loading 2610.28388v1…