PaperlayerView exact revision on arXiv

Source-linked AI summary

SHARPO: Segment-Level Credit Assignment for Agentic Reinforcement Learning

Xinchen Du, Zhengze Zhou, Wenhui Zhu, Han Yu, Sen Na, Rohit Jain, Alborz Geramifard

arXiv:2610.00838v1cs.LGcs.AI

TL;DR

Sparse terminal rewards offer little direct information about which intermediate decisions contributed to an agent’s outcome. SHARPO uses successful peer rollouts to apply teacher-informed credit adjustments within interaction segments. It achieves higher final validation success than the evaluated baselines on ALFWorld and WebShop at both tested mixing weights.

  • Problem

    In multi-turn tasks, terminal rewards provide little direct information about the contribution of each intermediate decision, so useful actions and local mistakes can receive the same trajectory advantage.

    Source: §1 · p. 1, §2.1 · p. 4

  • Method

    SHARPO reuses a successful peer rollout as teacher context and aggregates teacher–student log-probability feedback within interaction segments to adjust GRPO advantages.

    Source: §1 · p. 2

  • Results

    SHARPO has the highest mean final validation success on ALFWorld and WebShop at both mixing weights, gaining 13.80–14.32 points over GRPO on ALFWorld and 8.85–9.11 on WebShop.

    Source: §1 · p. 3, §3.2 · p. 7

  • Takeaways & Limitations

    The results support using meaningful interaction segments to organize self-distillation feedback for agent training.

    Source: §5 · p. 9

Abstract

from arXiv · show

Agentic reinforcement learning (RL) trains a large language model (LLM) to act over long, multi-step interactions. However, a single localized error can cause task failure, while trajectory-level rewards provide limited guidance for assigning credit to individual decisions. To address this limitation, we introduce Segment-level Hindsight Advantage Reweighting for Policy Optimization (SHARPO), a credit-assignment mechanism that refines Group Relative Policy Optimization (GRPO) at the level of environment-facing segments. Inspired by the existing on-policy self-distillation (OPSD) method, SHARPO computes teacher-student log-probability gaps within each segment and uses the resulting signal to compute a bounded multiplier on the GRPO advantage. This multiplier is shared by all tokens within the segment, allowing credit to vary across different segments. With Qwen2.5-7B-Instruct, SHARPO outperforms existing baselines on the ALFWorld and WebShop benchmarks, including GRPO, SDAR, RLSD, and StepOPSD.

1 Introduction

Sparse terminal rewards and GRPO’s shared trajectory advantage leave useful intermediate decisions difficult to distinguish from local mistakes. SHARPO uses successful-peer self-distillation to reweight credit at the level of interaction segments.

1 Introduction

  • With terminal outcome supervision, a trajectory receives one GRPO advantage, so a useful intermediate decision and a local mistake can receive the same credit.The introduction illustrates this with a shopping agent that finds a suitable product but selects an incorrect option.Source: §1 · p. 1
  • SHARPO outperforms GRPO, RLSD, SDAR, and StepOPSD on both benchmarks; Figure 1 reports 84.90% on ALFWorld (+14.32 points) and 75.26% on WebShop (+9.11 points) versus GRPO.The reported results are at both tested mixing weights.Source: Figure 1 · p. 2, §1 · pp. 2–3
  • SHARPO reuses a successful rollout from the same GRPO group as teacher context and uses teacher–student log-probability differences to adjust outcome-derived advantages.It supplements sparse outcome supervision without an external teacher or additional environment rollouts.Source: §1 · p. 2
  • SHARPO treats meaningful interaction spans, including reasoning, tool calls, and actions, as segments and applies one teacher-informed advantage weight uniformly within each segment.The segment-level feedback adjusts the strength of reinforcement or penalization for intermediate substeps.Source: §1 · p. 2
  • Unlike RLSD, SDAR, and StepOPSD, SHARPO aggregates teacher feedback within selected segments before assigning each segment a shared advantage weight.It considers favorable and unfavorable evidence across a substep jointly.Source: §1 · p. 2
  • The paper also provides a theoretical explanation of why self-distillation feedback is useful and how it supports finer-grained credit assignment.Source: §1 · p. 3

2 Method

Section 2 establishes that trajectory-wide GRPO advantages obscure local contributions, then develops SHARPO’s successful-peer, segment-level reweighting and analyzes its credit signal.

2.1 Preliminaries: GRPO and OPSD

  • GRPO shares its terminal-reward group advantage across all generated tokens; SHARPO adds peer-informed local feedback to refine segment credit under sparse rewards.The paper frames this as addressing the limited information terminal rewards provide about intermediate decisions.Source: §2.1 · pp. 3–4

2.2 SHARPO

  • SHARPO selects coherent, nonempty disjoint segments and evaluates them with a policy-copy teacher conditioned on a successful peer rollout.The teacher’s support can vary across segments in the same failed trajectory, although those segments share the same outcome-derived advantage.Source: §2.2 · pp. 4–5
  • A length-normalized teacher–student gap becomes a bounded segment weight, then length balancing and interpolation with the base advantage set the effective advantage.These adjusted advantages are used in the policy update.Source: §2.2 · p. 5

2.3 Understanding Segment-Level Credit Assignment

  • Under an ideal success-conditioned reference, hindsight feedback aligns with a segment’s local value and does not worsen credit-assignment errors for any ρ > 0.The analysis identifies a credit-assignment error when a useful decision in a failed rollout receives negative credit.Source: §2.3 · p. 6

3 Experiments

The experiments evaluate SHARPO on ALFWorld and WebShop with Qwen2.5-7B-Instruct, comparing final validation success and finer-grained task performance against RL and self-distillation baselines.

3.1 Setup

  • Experiments fine-tune Qwen2.5-7B-Instruct on ALFWorld and WebShop, which use binary terminal success rewards, and evaluate step-200 success on 128 held-out tasks.The reported metric is the mean and sample standard deviation over three independent runs.Source: §3.1 · p. 6

3.2 Main results

  • SHARPO gains 13.80–14.32 percentage points over GRPO on ALFWorld and 8.85–9.11 on WebShop, while beating StepOPSD by 6.77 and 5.99–7.55 points, respectively.It achieves the highest mean final validation success rate on both benchmarks at both tested mixing weights.Source: Table 1 · p. 7, §3.2 · p. 7
  • Across three runs, changing λ fourfold shifts SHARPO’s mean success rates by only 0.52 points on ALFWorld and 0.26 on WebShop; variability is lower than GRPO’s on ALFWorld and comparable on WebShop.ALFWorld standard deviations are 2.82 and 1.19 for SHARPO versus 7.83 for GRPO; WebShop values are 1.19 and 1.35 versus 1.19.Source: §3.2 · p. 7

3.3 Fine-grained analysis

  • On Pick2, SHARPO exceeds the strongest baseline by 7.02 points at λ = 0.2 and 10.10 at λ = 0.05; on Cool by 6.89 at λ = 0.2; on Heat by 6.67 at λ = 0.05 over SDAR.Source: §3.3 · p. 7
  • On ALFWorld task families, SHARPO beats all evaluated baselines on Pick, Clean, Cool, and Pick2 at both mixing weights and is within 0.30 points of the best baseline on Look.The task-family rates are averaged separately; the gains persist across mixing weights differing fourfold.Source: §3.3 · p. 7, Table 2 · p. 8
  • WebShop task score, which credits partial product matches, exceeds StepOPSD by 0.0591 at λ = 0.2 and 0.0212 at λ = 0.05, with both settings also exceeding GRPO.The higher scores indicate better average satisfaction of shopping requirements alongside higher success rates.Source: Table 3 · p. 8, §3.3 · p. 8

4 Related Work

Prior work assigns credit through critics, token-level methods, or other feedback structures. SHARPO retains GRPO’s trajectory advantage while scoring successful-peer-conditioned segments for local credit.

4 Related Work

  • Related credit-assignment approaches use utterance-level advantages, recurring states, Monte Carlo resampling, process reward models, or role-typed segments.The cited examples include ArCHer [Zhou et al., 2024], TRIAGE [Xu et al., 2026], and POAD.Source: §4 · pp. 8–9
  • SHARPO keeps GRPO’s critic-free trajectory advantage and uses student and successful-peer-conditioned teacher policies to score sampled segments, producing bounded shared segment weights without critic fitting or temporal-difference backups.The approach also avoids external judges and additional environment rollouts.Source: §4 · p. 9
  • Unlike RLSD’s token-level advantage reweighting, SDAR’s auxiliary distillation-loss gating, and StepOPSD’s token-specific action-span weights, SHARPO aggregates feedback within each selected segment.RLSD [Yang et al., 2026]; SDAR [Lu et al., 2026]; StepOPSD [Zhang et al., 2026].Source: §4 · p. 9

5 Conclusion

SHARPO refines GRPO credit with successful-peer feedback aggregated over intermediate interaction segments. It improves final validation success over GRPO and evaluated self-distillation baselines on ALFWorld and WebShop at both tested mixing weights.

5 Conclusion

  • SHARPO improves final validation success over GRPO and evaluated self-distillation baselines on ALFWorld and WebShop at both tested mixing weights.The results support using meaningful interaction segments to organize self-distillation feedback for agent training.Source: §5 · p. 9

A Theory of segment-level credit refinement

The analysis fixes the sampled trajectory, base advantages, teacher feedback, and segment weights during each policy update. It considers one selected segment of positive length using the old policy.

A Theory of segment-level credit refinement

A.1 Value-aligned credit correction

The proof establishes that the correction aligns with Dπ(c, a) when nonzero and, in a failed rollout, changes penalty strength without changing its negative sign.

A.1 Value-aligned credit correction

  • The function u(z) = clip(2σ(z) − 1, −α, α) is odd, preserves sign, and satisfies |u(z)| ≤ α < 1.Source: §A.1 · p. 11
  • The correction λρ|A|u(∆⋆) has the direction of Dπ(c, a) whenever it is nonzero.When Qπ(c, a) = 0, the same conclusion holds with u(∆⋆) = −α.Source: §A.1 · p. 12
  • For A < 0 and λ > 0, B remains negative; D > 0 gives B < eA⋆ < 0, while D < 0 gives eA⋆ < B < 0.These inequalities establish weaker and stronger penalties at the same length factor.Source: §A.1 · p. 12

A.2 Computing segment feedback

Segment feedback is computed from teacher–student conditional token log-probability gaps evaluated at a shared sampled prefix, then averaged over the segment.

A.2 Computing segment feedback

  • The length-normalized segment gap in Eq. (5) equals the average of its conditional token gaps.The sampled segment has m > 0 tokens, and both policies assign it positive likelihood under their respective contexts.Source: §A.2 · p. 12
  • Each token gap δj subtracts the student log probability under ci,k from the teacher log probability under ci,k ⊕ hi, using the same prefix z<j.The student is evaluated at θold and the teacher is additionally conditioned on hi.Source: §A.2 · p. 12

A.3 Length-balancing rationale

Length balancing equalizes the reweighted-component coefficient across equally weighted segments under unclipped length factors, while leaving the base component unchanged.

A.3 Length-balancing rationale

  • With all length factors unclipped, equally weighted segments receive the same accumulated reweighted-component coefficient regardless of segment length.This follows from ρi,k = ¯mi/mi,k.Source: §A.3 · p. 12
  • Length balancing leaves the base component unchanged and does not balance the full update or segment gradient norms.Source: §A.3 · p. 12
  • When all wi,k = 1, the total advantage coefficient is unchanged; clipping bounds wi,k to [1 − α, 1 + α] and the common multiplier to [1 − λα, 1 + λα].The numerator ¯mi restores the reweighted component’s overall coefficient scale compared with using 1/mi,k alone.Source: §A.3 · pp. 12–13
  • Clipped length factors may break the coefficient equalities, but the factors remain positive and bounded, so eAi,k preserves the sign of Ai.eAi,k = 0 when Ai = 0.Source: §A.3 · p. 13
Loading 2610.00838v1…