PaperlayerView exact revision on arXiv

Source-linked AI summary

TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learning

Yuanda Xu, Zhengze Zhou, Hejian Sang, Xiaomin Li, Jiaxin Zhang, Xinchen Du, Sen Na, Zhipeng Wang, Alborz Geramifard

arXiv:2606.32017v2cs.LGcs.AI

TL;DR

Outcome-only GRPO cannot assign appropriate credit to heterogeneous agentic actions, especially useful exploration in failures and regression in successes. TRIAGE adds structured role typing and bounded role-conditioned corrections while retaining verifier outcomes as the optimization direction, improving success across three benchmarks and two policy models.

  • Problem

    Standard GRPO broadcasts final verifier outcomes uniformly across agentic action tokens, leaving local credit assignment for heterogeneous environment-facing segments unresolved.

  • Method

    TRIAGE uses a structured judge to classify segments as decisive progress, useful exploration, no-progress infrastructure, or regression, then applies fixed bounded role-conditioned rewards to GRPO advantages.

  • Results

    TRIAGE consistently improves over GRPO across ALFWorld, Search-QA, and WebShop for two policy models and also outperforms scalar judge-derived rewards and an outcome-supervised value baseline.

  • Takeaways & Limitations

    Role typing corrects outcome-only credit’s two blind spots while preserving verifier outcomes as the optimization direction.

  • Takeaways & Limitations

    Role labels are semantic estimates rather than ground truth, and role-aware credit improves local attribution without providing causal identification.

Abstract

from arXiv · show

Agentic reinforcement learning requires assigning credit to environment-facing actions such as searches, clicks, edits, navigation commands, and object interactions. Standard GRPO uses the final verifier outcome as a uniform advantage over all action tokens. This outcome signal is useful but structurally incomplete: it punishes useful exploration in failed rollouts and reinforces redundant or regressive actions in successful rollouts. We propose TRIAGE, a role-typed credit assignment framework that adds a semantic role axis to outcome credit. A structured judge classifies each segment as decisive progress, useful exploration, no-progress infrastructure, or regression, and a fixed role-conditioned rule maps these labels to bounded segment-level process rewards. This keeps verifier outcomes as the source of optimization direction while correcting the two main blind spots of outcome-only credit. We further show that role-conditioned credit is the optimal segment-level correction expressible from role labels alone -- a projection of the per-segment advantage residual onto the role variable -- so that the fixed role constants reduce advantage estimation error whenever the judge is reliable, and we connect this to lower-variance policy gradients. Across ALFWorld, Search-QA, and WebShop, TRIAGE improves success rates over GRPO for two policy models and outperforms both a scalar judge-derived process reward and an outcome-supervised shared-backbone value baseline. Ablations show that the gain comes from role typing rather than merely adding dense rewards: reliable detection of regression inside successful trajectories is the dominant contributor, while exploration credit provides a consistent secondary gain; on completed ALFWorld and WebShop rollouts, TRIAGE also reduces environment-facing turns by an additional $10.4\%$ and $14.8\%$ relative to GRPO.

1 Introduction

Agentic RL changes credit assignment because final verifier outcomes are broadcast across heterogeneous environment-facing actions. TRIAGE adds semantic role typing while retaining GRPO outcomes as the optimization direction.

  • Results: Across two policy models and ALFWorld, SearchQA, and WebShop, TRIAGE consistently improves over GRPO and supports gains attributed to role-conditioned credit.The evaluation also compares TRIAGE with scalar judge-derived rewards and an outcome-supervised value baseline.
  • Motivation: Agentic trajectories contain environment-facing segments including searches, clicks, edits, navigation commands, and object interactions.These segments can change external state or the agent’s information state.
  • Motivation: Outcome-only credit punishes useful exploration in failed rollouts and reinforces redundant or harmful actions in successful rollouts.Both blind spots arise because all segments inherit the same trajectory-level outcome signal.
  • Method: TRIAGE assigns each segment one of four roles: decisive progress, useful exploration, no-progress infrastructure, or regression.Role labels distinguish semantic function rather than merely the amount of immediate progress.
  • Method: TRIAGE uses a structured judge for role diagnosis while preserving the GRPO outcome advantage as the source of optimization direction.The judge adds bounded segment-level process rewards or penalties rather than replacing verifier supervision.

2 Problem Setup: Segment Credit in Agentic RL

The problem setup treats environment-facing segments as the unit of agentic credit assignment. Standard GRPO broadcasts one trajectory advantage uniformly, motivating a role-sensitive segment advantage.

  • Outcome Credit: GRPO samples multiple trajectories, scores them with a verifier, and broadcasts each trajectory’s group-normalized advantage to every token.Training binarizes raw success rewards to ri ∈ {0, 1}.
  • Segment Credit: An agentic trajectory consists of environment-facing action segments and their resulting observations.Segment-level credit is therefore finer-grained than trajectory-level outcome credit.
  • Problem: Uniform broadcasting treats decisive purchases, diagnostic reads, harmless no-ops, and wrong edits identically.The setup seeks a segment advantage that reflects both action quality and semantic role.
  • Problem: Table 1 frames success or failure as the trajectory-level direction and local segment role as the determinant of whether that direction should transfer unchanged.Table 2 supplies the role taxonomy used for this distinction.

3 Why Outcome Credit Is Structurally Incomplete

Outcome credit is structurally incomplete because it cannot distinguish local semantic roles within successful or failed trajectories. The key distinction is that useful exploration changes information state without necessarily completing a task subgoal.

  • Outcome Axis: Outcome credit gives every local decision the same sign within a successful or failed rollout.This creates conflict when local segment value differs from the final trajectory outcome.
  • Role Boundaries: Useful exploration improves the information state, whereas decisive progress changes verifier-checkable task state.Exploration can enable later decisive actions despite producing no immediate subgoal completion.
  • Role Boundaries: No-progress infrastructure changes neither task state nor information state and should not receive positive progress credit.Examples include empty traversal and generic commands that do not affect the next decision.
  • Role Boundaries: Regression is locally harmful or redundant without information gain, including wrong edits, wrong purchases, corrupted state, and repeated known inspections.It can occur in successful trajectories after later recovery.
  • Diagnostics: The judge’s key capability is finding regressions in successful rollouts and useful segments in failed rollouts.Perfect agreement on decisive-progress versus exploration boundaries is not required for this asymmetric correction.

4 TRIAGE: Role-Conditioned Segment Credit

TRIAGE adds structured role diagnosis and fixed role-conditioned process rewards to standard GRPO segment advantages. Its theoretical justification treats role conditioning as the optimal correction available from role labels alone.

  • Framework: TRIAGE splits rollouts into environment-facing segments, assigns each a structured semantic role, and adds a bounded role-dependent process reward to GRPO.The policy update remains the standard GRPO update, and the judge is not used at evaluation time.
  • Credit Rule: The role-conditioned advantage combines the trajectory outcome advantage with a fixed role reward scaled by λ.Auxiliary judge scores help choose the role label but are not additional training-time reward terms.
  • Credit Rule: Decisive progress receives +1, useful exploration +0.5, no-progress infrastructure −0.1, and regression −0.5.These constants remain fixed across tasks; λ is the only tuned hyperparameter.
  • Optimization: The resulting segment advantages are whitened within each batch before being broadcast to tokens in the corresponding segment.Prompt and observation tokens are excluded from the policy loss.
  • Theory: Under a sufficiency assumption, the conditional expectation of the credit residual given role is the MSE-optimal role-measurable correction.Role-conditioned credit reduces estimation error when fixed constants align with this optimum.
  • Theory: Role labels help when they explain nonzero credit residual, with regression requiring negative correction and exploration in failed rollouts positive correction.This connects the taxonomy to the two outcome-credit conflict cells.

5 Experiments

Experiments evaluate TRIAGE across three agentic benchmarks, policy models, judge variants, and credit-assignment baselines. Results show gains depend on reliable role typing, especially regression detection in successful trajectories, while exploration credit adds a smaller consistent benefit.

  • 5.2 Main Results: TRIAGE improves over GRPO on all three benchmarks for both evaluated policy models, with the largest gains on ALFWorld and WebShop.The strongest gains occur in the audited environments with the highest regression mass.
  • 5.2 Main Results: Replacing the thinking judge with a no-think judge drives TRIAGE below GRPO on ALFWorld and WebShop for both policies.Removing evidence keeps TRIAGE above GRPO but trails the default prompt, indicating that judge reliability is central.
  • 5.3 Does the Judge Recover the Conflict Cells?: 88.1% raw agreement was reached between two annotators labeling 135 environment-facing segments, with adjudication producing ground truth.The audit covers 18 logged trajectories across ALFWorld, WebShop, and Search-QA.
  • 5.3 Does the Judge Recover the Conflict Cells?: Thinking raises regression-in-success F1 from roughly 24 to 82, while exploration-in-failure exceeds 82 even without thinking.The difficult conflict cell is detecting regression where the verifier says the rollout succeeded.
  • 5.4 Comparisons and Ablations: The shared-backbone value baseline improves ALFWorld from 79.6 to 85.2 and Search-QA from 43.3 to 46.8, but WebShop changes only from 70.1 to 70.8.TRIAGE reaches 77.2 on WebShop because role typing distinguishes productive clicks from redundant repeats.
  • 5.4 Comparisons and Ablations: The scalar process-reward baseline improves over GRPO but remains below TRIAGE on every benchmark, showing that role-specific mapping matters beyond dense judge feedback.The ablation controls for judge access, local context, and dense reward shaping.
  • 5.4 Comparisons and Ablations: Zeroing the regression penalty costs 1.8–6.1 points across benchmarks, while the exploration bonus contributes a consistent 0.6–1.7 points.Completed-rollout length also falls by 10.4% on ALFWorld and 14.8% on WebShop relative to GRPO.

6 Discussion and Limitations

TRIAGE’s benefits depend on semantic role estimates that are context-sensitive and imperfect. The framework remains auditable and extensible, but role-aware credit improves attribution rather than establishing causal necessity.

  • Limitations: Role labels are semantic estimates rather than ground truth, so judges may overvalue exploration, miss subtle regressions, or follow final outcomes.Keeping verifier outcomes as the base signal mitigates but does not eliminate judge error.
  • Limitations: Role usefulness depends on local state and redundancy, because the same search, read, or test can be informative once and redundant later.Action strings alone are insufficient for classification.
  • Limitations: Role-aware credit improves local attribution but is not causal identification; counterfactual environment interventions would be needed to establish necessity.This limits the interpretation of segment-level credit as an explanation of task success.
  • Future work: A soft role distribution could represent mixed segments, but it would require reliable calibration and stronger audit procedures.The proposed extension replaces one primary role with probabilities over decisive progress, exploration, infrastructure, and regression.
  • Future work: TRIAGE can combine segment bucketing with outcome-statistical estimators, using bucketing for shared evidence and roles for interpreting that evidence.This is proposed for settings with sparse action arguments and rare repeated segments.
  • Future work: On harder tasks or stronger agents, the framework may shift from coarse failure detection toward finer-grained progress or belief-state rewards within each role.The discrete four-role label is described as a first layer of role-aware judging.

7 Related Work

Related work assigns credit through matched states, scalar process scores, outcome statistics, token weighting, or LLM judgments. TRIAGE differs by operating on environment-facing segments and adding semantic role interpretation while retaining outcome credit.

  • Agentic credit assignment: State-anchored methods compare actions from matched states, whereas stepwise process methods learn scalar dense scores for intermediate steps.These approaches represent structural and scalar alternatives in agentic credit assignment.
  • Process reward models and LLM judges: LLM judges evaluate outputs, critique trajectories, or assign rubric scores, but unstructured process scores can conflate exploration with lack of progress.They may also punish correct actions in failed trajectories or over-credit plausible narration.
  • Exploration in language agents: Language agents use searches, inspections, reads, and test execution to gather information that changes belief state rather than immediately completing tasks.Related methods also exploit multiple sampled reasoning paths, search trees, or self-generated rationales.
  • On-policy distillation and token weighting: Token-importance and on-policy distillation methods mostly operate at token or response granularity, while TRIAGE operates at the agentic segment level.Role labels can gate which action turns receive stronger reinforcement or distillation losses.

8 Conclusion

TRIAGE adds role-conditioned credit to GRPO, preserving outcome direction while correcting exploration, infrastructure, and regression errors. Across three tasks, it improves success and efficiency, with regression suppression the dominant gain.

  • TRIAGE improves success rates over GRPO across ALFWorld, Search-QA, and WebShop for two policy models.Reported gains reach 7.9 points on Qwen2.5-7B and 18.4 points on Qwen3-1.7B.
  • TRIAGE reduces completed-rollout length by 10.4%–14.8% relative to GRPO.The reduction is reported for ALFWorld and WebShop.
  • Ablations identify regression suppression inside successful trajectories as the dominant source of improvement.Exploration credit provides a consistent secondary gain.
  • Role-conditioned credit is the MSE-optimal correction expressible from role labels alone, with benefits tied to judge reliability.A manual audit measures judge reliability rather than assuming it.

A Additional Theory and Proofs

The theory frames role-conditioned credit as a projection of per-segment residual advantage onto role labels and characterizes when fixed constants reduce error. The benefit depends on informative, reliable role judgments.

  • The optimal role-measurable correction is the conditional expectation E[δ | ρ], obtained by L2 projection of residual advantage onto role labels.Uniform GRPO corresponds to the zero correction and has residual MSE E[δ^2].
  • Role-conditioned correction cannot increase residual MSE when the role labels explain nonzero residual variation.The MSE reduction equals E[(E[δ | ρ])^2].
  • Fixed role constants reduce MSE when their correction correlates positively with the residual and λ remains below the stated covariance-dependent bound.The condition is Cov(cˆρ, δ) > 0 and 0 < λ < 2 Cov(cˆρ, δ)/σ2.
  • The constants (cD, cE, cN, cR) = (1, 0.5, −0.1, −0.5) assign negative credit to regression and positive credit to exploration.This ordering targets over-crediting regressions in successful trajectories and over-punishing exploration in failed ones without per-environment tuning.
  • Because role labels are action-history–measurable, aligned corrections can reduce policy-gradient variance without changing gradient unbiasedness.Batch whitening preserves sign and relative ordering, so aligned corrections remain aligned after normalization.
  • The guarantee fails when roles explain little residual credit or judge errors make Cov(cˆρ, δ) nonpositive.The paper cites ambiguous D/E boundaries and degradation from unreliable judging or overly aggressive correction.

C Training Hyperparameters

This section specifies the training setup and compares TRIAGE with a shared-backbone value baseline. TRIAGE achieves stronger benchmark results and shorter completed trajectories, while additional training alone does not close the gap.

  • TRIAGE uses an LLM judge call per environment-facing segment, adding wall-clock cost during training.The paper reports that extended GRPO training remains below TRIAGE results on ALFWorld and WebShop.
  • The shared-backbone value baseline adds a learned scalar value head trained on the same on-policy rollouts while retaining the GRPO policy update.It isolates dense outcome-based value supervision from TRIAGE’s semantic role signal.
  • 87.5/48.1/77.2 are TRIAGE’s reported results on ALFWorld/Search-QA/WebShop, exceeding the corresponding shared-backbone value baseline results.The value baseline reaches 85.2, 46.8, and 70.8 on those benchmarks.
  • The value baseline barely improves WebShop because near-identical observations give productive and redundant clicks similar Monte-Carlo targets.The reported WebShop change is 70.1 →70.8, within run-to-run variance.
  • 21.90 segments is TRIAGE’s average completed-trajectory length on ALFWorld, versus 24.45 for GRPO.This is an additional 10.4% reduction relative to GRPO.
  • 6.82 segments is TRIAGE’s average completed-trajectory length on WebShop, versus 8.00 for GRPO.This is an additional 14.8% reduction relative to GRPO.

F Sensitivity to Role Constants and λ

Sensitivity analyses show that TRIAGE is robust around its default role constants and mixing strength, but performance degrades when penalties or mixing become too aggressive. Whitening preserves correction ordering while rescaling its magnitude.

  • The default configuration fixes (cD, cE, cN) = (1, 0.5, −0.1), |cR| = 0.5, and tunes λ per environment using training data.The sensitivity sweeps are post-hoc test-set diagnostics after λ selection.
  • Success rate remains stable across the interior λ × |cR| grid and degrades only at corners with λ = 0.4 or |cR| = 1.0.The degradation is attributed to over-punishing segments misclassified as regression.
  • 87.5/85.9/85.1 are ALFWorld success rates for |cR| = 0.5/0.25/1.0 at default λ, showing erosion when the penalty is doubled.The corresponding Search-QA values are 48.1/47.6/46.9.
  • Whitening preserves the relative ordering and sign of role corrections while changing their magnitude.A lower combined advantage for an R segment remains lower than for non-R peers in the same outcome group.
  • The role correction contributes a raw standard deviation of only 0.09–0.28 under λ ≤0.4 and the audited role distribution.It therefore perturbs rather than dominates batch whitening statistics.
  • Performance degradation appears only when λ or |cR| reaches grid corners, where the correction becomes large enough to compete with the outcome advantage.Interior stability supports the intended role signal under the operating range used.

G.1 ALFWorld Trajectories

The audited ALFWorld and WebShop trajectories illustrate how role-conditioned credit preserves useful exploration while suppressing redundant or regressive actions, especially in successful rollouts. On already efficient trajectories, TRIAGE behaves nearly like vanilla GRPO without degrading outcomes.

  • ALFWorld: On the clean six-step ALFWorld trajectory, TRIAGE and vanilla GRPO behave nearly identically, confirming that role conditioning does not hurt an already efficient rollout.The trajectory contains five decisive segments and one exploratory segment, with no redundant or regressive segments.
  • ALFWorld: 18% of segment positions carry the bulk of the gradient under TRIAGE in the 22-step lucky-recovery ALFWorld success, versus 100% under vanilla GRPO.TRIAGE lowers credit for 10 regression segments while preserving positive credit for four decisive and genuine exploratory segments.
  • ALFWorld: 34 segments receive uniform positive credit under vanilla GRPO in the pathological ALFWorld loop, while TRIAGE concentrates strong credit on five decisive and moderate credit on eight exploratory segments.TRIAGE suppresses the 20 regression segments, including a 15-step no-op inspection loop.
  • WebShop: On the clean six-step WebShop trajectory, TRIAGE slightly concentrates credit on genuine decisive actions and penalizes a duplicate purchase without changing the outcome-level behavior.The duplicate buy-now action is the only no-progress segment.
  • WebShop: TRIAGE gives failed-trajectory recovery searches bounded positive credit while strongly penalizing wrong-product clicks, avoiding the spurious lesson to never re-search after a mistake.The recovery searches are labeled E, whereas both wrong-product clicks are labeled R.
  • Cross-trajectory takeaway: Regression is common as redundant repetition in logged successes, making regression-in-success the most important diagnostic cell for TRIAGE.The paper motivates calibrating the role-conditioned mixing coefficient on a small per-environment annotated sample.

H Judge Model Audit: Side-by-Side Hand vs Qwen3-8B-Thinking Labels

The judge audit evaluates local role classification for environment-facing segments using bounded context and explicit evidence strings. It identifies a representative disagreement and a calibration blind spot: decisive progress in failed trajectories is absent from the audit set.

  • Judge context: The judge sees the task, up to five previous and five future action-observation pairs, the current action, and its immediate observation.The bounded window is intended to keep classification focused on local causal role rather than distant recovery patterns.
  • Audit setup: 18 logged trajectories across ALFWorld, WebShop, and Search-QA provide 135 independently hand-labeled segments for auditing the Qwen3-8B-thinking judge.The audit includes nine successes and nine failures, with two annotators labeling every segment independently.
  • Audit protocol: The judge must classify only the current action into one of four roles and provide a short evidence string grounded in the local action-observation context.This format is intended to make label audits easier and improve consistency.
  • Role definitions: The taxonomy distinguishes decisive task-state changes, information-gathering exploration, no-progress actions, and regression, including redundant repeats that reveal no new information.A first inspection is E, while repeated inspection or re-clicking an already selected attribute is R.
  • Disagreement: The sole Search-QA disagreement labels an off-topic third search as exploration, whereas human annotators label it regression because it shifts to a declaration-of-independence query.The example shows that local semantic relevance can remain ambiguous for the judge.
  • Audit limitation: None of the nine failure trajectories contains a hand-labeled decisive segment, so decisive progress within failed rollouts is not measurable in this calibration set.A larger follow-up audit is required to test cases where intermediate progress precedes an eventual failure.
Loading 2606.32017v2…