Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimization

Jingxiao Yang; Wangjie Gan; Yingxuan Zhuang; Wenqi Zhang; Jintao Chen; Xuhong Zhang

Source-linked AI summary

Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimization

Jingxiao Yang, Wangjie Gan, Yingxuan Zhuang, Wenqi Zhang, Jintao Chen, Xuhong Zhang

arXiv:2608.31077v1Aug 31, 2026cs.AI

TL;DR

Long-horizon agentic reinforcement learning provides verified trajectory outcomes but lacks precise credit for intermediate executable decisions. TASPO aligns privileged information to target trajectories and redistributes outcome credit at the action level, yielding consistent improvements over GRPO across agentic benchmarks.

Problem
Outcome-based agentic reinforcement learning provides verified trajectory feedback but leaves ambiguity about which intermediate executable decisions deserve credit.
Method
TASPO aligns privileged information from successful sibling trajectories to target executions and redistributes verified outcome advantage using bounded, mean-preserving action-level support.
Results
TASPO consistently improves over matched GRPO baselines across agentic environments and model scales, with action-level allocation reducing training variance and smoothing optimization dynamics.
Takeaways & Limitations
The results suggest that aligned privileged supervision can provide finer learning signals without additional environment interaction.
Takeaways & Limitations
PI-induced likelihood shifts measure sensitivity to privileged information rather than reward or token-level action value.

Abstract

from arXiv · show

Outcome-based reinforcement learning provides verified feedback for language-model agents, but assigns trajectory-level advantage uniformly to all decisions, yielding coarse credit over long-horizon interactions. On-policy self-distillation offers finer supervision by re-evaluating sampled behavior with privileged information (PI) available only during training. However, fine-grained supervision is not necessarily fine-grained credit: PI-induced likelihood changes describe how additional information alters policy preference, but do not directly determine how an executable action should inherit the verified task outcome. This creates a supervision-credit gap. Privileged signals may be irrelevant to the current interaction state, operate at a token granularity misaligned with executable decisions, and lack the outcome semantics required for reinforcement. We introduce TASPO, which converts privileged supervision into outcome-grounded action credit. TASPO constructs decision-applicable PI from verified successful experience, aggregates PI-induced likelihood shifts at the executable-action level, and converts relative action support into positive, bounded, mean-preserving weights on the original trajectory advantage. Thus, the verified outcome determines the update direction and average scale, while PI only redistributes credit across actions. Across three agentic benchmarks, TASPO improves over GRPO by 10.6\% and generalizes better to unseen tasks. Further analysis indicates that TASPO reduces supervision mismatch and that action-level assignment stabilizes the policy optimization process. These findings offer the community another interesting perspective.

1 Zhejiang University · 1 Introduction

Agentic RL must combine reliable trajectory-level outcome credit with applicable, decision-level process supervision. TASPO addresses this gap by aligning privileged information to target trajectories, aggregating it over executable actions, and redistributing—not replacing—the verified outcome signal.

1 Introduction: GRPO supplies a reliable trajectory-level direction from verified outcomes but cannot distinguish different decisions within the same trajectory, whereas OPSD provides denser token-level supervision.Token-level supervision may not correspond to executable decisions or preserve outcome semantics.
1 Introduction: Existing process-supervision alternatives trade reliability against applicability: task-specific rewards need verifiable signals, while evaluators and value models face estimation error and distribution shift.Training-time privileged information reduces dependence on task-specific feedback or additional evaluation models but still requires state and relevance alignment.
1 Introduction: Process supervision must remain consistent with verified outcomes and evaluate complete environment decisions, because dense process signals can conflict with uniform outcome credit.Action-level aggregation is therefore more appropriate than token-level aggregation for trajectory outcomes and action qualities.
1 Introduction: TASPO constructs privileged information from verified successful sibling trajectories, retains only guidance supported by the target execution path, and abstains otherwise.When alignment is unavailable, it falls back to the original GRPO update.
1 Introduction: TASPO consistently improves matched GRPO baselines across ALFWorld, Search-QA, and WebShop, multiple backbone scales, and training budgets.The framework also achieves faster and more effective policy improvement under the same training budget.
1 Introduction: Controlled analyses show that trajectory alignment, action-level aggregation, and outcome-anchored credit redistribution each contribute to TASPO’s benefits.Generic skills and direct reuse of successful trajectories are less effective supervision sources than extracted, target-aligned successful experience.
1 Introduction: TASPO aggregates privileged-information likelihood shifts over complete executable actions and uses constrained weights to redistribute the original GRPO advantage.This preserves verified outcomes as the optimization direction while allocating credit more locally.

2 Related Work

Prior work frames interactive agents as beneficiaries of outcome-based reinforcement learning, while highlighting sparse-feedback challenges in assigning credit to intermediate decisions. Knowledge distillation, especially on-policy self-distillation with training-only privileged information, provides complementary token- or sequence-level supervision.

Outcome-based reinforcement learning: Interactive agents increasingly reason, search, invoke tools, and act over long horizons, motivating outcome-based reinforcement learning from environment or verifier feedback [5] [6] [7] [12] [22] [24].Sparse trajectory outcomes provide limited information about which intermediate decisions caused success or failure, reflecting delayed credit in sequential learning.
Knowledge distillation: Knowledge distillation transfers behavioral information from a teacher through token- or sequence-level supervision [11] [13].The passage distinguishes token- and sequence-level supervision as established distillation settings.
On-policy and self-distillation: On-policy distillation supervises student-generated samples to reduce distribution mismatch between optimized data and the current policy [1] [20].Recent self-distillation methods construct privileged teachers from additional contexts, reasoning traces, skills, or hindsight available only during training [18] [23] [25] [29] [30] [33] [35].

3 Method

TASPO converts privileged information from successful experience into executable-action credit while preserving the verified outcome’s update direction and average scale. It matches grounded guidance to target trajectories, measures relative action support, and redistributes trajectory advantage through positive, bounded, mean-preserving weights.

3 Method: TASPO separates roles: verified outcomes determine update direction and average scale, while privileged information redistributes credit across actions within each trajectory.The method preserves the original trajectory advantage rather than estimating a new local reward from privileged information.
Privileged-information construction: TASPO extracts conditional guidance from successful sibling trajectories, matches it to each target’s realized path, and abstains to original GRPO credit when no grounded item remains.Guidance includes requirements, ordering constraints, progress rules, or valid alternatives, with source actions and observations providing explicit evidence.
Action-level PI scoring: TASPO measures PI-induced likelihood shifts only over parsed executable-action tokens, filters changes below margin κ, and normalizes by complete action length.These shifts measure sensitivity to PI, not rewards or token-level action values; [31] motivates filtering because token roles are affected unevenly.
Outcome-oriented credit redistribution: Relative PI support is centered within each trajectory, oriented by advantage sign, and converted into positive weights bounded by [1 −ϵw, 1 +ϵw] with mean one.For positive advantage, stronger support receives more credit; for negative advantage, it attenuates the assigned negative credit.
Policy optimization: TASPO applies each action-derived weight to all valid policy tokens in its turn, retains the standard GRPO token loss, and requires no additional environment interaction.If Ai = 0 or fewer than two turns are aligned, all weights equal one.

4 Experiments

Across ALFWorld, Search-QA, and WebShop, TASPO consistently outperforms GRPO across backbone scales, with faster learning and higher final performance under the same budget. Analyses attribute these gains to decision-applicable privileged information, robustness to analyzer choice, and action-level rather than token-level credit allocation.

Consistent improvements across agent benchmarks: TASPO consistently outperforms GRPO across ALFWorld, Search-QA, and WebShop and backbone scales; on ALFWorld, gains are 12.1%, 11.1%, and 27.4% for Qwen2.5-3B, Qwen2.5-7B, and Qwen3-1.7B.The average ALFWorld gain across these models is +16.9%, with similar improvements on Search-QA and WebShop.
Faster and more effective: TASPO learns faster after initial exploration and reaches a higher final ALFWorld success rate than GRPO and SDAR under the same training budget.Figure 4 reports the training-progress comparison, while the text attributes the improvement to a more informative privileged-supervision signal.
Privileged information construction: Decision-applicable privileged information outperforms generic skills, showing that construction quality and trajectory alignment matter more than merely adding information.TASPO improves over GRPO by 12.1% and over the strongest trajectory-based baseline by 5.2%, while extracting context-conditioned guidance rather than directly imitating trajectories.
Robustness to the analyzer model: Replacing the default analyzer with GLM or Qwen3.5-35B-A3B causes only minor variation in retained PI coverage and final task success.Comparable coverage and downstream performance indicate that TASPO is robust to the training-time analyzer and that gains mainly arise from evidence-grounded, trajectory-aligned construction.
Effect of credit allocation granularity: Action-level allocation improves over token-level allocation across benchmarks, raising ALFWorld average score from 79.7 to 86.3 and WebShop success rate from 72.1 to 78.1.On ALFWorld, seed variation falls from 2.9 to 1.4; on Search-QA, average score increases from 42.9 to 43.4.
Optimization dynamics: Action-level allocation produces smoother, more consistent training with fewer KL spikes, whereas token-level allocation fluctuates more despite identical privileged information and optimization procedures.The results support filtering token-specific noise while preserving decision-relevant supervision for more stable optimization.

5 Conclusion

TASPO provides finer credit assignment for long-horizon agentic reinforcement learning by aligning privileged information with executable decisions and redistributing verified outcome credit without changing its sign. Experiments show consistent improvements over GRPO and recent baselines, with analyses supporting privileged-information alignment and action-level allocation.

TASPO introduces a trajectory-aligned framework for finer credit assignment in long-horizon agentic reinforcement learning.
TASPO constructs decision-applicable privileged information from successful experience and uses relative action support to redistribute verified outcome credit without changing its sign.
Experiments across diverse agent benchmarks show consistent improvements over GRPO and recent baselines, while analyses validate privileged-information alignment and action-level allocation.These findings indicate that reliable privileged supervision can provide a finer learning signal.