Source-linked AI summary
IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents
Bo Ren, Yirong Mao, Yi Yang, Wenhui Que
TL;DR
Multi-turn service agents receive task-relevant information progressively, making outcome-only rewards insufficient for assigning credit to intermediate actions. IAPO builds typed influence-dependency graphs from completed rollouts to route the same trajectory-level advantage through support and error dependencies. It improves performance across three service-agent benchmarks while preserving comparable BFCL-v4 Multi-Turn function-calling performance.
Problem
Progressive user clarifications and tool responses make it difficult for outcome-only rewards to identify which intermediate actions contributed to task resolution.
Method
IAPO extracts typed influence-dependency graphs from completed rollouts and converts support-use and failed-use structure into bounded routing weights for the original trajectory-level advantage.
Results
IAPO consistently improves over GRPO and competitive credit-assignment baselines across three interactive service-agent benchmarks, while BFCL-v4 Multi-Turn remains comparable.
Takeaways & Limitations
Completed user–tool transcripts provide sufficient realized dependency structure to refine action-level credit without replacing the final outcome reward or requiring counterfactual continuations.
Takeaways & Limitations
The annotator extracts dependency labels but does not score success or create an additional reward, so routing depends on the completed-rollout graph evidence.
Abstract
from arXiv · showhide
Large Language Model (LLM) agents increasingly solve long-horizon tasks through multi-turn interactions with users and external tools. In these settings, relevant task information often unfolds over time rather than being fully specified at the initial prompt. Service agents make this challenge especially concrete: users may clarify or revise their goals, while tool responses provide information needed for subsequent decisions. Thus, a final reward alone cannot indicate which actions contributed to resolving the task. Recent methods rely on comparative evidence from other trajectories or resampled continuations, or on separately constructed step-level learning signals, to refine credit. However, a completed rollout already records how information and errors flow between agent actions. We introduce Influence-Aware Policy Optimization (IAPO), which represents each rollout as a typed influence-dependency graph over trainable agent actions, with user and tool observations serving as evidence. IAPO converts support-use and failed-use structure into routing weights that redistribute the same trajectory-level advantage. Experiments with Qwen3-4B and Qwen3-8B demonstrate superior performance over multi-turn reinforcement learning (RL) baselines across three service-agent benchmarks: {τ^2}-Bench, UserBench, and AgentChangeBench. BFCL-v4 Multi-Turn further shows that these gains do not compromise multi-turn function-calling performance. This work advances the understanding of credit assignment in multi-turn user interactions and provides a principled approach to training service agents from sparse outcome feedback.
Introduction
Multi-turn service agents must assign credit when users and tools progressively reveal how earlier actions support or undermine later decisions. IAPO uses completed-rollout influence dependencies to route outcome feedback more precisely than uniform advantage assignment.
- Motivation: Users may clarify goals and tools may expose operational states only after earlier agent actions, making contributions identifiable through later information use.The paper calls this evolving-state credit assignment.
- Motivation: GRPO broadcasts one group-relative advantage to every trainable token, so it does not distinguish intermediate actions’ contributions to the final outcome.This motivates finer-grained credit assignment for evolving-state interactions.
- Limitations of prior approaches: Existing finer-credit methods require comparable cross-rollout states, resettable environments, continuation samples, or separately constructed step-level signals.Free-form user and tool dialogue rarely provides the structure needed by cross-rollout or reset-based approaches.
- IAPO: IAPO represents each completed rollout as a typed influence-dependency graph and routes the original trajectory advantage through observed support-use and failed-use dependencies.User and tool observations provide evidence but do not receive policy-gradient credit.
- IAPO: IAPO preserves the reward, rollout process, group normalization, and clipped-loss implementation while changing which action tokens carry feedback.Figure 2 depicts rollout scoring, graph extraction by a frozen annotator, and bounded sign-conditioned routing weights.
- Evaluation: The paper reports leading performance across three service-agent benchmarks and competitive out-of-distribution retention on BFCL-v4 Multi-Turn.The contribution list attributes these results to typed within-rollout dependencies and routed trajectory-level advantage.
Related Work
Prior work refines outcome-only credit using cross-rollout comparisons, resampled continuations, or separately learned step-level signals. IAPO instead reweights actions using influence links realized within the sampled rollout.
- Interactive benchmarks: Interactive-agent benchmarks extend task-oriented dialogue with simulated users, stateful tools, and multi-turn feedback across dialogue, web, embodied, and API-use settings.These benchmarks frame agents as sequential decision makers interacting with users and environments.
- RL and credit assignment: LLM-agent reinforcement learning builds on PPO, GRPO, and related variants, but outcome-only rewards leave intermediate steps unattributed.IAPO is presented as an alternative credit-assignment approach for these agents.
- RL and credit assignment: IAPO reweights tokens from a rollout already sampled by the baseline, using only influence links realized among the agent’s own actions.This avoids relying on additional cross-rollout structure in the described approach.
Problem Setup
The problem setup models a multi-turn service episode as alternating observations and assistant actions, with a terminal reward scored over groups of sampled trajectories. IAPO redistributes each trajectory advantage across action tokens using normalized dependency weights while leaving observations outside policy-gradient credit.
- Episode and trajectory: A service episode contains user messages and tool returns that progressively expose task state, while the policy controls assistant replies and tool calls.User messages, tool returns, and rule feedback are provenance observations rather than credited policy actions.
- Episode and trajectory: A trajectory alternates observations and assistant actions as τ = (o1, a1, o2, a2, . . . , oT, aT).The setup distinguishes observation sequence elements from trainable assistant actions.
- Episode and trajectory: Each action has token length Li, total length L is aggregated across actions, and the environment returns terminal reward Rτ.These quantities support token-level advantage allocation.
- Baseline credit: GRPO samples a group of trajectories and broadcasts the trajectory advantage to every action token.IAPO replaces this flat allocation with action-dependent routing.
- IAPO routing: IAPO assigns token-level advantage Ai,ℓ = Âτwi, where weights are positive, length-normalized, and uniformly fall back when the graph is uninformative.The annotator extracts dependencies but neither scores success nor creates an additional reward.
Method
IAPO converts completed user–tool rollouts into typed influence-dependency graphs and uses observed support and error dependencies to redistribute trajectory-level advantage across actions. Its sign-conditioned, bounded routing preserves total advantage, sign, and the existing optimization interface while replacing GRPO’s uniform token credit.
- Influence-Dependency Graph: IAPO builds a typed graph over trainable assistant actions, using user turns and tool returns as evidence but never assigning them advantage.User replies are resolved to the assistant actions that elicited them, and tool returns to the calls that produced them.
- Influence-Dependency Graph: Support-use edges capture downstream consumption of information, while failed-use edges capture repeated errors, invalid outputs, or fallback caused by earlier actions.Edges require necessity and explicit binding to a concrete argument, branch, recovery act, or other decision element.
- From Features to Bounded Weights: Graph features count downstream consumers, upstream sources, direct inputs, and observable error evidence before being converted into bounded, length-normalized weights around one.Features are standardized, clipped, and renormalized so no single hub dominates the update; every token in an action receives the same action weight.
- Sign-Conditioned Routing: Positive-advantage rollouts shift credit away from steps whose errors later actions repaired, whereas negative-advantage rollouts route blame toward steps with stronger observable error evidence.The negative branch activates only after an observable error and uses positive multipliers, so more negative advantage reaches stronger error evidence.
- Sign-Conditioned Routing: The final action weight follows the rollout sign and assigns token advantage as Ai,ℓ = Âτ wi, replacing GRPO’s flat per-token advantage.At zero trajectory advantage, the token advantage is zero; the resulting update uses the same clipped-loss implementation as GRPO.
- Safeguards: IAPO preserves total token mass, sign, and uniform fallback when active graph features are constant, while retaining GRPO’s reward, normalization, prompts, tools, sampling, and optimization interface.The method intentionally changes the token-weighted policy-gradient estimator without changing the terminal reward or surrounding training interface.
Experiments
IAPO is evaluated against matched and complementary multi-turn RL baselines across service-agent benchmarks, transfer checks, routing ablations, annotator audits, and a banking case study. It improves service-agent performance while retaining function-calling behavior, with gains linked to non-uniform support- and error-aware credit routing.
- Evaluation setup: IAPO is compared with GRPO under the same training protocol, differing only in the trajectory-to-token advantage map, alongside GiGPO and InfoPO.Evaluation covers success rate, domain breakdown, out-of-domain transfer, and annotator stability.
- Main results: At 8B, IAPO raises τ 2-Bench domain-macro pass1 from 29.61% to 42.18%, a +12.57 pp gain over matched GRPO.It also improves UserBench by +4.61 pp and AgentChangeBench by +4.41 pp.
- Per-split results: Telecom gains the most on τ 2-Bench, rising from 21.74 to 39.93 (+18.19 pp), where account or line state must persist until later clarification.Airline improves by +6.94 pp and retail by +12.57 pp.
- Generalization: IAPO improves every reported out-of-domain split, including UserBench travel splits and AgentChangeBench banking and education tasks.At 4B, macro tool-call redundancy falls from GRPO’s 46.00 to 29.74, consistent with reuse of acquired state instead of repeated queries.
- Routing analysis: The router departs substantially from flat credit: positive weights span 0.637–1.505, and 86.5% of audited steps move by more than 10%.The negative branch is sparser, with 27.1% of steps exceeding the same threshold.
- Ablations: The default c=0.25 provides the strongest joint in-domain and transfer profile, while larger clipping bounds raise τ 2 but reduce UserBench and AgentChangeBench scores.At c=0.5, the in-domain score reaches 42.29, while UserBench and AgentChangeBench fall to 17.78 and 25.67.
Conclusion
IAPO addresses credit assignment in evolving-state multi-turn service-agent training by routing trajectory-level outcome signals through within-rollout dependencies. Across three interactive service-agent benchmarks, it improves over GRPO and other credit-assignment baselines, with larger gains in interactions containing richer user and tool exchanges.
- IAPO uses a typed influence-dependency graph to assign fine-grained per-step credit while retaining the original trajectory-level outcome signal.The graph extracts support-use and failed-use dependencies from completed rollouts.
- The method avoids independently learned local rewards and counterfactual continuations by deriving credit from completed rollout dependencies.
- IAPO consistently improves over GRPO and competitive credit-assignment baselines across three interactive service-agent benchmarks.
- The largest gains occur in interactions with rich user and tool exchanges, where dense dependencies make uniform trajectory-level credit least informative.