PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Better Agents for Multi-Turn User Interaction: The Next User Turn Is More Than Context

Yiwen Zhao, Zhihao Wen, Yuchen Mao, Mingxuan Jiang, Yihao Hu, Pan Wang, Xin Zhang, Wei Wu

arXiv:2608.17499v1cs.AI

TL;DR

Multi-turn tool agents must use evolving user interactions to assign credit beyond terminal outcomes. FACA aligns next-turn reactions with preceding segments and improves the nine-domain τ-family average over outcome-only Interactive GRPO by 5.91 and 10.22 percentage points at 8B and 14B.

  • Problem

    Outcome-only interactive reinforcement learning leaves noisy, temporally local evidence in next-user reactions unused for credit assignment.

  • Method

    FACA aligns each next-user reaction with its preceding segment, locally normalizes reaction evidence, and adds process advantage to verified terminal outcome advantage.

  • Results

    5.91 and 10.22 percentage points: FACA improves the nine-domain τ-family average over matched outcome-only Interactive GRPO at 8B and 14B, respectively.

  • Takeaways & Limitations

    Next-turn reactions provide actionable local credit for multi-turn user-interacting agents when they carry informative, aligned structure.

  • Takeaways & Limitations

    The approach relies on private heuristic strategy metadata from a frozen simulator, while causal attribution from observable utterances or real users remains open.

Abstract

from arXiv · show

User-facing tool agents must coordinate dialogue and tool use as user goals unfold over multiple turns. Yet interactive reinforcement learning typically reduces each rollout to a terminal reward, assigning the same credit to effective elicitation, errors, and later repair. The next user turn is more than context: it also provides noisy, temporally local evidence about the preceding user-to-user segment. We introduce \textbf{F}eedback-\textbf{A}ware \textbf{C}redit \textbf{A}ssignment (\textsc{FACA}), which aligns each reaction with that segment, derives a locally normalized reaction advantage, and adds it to verified terminal outcome advantage without an extra critic or rollout. Against an outcome-only Interactive GRPO control matched in simulator, visible dialogue, initialization, rollout, and optimization, \textsc{FACA} improves the nine-domain $τ$-family average across three independently trained runs by 5.91 and 10.22 percentage points at 8B and 14B, respectively. Gains concentrate in Telecom; at 8B, randomizing reaction polarity removes the Telecom gain. The same ordering holds zero-shot on Pare-Bench and Co-Gym. These results demonstrate that next-turn user reactions provide actionable local credit for improving multi-turn user-interacting agents.

1 Introduction

Multi-turn tool agents must coordinate dialogue, tools, and user decisions across user-to-user segments as goals evolve. FACA uses the next user reaction as local credit while retaining terminal outcomes, improving benchmark performance and transferring zero-shot.

  • Problem: Outcome-only reinforcement learning supports multiple valid trajectories but collapses effective elicitation, errors, and later repair into terminal credit.Interactive τ-family benchmarks and simulated-user frameworks expose and train these multi-turn demands [Yao et al., 2025, Barres et al., 2025, Shi et al., 2026, Zhao et al., 2025b, Qian et al., 2025b].
  • Motivation: A U2U segment contains agent messages, tool interactions, and tool results between adjacent user turns, whose reactions may inform, approve, reject, or correct preceding actions.These evolving interactions require coordinating dialogue, tool use, and user decisions rather than executing fully specified tasks.
  • Method: FACA aligns each next-user reaction with its preceding U2U segment, locally normalizes reactions, and adds the resulting process advantage to terminal outcome advantage.It changes only credit assignment, without a learned critic, additional rollouts, or agent-visible reaction labels.
  • Findings: 5.91 and 10.22 percentage points improve the nine-domain τ-family average at 8B and 14B, respectively, across three independent runs.FACA also outperforms matched controls zero-shot on Pare-Bench and Co-Gym.
  • Findings: FACA’s gains are heterogeneous and reaction-sensitive, concentrating in Telecom, where randomizing reaction polarity removes the 8B gain.This supports benefits when user reactions contain informative local structure rather than universally reliable supervision.

2 Related Work

Prior work addresses turn-level credit assignment because terminal rewards cannot identify which interaction turns elicited information, caused errors, or enabled recovery. Existing approaches use learned critics, intermediate evaluators, or trajectory decomposition with monitoring-derived rewards.

  • Credit assignment in multi-turn agents: Terminal task rewards verify overall success but cannot identify which turns elicited useful information, introduced errors, or enabled recovery.
  • Credit assignment in multi-turn agents: Prior work derives turn-level credit from learned critics or intermediate evaluators [Zhou et al., 2024, 2025, Choudhury, 2025, Wei et al., 2025].
  • Credit assignment in multi-turn agents: Agent Lightning decomposes trajectories into transitions and converts monitoring signals into intermediate rewards [Luo et al., 2025].

3 Feedback-Aware Credit Assignment

FACA augments verified terminal outcome credit with locally normalized, reaction-grounded credit assigned to each user-to-user segment. It uses the next visible user reaction and private behavioral strategy without adding a critic, evaluator, or rollout.

  • The reaction branch supplements the verified terminal outcome advantage, whereas disabling it reduces FACA to the matched outcome-only Interactive GRPO control.Reaction labels can still differ across rollouts with identical terminal outcomes, enabling local distinctions when terminal rewards are homogeneous.
  • FACA aligns each private user strategy with the immediately preceding agent segment, while only the visible utterance is appended to the agent context.Temporal adjacency is an inductive bias, not a claim that every reaction is caused solely by that segment.
  • Progress-consistent reactions receive positive polarity, friction-consistent reactions negative polarity, and vague or invalid outputs neutral credit.The labels capture interaction movement rather than sentiment or verified task correctness.
  • FACA locally normalizes reaction evidence among rollouts reaching the same user-to-user index and assigns each trainable token in that segment the resulting reaction advantage.It uses only the immediate next reaction; singleton or constant anchors receive zero, and later reactions are not propagated backward.
  • FACA trains generated language and tool-call tokens while masking user and raw tool-result tokens, requiring no learned critic, separate evaluator, or additional rollout.The reaction metadata is produced within the existing user call.

4 Experiments

Experiments evaluate FACA against outcome-only Interactive GRPO on nine multi-turn τ-bench domains, using tightly matched training conditions and zero-shot transfer tests on Pare-Bench and Co-Gym.

  • τ-bench family: The evaluation spans nine τ-bench-family domains covering Airline, Retail, Telecom, and Bank tasks that require multi-turn tool use and communication.The suite combines τ-bench, τ2-bench, and τ3-bench domains; Telecom includes device-side user actions, while τ3 Bank requires unstructured-information grounding.
  • Transfer evaluation: Zero-shot evaluation tests transfer to Pare-Bench’s proactive mobile-agent scenarios and Co-Gym’s asynchronous, bidirectional collaboration tasks.Pare-Bench includes 143 scenarios, while Co-Gym contains Travel Planning, Related Work Writing, and Tabular Analysis tasks under dual control.
  • Training setup: FACA and Interactive GRPO are compared as alternative RL continuations from the same SFT checkpoint, with three independent seeds per method and scale, using step-120 checkpoints.Experiments use Qwen3-8B and Qwen3-14B initialized by SFT on the public MUA-RL release.
  • Controlled comparison: The comparison isolates credit assignment by matching simulator, prompts, visible utterances, initialization, data, rollouts, optimizer, and horizon across methods.Interactive GRPO uses only terminal outcome advantage, whereas FACA additionally constructs U2U-level process advantage from private simulator reaction metadata unavailable to the agent.
  • Evaluation protocol: τ-bench performance is measured with strict pass@1 from verified final environment states, averaging the nine domains equally across three independently trained checkpoints.Each checkpoint is evaluated once under a shared protocol; Base and SFT serve as fixed pre-RL references, and run-level dispersion is reported separately.

5 Results

FACA outperforms matched outcome-only Interactive GRPO at both model scales and preserves the ordering across two unseen interaction protocols. Controlled ablations show that gains require signed, reaction-content-preserving credit aligned to the preceding agent span.

  • Overall results: At 8B, FACA raises the three-run average from 34.66±0.25 to 40.57±1.04, a 5.91-point gain over Interactive GRPO; the same ordering holds at 14B.The arms share SFT initialization, user-generation path, observations, and optimization setup, isolating reaction-grounded versus outcome-only credit.
  • Zero-shot transfer: On Pare-Bench, strict Pass@1 increases from 6.29% to 10.49% at 8B and from 10.49% to 13.29% at 14B, with Pass@4 showing the same ordering.The transfer evaluation uses the same 143-scenario full split; Figure 3 reports the matched Pass@1 and Pass@4 comparison.
  • Zero-shot transfer: FACA leads 13 of 16 Co-Gym scale–metric cells, although Travel DR decreases slightly at both scales.The shared ordering across Pare-Bench and Co-Gym is comparative evidence, while low Pare-Bench rates and Travel regressions limit broader conclusions.
  • Ablations: In the 8B ablation, aligned FACA with λ=0.5 reaches 39.37 overall and 47.37 on Telecom, while shifted or randomized reactions erase the gain over outcome-only training.The ablation is single-run and uses a one-shot evaluation protocol.
  • Ablations: Relative to outcome-only, aligned λ=0.5 adds 4.67 points to Avg. and 17.11 points to Tel.2, outperforming λ=0.1 and reversing-sign controls.Shifted and randomized credit fall below outcome-only, indicating that token-level variation alone is insufficient without reaction content and temporal association.

6 Analysis

Analysis supports a conditional account of FACA: reaction credit helps when subsequent user behavior exposes consequences of the preceding segment, while transfer is not uniformly beneficial. Telemetry and trajectory evidence show that reaction credit adds differential signal without replacing verified terminal outcomes.

  • Reaction observability: Telecom is especially feedback-rich because users repeatedly report consequences of identity grounding, device-state inspection, operations, validation, diagnosis, and verification.Progress-consistent reactions can reinforce effective elicitation and repair, while friction-consistent reactions can localize unclear instructions or failed proposals.
  • Out-of-distribution analysis: FACA retains its ordering over the outcome-only control on Pare-Bench and Co-Gym, but Co-Gym Travel regresses, showing that reaction credit is not universally beneficial.The advantage transfers when subsequent user behavior remains informative about the preceding segment.
  • Credit decomposition: 36.00% and 33.79% of the pre-optimization L1 advantage-magnitude proxy came from the process branch at 8B and 14B, which exceeded outcome magnitude in only 6/120 and 1/120 steps.The branch remained active without dominating terminal credit, supporting auxiliary reaction credit alongside verified outcomes rather than replacing task completion.
  • Training telemetry: 69.95% to 82.51% positive reactions coincided with nonzero process-advantage coverage falling from 86.16% to 82.50% across the matched 8B replication.Telemetry and tensor counts exactly agreed across all 154,219 U2U segments; Figure 4 reports the segment-weighted phase averages.
  • Final checkpoint analysis: 96/114 versus 30/114 tasks on τ2 Telecom and 95/114 versus 58/114 on τ3 Telecom favor FACA over Interactive GRPO in the final 14B evaluation.FACA led on Mobile and MMS but trailed on τ3 Service, so the paired advantage was not uniform.
  • Outcome-credit diagnostics: 39.27% of 8B and 31.04% of 14B group-steps had zero group-normalized outcome advantage, identifying cases where reaction credit can supply differential signal.Figure 5 shows all-correct groups becoming more frequent while all-wrong groups decline, but prevalence alone does not establish causality.

7 Conclusion

FACA treats the next user turn as noisy evidence about the preceding user-to-user segment, converting it into locally normalized process credit alongside verified terminal credit. Under a strict outcome-only Interactive GRPO control, it improves the nine-domain τ-family average at both model scales and retains the ordering on two zero-shot interaction protocols.

  • Conclusion: FACA converts the next user turn’s reaction into locally normalized process credit and combines it with verified terminal credit.The method uses the next user turn as noisy evidence about the preceding user-to-user segment.
  • Conclusion: FACA improves the nine-domain τ-family average at both 8B and 14B under a strict outcome-only Interactive GRPO control.
  • Conclusion: The improvement ordering is retained on two zero-shot interaction protocols without changing the frozen simulator, exposing labels, training a critic, or adding rollouts.

Limitations · A Zero-Shot Transfer Protocols · A.1 Pare-Bench

The paper reports aggregate zero-shot transfer gains over outcome-only training across scales and both transfer benchmarks, while limiting causal claims because evaluation relies on simulator metadata. Pare-Bench uses held-out scenarios and fixed evaluation settings to assess transfer without training or tuning on its scenarios.

  • Limitations: Evaluation relies on a frozen DeepSeek-V4-Flash simulator that exposes private strategy metadata alongside each utterance.The metadata is treated as heuristic evidence for the preceding user-to-user interaction.
  • Limitations: Event-level causal attribution and recovery from observable utterances or real users remain open limitations.Validation across simulator families, architectures, real users, and diverse deployment conditions is left for future work.
  • Limitations: Aggregate gains over outcome-only training hold across model scales and both zero-shot transfer benchmarks, with the strongest gains in feedback-rich Telecom tasks.Evidence covers Qwen3 8B/14B agents, nine τ-family domains, and zero-shot Pare-Bench/Co-Gym evaluation.
  • A.1 Pare-Bench: Pare-Bench uses 143 full-split scenarios, with no Pare-Bench scenario used for training, checkpoint selection, or prompt tuning.The four plotted checkpoints were trained only on Airline and Retail training splits of τ-bench.
  • A.1 Pare-Bench: DeepSeek-V4-Flash serves as the active reasoning-enabled user simulator, while evaluated agents use temperature 0, disabled thinking, and a 2,048-token response limit.The simulator uses high reasoning effort; the same checkpoint powers the Observe and Execute roles.
  • A Zero-Shot Transfer Protocols: The evaluation protocol uses the same checkpoint for Observe and Execute roles, supporting a controlled comparison across the four plotted reinforcement-learning checkpoints.All checkpoints use the same 143-scenario full split.

A.2 Collaborative Gym … D.2 Evaluation and Run Aggregation

The study evaluates matched Interactive-GRPO and Faca agents across collaborative environments, using frozen-user protocols, reaction-based credit extraction, and three-run τ-bench aggregation. Co-Gym results show Faca improves overall DR and CS over Interactive-GRPO at both model scales.

  • A.2 Collaborative Gym: Co-Gym evaluates each checkpoint on 312 unseen collaborative tasks spanning Travel Planning, Related Work Writing, and Tabular Analysis under non-turn-taking notifications.The tasks require agents and simulated users to communicate while acting in shared editors, search interfaces, or notebooks.
  • A.2 Collaborative Gym: Overall DR improves by 5.5 points at 8B and 2.2 points at 14B, while Overall CS improves by 3.3 and 3.7 points, respectively, for Faca over Interactive-GRPO.Table 3 reports matched pairs on a 0–100 scale; the comparison is restricted to the matched RL pair.
  • B Reaction Mapping and Extraction: Missing, malformed, or ambiguous strategy metadata maps to neutral and supplies no signed process credit, while terminal verification remains responsible for world-state correctness.The mapping is an inductive bias because reaction categories can reward unnecessary questioning, follow correct refusals, be exogenous, or overlap with terminal success.
  • C Shared Frozen-User Prompt: The frozen simulator emits private reaction strategy metadata and a visible utterance together, but only the utterance enters the agent context.Interactive GRPO discards the remaining fields, whereas Faca reads strategy only after rollout construction.
  • C Shared Frozen-User Prompt: The simulator prompt requires one short natural reply, faithful facts, JSON output, and explicit strategies including reveal_piece, clarification, challenge_solution, confirmation, and close.The utterance is the only field visible to the agent simulator, while change_mind and be_vague are rare behaviors.
  • D.1 SFT and RL Configuration: Qwen3-8B and Qwen3-14B are cold-started with SFT on 1,580 annotated multi-turn tool-use trajectories before agent-only reinforcement learning.SFT supervises assistant responses and tool calls while masking user and tool-result tokens.
  • D.1 SFT and RL Configuration: Interactive GRPO and Faca continue from the same SFT checkpoint with three seeds, shared training data and optimization, and the frozen DeepSeek-V4-Flash user path.The matched arms share prompts, visible utterances, rollout construction, optimizer, and horizon; Faca additionally uses process advantage A_p while Interactive GRPO uses only outcome advantage A_o.
  • D.2 Evaluation and Run Aggregation: τ-bench pass@1 uses verified final states and fixed step-120 checkpoints, with domain means over three runs and Avg. as the unweighted mean across nine domains.Pare-Bench and Co-Gym instead use one designated step-120 checkpoint per scale and method and are reported separately from the three-run τ aggregate.

D.3 Seed-Level Results and Ablations

Seed-level evaluation uses three independently trained checkpoints per condition, each evaluated once under the same strict protocol. Matching training seeds yields nine-domain improvements of 4.67, 6.33, and 6.75 points at 8B and 8.62, 10.14, and 11.91 points at 14B.

  • Seed-Level Results: Each s1–s3 checkpoint is an independently trained RL run evaluated once under the same strict protocol, so run-level standard deviations reflect complete-evaluation dispersion and residual evaluation stochasticity.They are not repeated evaluations of a single checkpoint, and the standard deviations do not separately identify those components.
  • Seed-Level Results: 8.62, 10.14, and 11.91 points are the matched-seed nine-domain improvements at 14B, while the corresponding 8B improvements are 4.67, 6.33, and 6.75 points.The comparison matches methods by training-seed identifier across independently trained runs.
  • Ablations: The Aligned (λ= 0.5) and Outcome-only rows in Table 4 correspond exactly to the 8B s1 checkpoints, while other ablations remain within the same controlled reference experiment family.This design keeps ablations internally controlled while the main table estimates performance across independent training runs.

E Strategy and Credit Telemetry · F Telecom Audit and Case Study · F.1 Matched Multi-Fault Telecom Case

The Telecom audit shows that Faca’s gains arise from paired task-level improvements and distinct interaction failure modes, while telemetry validates segment-level credit records but limits some distributional interpretations. A matched case illustrates Faca completing the task where Interactive GRPO exhausts its error budget.

  • E Strategy and Credit Telemetry: The step-120 Faca logs contain 15,360 trajectories per scale, with telemetry summaries verifying 16 prompt groups per step and reproducing Table 6 aggregates.Zero consistency, span, and debug counts are mechanical implementation checks, not evidence that reaction labels are semantically correct.
  • E Strategy and Credit Telemetry: A matched 8B replication records 154,219 U2U segments across 120 telemetry files, with file counts matching tensor-level counts and each record carrying strategy, polarity, anchor, advantage, and credit fields.The telemetry contains zero embedded-step mismatches or write errors; Table 7 reports complete U2U strategy distributions, including confirm +10.75 points and be_vague −9.06 points.
  • E Strategy and Credit Telemetry: The telemetry cannot reconstruct phasewise reaction distributions for evaluated 8B and 14B runs or valid nonconstant-anchor rates from their incomplete logs.Singleton counts do not identify total anchors or constant-reaction non-singleton anchors.
  • E Strategy and Credit Telemetry: Process-credit quantities are pre-clipping advantage-magnitude proxies rather than optimizer-update or gradient mass, including within outcome-homogeneous groups.This qualification applies to 0.5E|A_p| and E|A_o| and prevents interpreting them as branch-level updates.
  • F Telecom Audit and Case Study: On matched 114-task Telecom evaluations, Faca wins 73 versus 7 Interactive-GRPO-only tasks on τ2 and 48 versus 11 on τ3, with exact paired tests significant.Both methods succeed on 23 τ2 tasks and 47 τ3 tasks; neither succeeds on 11 and 8, respectively.
  • F Telecom Audit and Case Study: Faca-only wins usually reflect Interactive GRPO exhausting the environment error budget, whereas every Interactive-GRPO-only win reflects an unsuccessful Faca user stop.Interactive GRPO exhausts the budget on 68/73 τ2 and 28/48 τ3 Faca-only wins; Table 8 retains the negative τ3 Service slice.
  • F.1 Matched Multi-Fault Telecom Case: In a matched τ2 Telecom case, Faca completes four device changes and both terminal checks at 275 Mbps, while Interactive GRPO’s fifth rejected call exhausts the error budget.Both trajectories reached the four required state changes, but Interactive GRPO ended before final speed-test verification.

F.1.1 Successful FACA trajectory · F.1.2 Failed Interactive-GRPO trajectory

The successful FACA trajectory resolves the Telecom issue despite execution-authority errors, while Interactive GRPO fails after repeated invalid agent-side tool calls exhaust the error budget. Both trajectories elicit the needed user-side state changes, but only FACA reaches terminal validation with a 275 Mbps result.

  • F.1.1 Successful FACA trajectory: FACA succeeds with reward 1, ultimately producing a 275 Mbps speed-test result that satisfies both terminal assertions.The agent recovers from invalid tool attempts by transferring device actions and the speed test to the user.
  • F.1.1 Successful FACA trajectory: FACA correctly identifies the Premium Plan, 8.7 GB usage under a 15 GB limit, and the need to troubleshoot connectivity rather than data exhaustion.The trajectory establishes the account and line details before diagnosing network mode and device-state problems.
  • F.1.1 Successful FACA trajectory: The successful trajectory also disables Data Saver before the agent’s unavailable speed-test tool triggers another recovery to user-executed testing.The user then reports an excellent 275 Mbps connection and that mobile data works perfectly.
  • F.1.2 Failed Interactive-GRPO trajectory: Interactive GRPO fails with reward 0 after its final agent-side speed-test attempt exhausts the error budget before terminal validation.The trajectory includes five rejected agent-side calls to user-only tools despite eliciting the four required state-changing user calls.
  • F.1.2 Failed Interactive-GRPO trajectory: The failed Interactive-GRPO dialogue repeatedly receives user requests for simpler, step-by-step guidance as technical detail and settings navigation cause confusion.The user explicitly asks what to do first, requests one-step-at-a-time instructions, and seeks help after finding 2G-only mode and an airplane icon.
  • F.1.2 Failed Interactive-GRPO trajectory: The comparison rules out a simple extra-interaction explanation: FACA succeeds with six fewer stored events, three fewer assistant messages, and two fewer tool calls.Both policies make execution-authority mistakes, but only FACA recovers sufficiently for terminal validation.

G Algorithm and Reproducibility

FACA assigns local reaction credit to user-to-user segments and broadcasts it with terminal outcome advantage during training. Main runs use matched configurations, fixed λ=0.5, and a common step-120 checkpoint for evaluation.

  • Algorithm: FACA groups assistant spans between adjacent user messages into user-to-user segments, attaches the following reaction, and combines normalized outcome and reaction advantages.The combined advantage is broadcast to assistant-language and generated tool-call tokens, while user and raw tool-result tokens are masked.
  • Compute and configuration: Main runs use one eight-GPU NVIDIA B200 node with VERL, SGLang rollouts, and official benchmark implementations.Matched methods share optimizer, batch and rollout settings, horizon, simulator endpoint, and container.
  • Compute and configuration: λ=0.5 is fixed for main FACA runs, with λ∈{−0.5, 0, 0.1, 0.5} compared in Table 4.All methods are evaluated from the fixed step-120 checkpoint without benchmark-specific peak selection.
Loading 2608.17499v1…