PaperlayerView exact revision on arXiv

Source-linked AI summary

MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool use

Weikang Zhao, Xili Wang, Chengdi Ma, Lingbin Kong, Zhaohua Yang, Mingxiang Tuo, Xiaowei Shi, Yitao Zhai, Xunliang Cai

arXiv:2508.18669v1cs.AI

TL;DR

Dynamic, evolving user demands make multi-turn tool use difficult, while existing RL approaches do not integrate genuinely dynamic users into training. MUA-RL adds LLM-simulated users to RL rollouts and combines cold-start training with task-completion rewards; across multiple benchmarks, it consistently outperforms base and cold-start models and can match or exceed much larger models.

  • Problem

    Existing reinforcement learning approaches for tool use do not integrate dynamic, user-driven interactions, although users may revise goals and feedback during multi-turn tasks.

    Source: §1 · p. 2

  • Method

    MUA-RL integrates LLM-simulated users into reinforcement learning rollouts, using cold-start data, real tool environments, GRPO, and rewards based on ultimate task completion.

    Source: §1 · p. 2, §3.3.3 · p. 5, §5 · p. 11

  • Results

    MUA-RL consistently outperforms base and cold-start counterparts across evaluated benchmarks and in several cases matches or exceeds much larger models.

    Source: §4.2.1 · p. 8, §4.2.2 · p. 8, §5 · p. 11

  • Takeaways & Limitations

    MUA-RL provides a scalable, efficient, and generalizable reinforcement learning paradigm for multi-turn user-interacting agentic tool use.

    Source: §5 · p. 11

Abstract

from arXiv · show

With the recent rapid advancement of Agentic Intelligence, agentic tool use in LLMs has become increasingly important. During multi-turn interactions between agents and users, the dynamic, uncertain, and stochastic nature of user demands poses significant challenges to the agent's tool invocation capabilities. Agents are no longer expected to simply call tools to deliver a result; rather, they must iteratively refine their understanding of user needs through communication while simultaneously invoking tools to resolve user queries. Existing reinforcement learning (RL) approaches for tool use lack the integration of genuinely dynamic users during the RL training process. To bridge this gap, we introduce MUA-RL (Multi-turn User-interacting Agent Reinforcement Learning for agentic tool use), a novel reinforcement learning framework that, for the first time in the field of agentic tool use, integrates LLM-simulated users into the reinforcement learning loop. MUA-RL aims to enable autonomous learning of models to communicate with users efficiently and use various tools to solve practical problems in dynamic multi-turn interactions. Evaluations are done on several multi-turn tool-using benchmarks (see Figure 1). Specifically, MUA-RL-32B achieves 67.3 on TAU2 Retail, 45.4 on TAU2 Airline, 28.3 on TAU2 Telecom, 28.4 on BFCL-V3 Multi Turn, and 82.5 on ACEBench Agent -- outperforming or matching the performance of larger open-source models such as DeepSeek-V3-0324 and Qwen3-235B-A22B in non-thinking settings.

1 Introduction

MUA-RL addresses the gap between dynamic, user-driven multi-turn tool use and reinforcement learning frameworks that largely omit such interactions. It integrates LLM-simulated users into RL rollouts and combines task-completion rewards with cold-start data and real database validation.

Related Work

  • Prior RL work incorporated external environments such as code sandboxes and Docker, but the presented motivation centers on adding dynamic user roles to tool-use training.Examples include Retool [3], SkyRL, and RAGEN [5].Source: §1 · p. 2

Motivation

  • MUA-RL targets reinforcement learning for interactions where users revise needs and expectations in response to agent behavior.The framework is motivated by the need for agents to adapt in real time to evolving goals and feedback.Source: §1 · p. 2

MUA-RL

  • MUA-RL integrates LLM-simulated users into reinforcement learning rollouts so agents refine user intent while invoking tools.Its reward is based solely on ultimate task completion rather than intermediate tool-calling formats or success rates.Source: §1 · p. 2

Contributions

  • The project combines a user-interacting RL framework with a real operational database and two cold-start data pipelines using simulated or real MCP tool responses.The authors also analyze training dynamics and behavior before and after MUA-RL.Source: §1 · p. 2

2 Related Work

Prior work develops RL optimization methods and tool-use systems, but the literature progresses from prompting and supervised training toward increasingly interactive tool invocation. These approaches provide foundations for extending RL to agentic tool use.

Reinforcement Learning

  • RL research progressed from PPO [6] and actor–critic methods [7] toward RLOO, GRPO [2], and DAPO [9] for scalable optimization.STaR [10] and MCTS are also described as iterative approaches within broader LLM fine-tuning frameworks.Source: §2 · p. 2
  • RL has been applied to downstream tasks including search [12], coding [13], and multimodal tasks [14].These developments support extending RL to new alignment and task-specialization settings.Source: §2 · p. 3

3 Methodology

MUA-RL formulates tool use as multi-turn interaction among agents, users, and databases, then combines cold-start training, synthetic or MCP-backed data, real-time user-interacting rollouts, GRPO, and task-completion rewards. The resulting framework supports both tool invocation and textual clarification across dynamic trajectories.

3.1 Task Formulation

  • Multi-turn tasks require agents to alternate between sequential database-tool calls and textual user communication to retrieve information, operate systems, and discern intent.At each turn, the agent autonomously decides whether and how many tools to invoke.Source: §3.1 · p. 3
  • The task formulation represents tools, messages, and database or user observations, with trajectories interleaving tool invocations, database outputs, user observations, and agent messages.The agent’s decisions determine how the trajectory progresses toward the user query.Source: §3.1 · p. 3
  • A lightweight supervised cold-start phase establishes basic multi-turn tool-use abilities before reinforcement learning.This addresses unfamiliar tools, text communication, iterative action, and error correction.Source: §3.2 · p. 3

3.2 Agentic Data Synthesis Pipeline

  • Cold-start trajectories are generated either with an LLM simulating tool responses or with real MCP servers executing the tools.The simulated pipeline uses agent, user, and tool LLMs, whereas MCP handles execution automatically after tools are presented.Source: §3.2 · pp. 3–4, Figure 3 · p. 4

3.3.1 Group Relative Policy Optimization

  • GRPO optimizes the policy from groups of sampled responses using groupwise outcome rewards, avoiding value-function approximation.For each query, responses are sampled from the old policy and the policy is optimized using their advantages.Source: §3.3.1 · p. 5

3.3.2 Multi-turn User-interacting Rollout

  • The framework moves beyond text-only and multi-step tool rollouts by adding an LLM-simulated user who interacts with the agent during tool-enabled rollouts.The agent must communicate to gather information and discern intent while using tools to interact with data.Source: §3.3.2 · p. 5, Figure 4 · p. 6

3.3.3 Reward

  • The reward is 1 only for complete task fulfillment and 0 otherwise, making evaluation invariant to dialogue or tool-call sequence while reducing format-based reward hacking.This design permits diverse behaviors as long as the task is resolved.Source: §3.3.3 · pp. 5–6

4 Experiments

Across multi-turn tool-use benchmarks, MUA-RL improves performance, robustness, and generalization over base and cold-start models while supporting more reliable training behavior. Analyses attribute gains to the full cold-start-plus-reinforcement-learning pipeline and structured multi-turn interactions rather than longer responses.

Benchmark Results

  • 33.4% accuracy on TAU2 Telecom makes MUA-RL-14B stronger than GPT-4o-2024-11-20 and DeepSeek-V3-0324 under dual-control dynamics.MUA-RL-32B also matches GPT-4.1 and surpasses larger models on TAU Retail and TAU Airline.Source: §4.2.1 · p. 8

Generalization

  • Cold-start models degrade on telecom, whereas MUA-RL remains competitive or leading across domains by counteracting SFT-induced domain biases.The contrast is strongest when evaluation domains differ from the cold-start training distribution.Source: §4.2.1 · p. 8

Benchmark Results

  • 54.3% TCR versus 32.9% for cold-start shows MUA-RL-14B completes more verification criteria even when full task success is not achieved.TCR provides a continuous measure of partial completion for TAU2 Telecom tasks.Source: §4.1 · p. 7, §4.2.1 · p. 8
  • 82.5 on ACEBench Agent and 28.4% on BFCL-V3 Multi Turn show MUA-RL-32B outperforming its base and cold-start models across diverse tasks.MUA-RL-32B approaches DeepSeek-V3-0324 on BFCL-V3 and trails only GPT-4.1 on ACEBench Agent among tested models.Source: §4.2.2 · p. 8

Training Dynamics

  • Training increases all-correct rollouts and suppresses all-wrong rollouts, while rollout turns stabilize near 21–23 and response length remains largely unchanged.These dynamics indicate improved structured interaction rather than gains from generating longer responses.Source: §4.3.1 · pp. 9–10

Ablation Study

  • Removing either cold-start or reinforcement learning reduces performance, while the complete pipeline consistently performs best across TAU2-Bench and BFCL-V3 Multi Turn.The ablation compares full MUA-RL-32B with variants without RL or without cold-start.Source: §4.3.2 · p. 10

5 Conclusion

MUA-RL combines cold-start training with GRPO on real-time interactive rollouts to address dynamic, user-driven tool use. Across multiple benchmarks, it improves over base and cold-start models and can match or exceed much larger models.

  • MUA-RL consistently outperforms its base and cold-start counterparts across TAU1-Bench, TAU2-Bench, BFCL-V3 Multi Turn, and ACEBench Agent, sometimes matching or exceeding larger models.The framework also shows strong generalization in challenging telecom dual-control tasks.Source: §5 · p. 11
  • MUA-RL combines a lightweight cold-start phase with GRPO on real-time interactive rollouts for multi-turn user-interacting agentic tool use.Its simplified, task-oriented reward is intended to support robust behavioral patterns without overfitting to specific trajectories.Source: §5 · p. 11
  • The framework is presented as a scalable, efficient, and generalizable paradigm for building more reliable interactive LLM-based agents.Future work may extend it to multimodal environments, finer-grained rewards, and stronger human-preference alignment.Source: §5 · p. 11

Appendix A Cold-start Data

The cold-start data appendix illustrates policy-grounded tool-use examples across university registration and AniList tasks. These examples emphasize authentication, constrained actions, confirmation requirements, and tool-mediated responses.

University course registration

  • University course-registration data encodes policies for enrollment, dropping, swapping, auditing, and updating student information.The policy specifies eligibility, deadlines, capacity, prerequisites, conflicts, academic standing, and confirmation requirements.Source: Appendix A · pp. 15–16
  • The university examples require authentication before action and explicit confirmation before changing enrollment or personal information.The contact-update trajectory authenticates the student, clarifies the requested field, confirms the new phone number, and then invokes the update tool.Source: Appendix A · pp. 15–17

AniList tool use

  • AniList examples constrain agents to one tool call at a time and prohibit unsupported information, speculation, or subjective recommendations.Ambiguous search criteria must first be clarified using available genres or media tags rather than invented filter values.Source: Appendix A · p. 17
  • The AniList example reports the first 5 results out of 5,000 total characters, demonstrating paginated retrieval with an option to continue or inspect a specific result.Source: Appendix A · p. 18

A case study

The case study contrasts baseline and post-training retail-agent behavior during an order-modification task. After MUA-RL, the model follows policy more reliably by authenticating, gathering options, explaining price differences, and confirming before acting.

Behavioral comparison

  • After MUA-RL, the agent authenticates the user, lists product variants, explains price differences, and obtains confirmation before modifying the order.The revised interaction is described as cautious, policy-grounded, and user-centered rather than rigidly policy-following.Source: A case study · p. 18
  • The baseline Qwen3-32B Non-thinking model prematurely modifies the order without explicit confirmation, violating policy and causing failure when the user requests another change.Source: A case study · p. 18

Retail policy constraints

  • The retail workflow requires user authentication before providing order information or taking actions affecting the account.Authentication is performed by locating the user ID through email or through name and ZIP code, even when a user ID is supplied.Source: A case study · p. 19
  • Pending-order modification is limited to shipping address, payment method, or product-item options, and the action can make further modification or cancellation unavailable.The policy requires collecting all intended item changes and confirming the complete request beforehand.Source: A case study · p. 20

Appendix C Details of the Evaluation Benchmarks

The evaluation benchmarks cover realistic multi-turn tool use across retail, airline, telecom, function calling, and agent collaboration settings. They vary in task structure, domain constraints, and whether users can also invoke tools.

TAU1-Bench

  • TAU1-Bench evaluates policy-compliant multi-turn tool use in retail and airline domains with realistic databases, tools, and confirmation constraints.Its curated set contains 115 retail tasks and 50 airline tasks, each designed around a single correct outcome.Source: Appendix C · pp. 24–25

TAU2-Bench

  • TAU2-Bench extends TAU1-Bench toward dual-control scenarios in which both the user and agent can independently call tools.TAU2 Telecom models situations such as technical support and collaborative troubleshooting.Source: Appendix C · p. 25

BFCL-V3 Multi Turn

  • BFCL-V3 Multi Turn provides 800 tasks across four categories, including standard interactions, missing parameters, missing functions, and long-context tool use.Source: Appendix C · p. 25

ACEBench Agent

  • ACEBench Agent evaluates multi-turn, multi-step tool use across flight booking, food delivery, finance, and communications scenarios with 50 tasks and 22 tools.Source: Appendix C · p. 25

Related papers

Loading 2508.18669v1…