Source-linked AI summary
UserRL: Training Interactive User-Centric Agent via Reinforcement Learning
Cheng Qian, Zuxin Liu, Akshara Prabhakar, Jielin Qiu, Zhiwei Liu, Haolin Chen, Shirley Kokane, Heng Ji, Weiran Yao, Shelby Heinecke, Silvio Savarese, Caiming Xiong, Huan Wang
TL;DR
UserRL addresses the limited support for training agents that can assist users across diverse and dynamic multi-turn interactions. It introduces standardized gym environments with simulated users and varies trajectory-level scoring and turn-level reward assignment under GRPO. The results identify SFT cold start and trajectory-level scoring as important for sustained and effective interaction learning, while open-source simulators offer a transferable, cost-effective training option.
Problem
Training user-centric agents is difficult because user interactions are diverse and dynamic, while existing benchmarks and pipelines lack standardized, realistic representations of these interactions.
Method
UserRL combines eight standardized gym environments, adaptive LLM-based user simulation, and configurable trajectory- and turn-level reward shaping for GRPO training and evaluation.
Results
UserRL experiments show that SFT cold starts sustain RL improvements, trajectory-level scoring outperforms fine-grained turn differentiation, and Qwen3-32B simulators transfer effectively to GPT-4o evaluation.
Takeaways & Limitations
Careful reward design and simulator selection are central to developing practical user-centric agents, alongside model scale.
Takeaways & Limitations
Turn-level reward heuristics can misrepresent contribution, and interaction effectiveness must be balanced against conversational efficiency.
Abstract
from arXiv · showhide
Reinforcement learning (RL) has shown promise in training agentic models that move beyond static benchmarks to engage in dynamic, multi-turn interactions. Yet, the ultimate value of such agents lies in their ability to assist users, a setting where diversity and dynamics of user interaction pose challenges. In this work, we propose UserRL, a unified framework for training and evaluating user-centric abilities through standardized gym environments paired with simulated users. We systematically vary turn-level reward assignment and trajectory-level score calculation to analyze how different formulations affect learning under the GRPO algorithm. Our experiments across Qwen3 models reveal three key findings: (i) SFT cold start is critical for unlocking initial interaction ability and enabling sustained RL improvements; (ii) deliberate trajectory scoring yields more efficient and effective multi-turn interactions; and (iii) while stronger simulated users (e.g., GPT-4o) facilitates training, open-source simulators (e.g., Qwen3-32B) remain a cost-effective and transferable option. Together, these results highlight that careful design of reward shaping and user simulation choice is as crucial as model scale, and establish UserRL as a practical pathway for developing robust user-centric agentic models. All codes and data are public for future research.
1 INTRODUCTION
UserRL addresses the challenge of training agents for diverse, dynamic, multi-turn user interactions through standardized gyms, simulated users, and reward-shaping analysis. The study finds that SFT initialization, trajectory-level scoring, and simulator choice materially affect interaction learning and performance.
- Motivation: UserRL asks how agentic models can acquire user-centric abilities despite diverse user behavior and evolving multi-turn interactions.The paper identifies the absence of standardized representations and realistic dynamic simulations as central training challenges.
- Approach: The framework combines standardized user-centric gym environments with multi-turn RL rollouts and adaptive LLM-based user simulation.The gyms support distinct interaction skills, while simulated users provide context-aware feedback during training.
- Findings: Across Qwen3 models, trajectory-level scoring is more decisive than fine-grained turn differentiation, while SFT cold start prevents early plateaus and enables over 100% gains on certain gyms.The study evaluates reward supervision under GRPO across 4B and 8B models.
- Findings: Qwen3-32B simulated-user training transfers effectively to GPT-4o evaluation, whereas stronger simulators accelerate learning and produce higher performance.The results position open-source simulation as a cost-effective alternative with transferability.
- Contribution: UserRL introduces a unified, extensible platform for benchmarking and training agents as adaptive partners rather than isolated task solvers.Its standardized tool interface supports customization and scalable RL pipelines.
2 RELATED WORK
Related work establishes multi-turn RL and user-centric evaluation as complementary directions, but UserRL unifies them in standardized environments for training and benchmarking interactive abilities.
- User-centric agent design and evaluation: Prior user-centric research evaluates underspecified, multi-intent, and evolving user needs through real-interaction benchmarks, multi-turn testbeds, and user modeling.These efforts examine feedback incorporation, tool use, adaptation, and personalization.
- UserRL: UserRL builds on these directions by jointly benchmarking and training user-centric abilities within standardized gym environments.The framework connects user-centered evaluation with reinforcement-learning training.
- Agentic RL training and adaptation: Recent reinforcement-learning work has improved agentic reasoning, tool use, and multi-stage interaction across web, code, games, and browsing tasks.The cited systems use multi-turn rollouts to coordinate tools and execute extended pipelines.
3 GYM CONSTRUCTION
UserRL constructs eight standardized gym environments that combine rule-based task evaluation with dynamic LLM-simulated users. A minimal Action, Search, and Answer interface makes diverse interaction skills trainable and extensible.
- Gym Construction: The framework provides eight gym environments unified under a reproducible interface, each emphasizing different user interaction capabilities.The environments span intent understanding, persuasion, planning, tool use, and creative, strategic, and mathematical reasoning.
- General Gym Components: Each environment combines deterministic task transitions and rewards with context-adaptive responses from an LLM-based simulated user.This design preserves formal evaluation while adding dynamic conversational, preference, or answer feedback.
- Standardized Tool Interface: The standardized interface reduces agent-environment interaction to Action, Search, and Answer operations.Action communicates with the simulated user, Search retrieves external knowledge, and Answer submits a candidate solution for verification.
- Standardized Tool Interface: Available operations vary by environment, allowing the interface to represent task-specific interaction constraints such as persuasion without verifiable answers.TurtleGym supports all three operations, whereas PersuadeGym restricts agents to Action.
- Specific Gym Designs: The gyms support both training and benchmarking, with some environments newly curated and others adapted from existing benchmarks.This combination demonstrates the framework’s flexibility and extensibility.
4 USERRL EXPLORATION
UserRL extends GRPO for multi-turn interaction by separating turn-level reward shaping from trajectory-level scoring. This modular design supports systematic comparison of reward-allocation and scoring choices while preserving a consistent optimization pipeline.
- 4.1 MULTI-TURN ROLLOUTS AND NOTATION: Each rollout trajectory records states, actions, turn rewards, and token sequences across T interaction turns.Turn-level rewards are transformed before being broadcast so all tokens within a turn share its signal.
- 4.2 TURN-LEVEL REWARD SHAPING: The framework evaluates Naive, Equalized, Reward-to-Go, and Exponential Mapping schemes, representing different biases for assigning credit across turns.Equalization treats turns uniformly, Reward-to-Go propagates discounted future credit, and Exponential Mapping preserves small positive progress while distinguishing larger rewards.
- 4.3 TRAJECTORY-LEVEL SCORING: Trajectory scores are computed with either Sum, which reflects total incremental progress, or Reward-to-Go, which adds a preference for earlier progress.GRPO requires one scalar trajectory score for group-wise normalization.
- 4.4 GROUPED ADVANTAGE ESTIMATION AND OBJECTIVE: Grouped advantages normalize each token’s transformed turn signal across rollout groups before UserRL applies a clipped PPO objective.The design retains GRPO’s comparable-rollout normalization while allowing flexible turn and trajectory reward definitions.
- UserRL generalizes GRPO to multi-turn interactive settings by decoupling turn-level reward shaping from trajectory-level scoring.The framework uses per-turn feedback from simulated user interaction and keeps the optimization pipeline consistent across reward designs.
5 EXPERIMENTS
UserRL evaluates multi-turn user-centric RL by varying trajectory-level scoring and turn-level reward shaping across standardized gym environments. Equalized/R2G performs best overall, while SFT initialization, interaction training, and simulator choice materially affect effectiveness and efficiency.
- 5.2 EXPERIMENT RESULTS: Equalized/R2G consistently achieves the best performance across Qwen3 4B and 8B models, outperforming alternative reward settings.Equalized/Sum performs worst, indicating that trajectory-level scoring is more decisive than fine-grained turn differentiation.
- 5.2 EXPERIMENT RESULTS: Gym-trained models surpass leading closed-source models on some interactive tasks, but remain weaker on tasks requiring external tools and strategic reasoning.Qwen3-8B outperforms Gemini-2.5-Pro and GPT-4o in TravelGym, PersuadeGym, and IntentionGym, while closed-source models lead in TurtleGym, TelepathyGym, and SearchGym.
- 5.2 EXPERIMENT RESULTS: Interaction training makes model scaling more effective: trained 4B and 8B models can exceed the performance gap between raw Qwen3 4B and 32B models.Raw model-size scaling yields only marginal gains without robust user-interaction abilities.
- 5.2 EXPERIMENT RESULTS: Introducing a standardized tool interface substantially lowers performance on TravelGym and TauGym relative to the original UserBench and Tau-Bench settings, even for top closed-source models.The raw test data and evaluation metrics remain unchanged; only the interaction interface is modified.
- 5.3 ANALYSIS: SFT cold start consistently enables RL improvements, with some tasks exceeding 100% gains, whereas models without SFT plateau early.SFT provides initial interaction ability that RL can subsequently refine and extend.
- 5.3 ANALYSIS: GPT-4o simulated users generally produce higher downstream performance than Qwen3-32B, although the cheaper open-source simulator still transfers effectively to GPT-4o evaluation.GPT-4o-based training also shows faster growth and higher plateaus in the reported curves.
- 5.3 ANALYSIS: RL-trained models use more effective interaction turns, but the best model averages only 6.6463 effective turns out of a 16-turn evaluation budget.The result shows improved interaction use alongside substantial remaining headroom.
- 5.3 ANALYSIS: R2G-based models consistently score higher on weighted timing, indicating more efficient acquisition of rewards earlier in interaction.The weighted metric assigns turn i a weight of 1/(i + 1).
6 DISCUSSIONS
UserRL exposes limits in both reward shaping and user simulation: turn-level heuristics can misvalue actions, simulated users must balance flexibility with benchmark rigor, and interaction efficiency can conflict with effectiveness.
- Limits of turn-wise reward differentiation: Turn-level reward heuristics can misjudge productive actions because they do not capture each turn’s contextual role in achieving the final outcome.EM treats all zero-reward turns alike, while R2G assumes later turns matter more; both assumptions can fail.
- Limits of turn-wise reward differentiation: Future reward shaping should be adaptive or learned and preserve the strengths of trajectory-level scoring rather than rely on one universal strategy.
- Balance of rigor and flexibility in user simulation: User simulation must trade response diversity and naturalness against rule-based consistency needed for fair, reproducible comparisons.UserRL combines LLM-driven responses with rule-based task-completion tracking to manage this tension.
- Balance of efficiency and effectiveness in user interaction: Effective Turns and Time-Weighted Performance capture a real tension: using many turns may improve effectiveness, while excessive probing can make conversations inefficient.Training and evaluation should therefore encourage agents to be both accurate and concise.
7 CONCLUSION
UserRL introduces a standardized framework for training and evaluating user-centric agents, then identifies reward, initialization, and simulation choices that improve multi-turn interaction. The paper also reports transfer across simulated and some real-user evaluations while outlining directions for richer environments and reward design.
- UserRL provides eight gym environments and a standardized interaction interface for training and evaluating user-centric agents.
- Experiments find that SFT cold start sustains RL improvements, trajectory-level scoring outperforms fine-grained turn differentiation, and LLM user simulators support scalable training.
- Trained models improve effectiveness and efficiency in multi-turn interactions and transfer across different simulated users and, in some evaluations, real users.
- Future work targets richer gyms, reward formulations that jointly capture effectiveness and efficiency, and further framework extensions.
A GYM CONSTRUCTION DETAILS
The gym suite operationalizes diverse user-centric abilities through specialized environments, LLM-based interaction evaluation, and reward mechanisms tied to task progress. A shared interaction interface supports varied action spaces while preserving reproducible, environment-specific evaluation.
- IntentionGym: IntentionGym tests whether agents uncover missing user intent through targeted clarification, rewarding important details and penalizing unfocused questions.
- IntentionGym: IntentionGym generates natural user responses and separately evaluates which missing details each question addresses.
- PersuadeGym: PersuadeGym evaluates strategic argumentation by measuring stance changes across multi-round conversations, with exponential scaling favoring substantial shifts.
- TurtleGym: TurtleGym evaluates creative reasoning and contextual adaptation through interactive story puzzles requiring strategic questions and explanations.
- TelepathyGym: TelepathyGym tests strategic hypothesis narrowing through yes/no questions and final entity guesses, assigning binary reward to exact matches.
- TravelGym: TravelGym evaluates preference elicitation and personalized planning across multiple travel-booking dimensions.
B TRAINING EXPERIMENT DETAILS
The training setup uses GPT-4o-distilled SFT data before RL, standardized tool interaction, and fixed evaluation conditions. Additional experiments compare cold-start and user-simulation choices within the training pipeline.
- Training Data: The SFT cold start uses 1K trajectories distilled with GPT-4o acting as both agent and simulated user before RL training.The trajectories are selected from five training environments after ranking interaction quality.
- Training Configuration: Training uses separate SFT and RL configurations, with SFT taking approximately 1 hour on 4 H200 GPUs and RL approximately 1.5 days on 8 H200 GPUs.
- Training Configuration: Additional experiments compare training curves with and without SFT cold start under the EM/R2G setting.
- Instructions and Tools: The standardized interact-with-environment tool exposes environment-specific actions through a common function interface for step-by-step interaction.
- Evaluation Configuration: Evaluation fixes model temperature at 0.0 and allows at most 16 interaction turns, while training and evaluation share each gym’s system prompt.
C ANALYSIS DETAILS
The analysis details explain how training curves and user simulations are configured, then examine supplementary SFT results and preliminary real-user testing. The results support SFT cold start while marking participant diversity as an evaluation boundary.
- Training curves: Training curves report summed turn-level rewards as normalized comparisons rather than raw training signals, with Gaussian smoothing and deviation bands.The plotted curves use consistent reward aggregation even when the underlying training signals differ.
- User simulation: Qwen3-32B is used for training-time user simulation because roughly 4M requests make closed-source simulation prohibitively expensive.GPT-4o is reserved for inference, where stronger instruction-following and more realistic behavior support final evaluation and transfer testing.
- Real user participation: Real-user testing recruited five unpaid computer-science PhD students who acted as task oracles across TelepathyGym and TurtleGym.The users sometimes provided subtle hints despite instructions, and the authors describe the test as preliminary because other participants may behave differently.
- Additional experimental results: SFT-initialized Qwen3 4B and 8B models consistently outperform non-SFT models across additional reward-shaping settings.The supplementary EM/R2G experiments reinforce SFT initialization as important for sustaining user-centric RL training.