Source-linked AI summary
UserBench: An Interactive Gym Environment for User-Centric Agents
Cheng Qian, Zuxin Liu, Akshara Prabhakar, Zhiwei Liu, Jianguo Zhang, Haolin Chen, Heng Ji, Weiran Yao, Shelby Heinecke, Silvio Savarese, Caiming Xiong, Huan Wang
TL;DR
Although LLM agents have advanced in reasoning and tool use, evaluating their alignment with vague, evolving, and indirect user goals remains limited. UserBench addresses this gap with a multi-turn, preference-driven environment using simulated users and tools. Its evaluation finds that current models often complete tasks without uncovering user intent, exposing a gap between technical execution and communicative alignment.
Problem
Existing agent evaluations rarely measure whether agents interpret and align with users’ underlying, evolving intent, despite strong task performance.
Method
UserBench evaluates agents in multi-turn travel-planning interactions where simulated users begin with underspecified goals, reveal preferences incrementally, and communicate indirectly through tool-mediated dialogue.
Results
Current models provide answers fully aligned with all user intents only 20% of the time on average and actively elicit less than 30% of user preferences at best.
Takeaways & Limitations
UserBench exposes that tool-proficient LLM agents still struggle to uncover evolving user intent and respond effectively to implicit signals.
Takeaways & Limitations
Performance gains can reflect sampling luck rather than robust reasoning, making pass-k sampling costly and deployment reliability challenging.
Abstract
from arXiv · showhide
Large Language Models (LLMs)-based agents have made impressive progress in reasoning and tool use, enabling them to solve complex tasks. However, their ability to proactively collaborate with users, especially when goals are vague, evolving, or indirectly expressed, remains underexplored. To address this gap, we introduce UserBench, a user-centric benchmark designed to evaluate agents in multi-turn, preference-driven interactions. UserBench features simulated users who start with underspecified goals and reveal preferences incrementally, requiring agents to proactively clarify intent and make grounded decisions with tools. Our evaluation of leading open- and closed-source LLMs reveals a significant disconnect between task completion and user alignment. For instance, models provide answers that fully align with all user intents only 20% of the time on average, and even the most advanced models uncover fewer than 30% of all user preferences through active interaction. These results highlight the challenges of building agents that are not just capable task executors, but true collaborative partners. UserBench offers an interactive environment to measure and advance this critical capability.
1 Introduction
Existing agent evaluations emphasize tool use and task execution, but UserBench targets the harder problem of aligning with vague, evolving, and implicit user intent through interaction. It introduces a multi-turn benchmark and finds substantial gaps between task performance and user alignment.
- Motivation: Existing agentic environments rarely assess whether agents interpret and align with users’ underlying, evolving intent.Agents may achieve strong task performance while failing to understand, adapt to, or collaborate with task initiators.
- Motivation: User instructions commonly exhibit underspecification, incrementality, and indirectness, making intent difficult to convey in one interaction.Goals may begin incompletely specified, emerge across turns, or be expressed implicitly for social or strategic reasons.
- UserBench: UserBench simulates multi-turn interactions in which users begin with vague instructions, gradually reveal preferences, and communicate some intentions implicitly.Agents must clarify goals, interpret subtle cues, and adapt their tool-assisted reasoning.
- Results: 20% of answers fully align with all user intents on average, while the best models elicit less than 30% of user preferences through active querying.These results indicate that current models struggle to uncover and act on evolving user preferences despite tool-use capabilities.
- Implication: UserBench is positioned as a foundation for agents that collaborate with users and align with nuanced human intent rather than merely execute tasks.The intended target is collaborative behavior that interprets subtle cues through meaningful interaction.
2 Related Work
Related benchmarks increasingly evaluate agents under user-centric and dynamic interaction conditions, addressing goals that are underspecified, incremental, or indirect. UserBench extends this direction with a scalable construction pipeline for grounded preference-driven scenarios.
- User-centric evaluation: Traditional benchmarks often assume fully specified prompts and overlook underspecified, incremental, or indirect user goals.User-centric benchmarks instead examine alignment with nuanced human intent under more realistic communication conditions.
- User-centric evaluation: User Reported Scenarios, Intention-in-Interaction, and WildBench evaluate preference satisfaction, multi-intent understanding, and alignment with nuanced real-world queries.These benchmarks compile real user queries and emphasize in-the-wild conditions.
- Dynamic interaction: MINT, PrefEval, τ-Bench, and τ 2-Bench focus on dynamic, multi-turn interactions between agents and users.Their emphasis complements datasets built from real user queries by testing interaction over time.
- UserBench construction: UserBench’s construction pipeline combines data gathering, preference curation, tool augmentation, environment setup, and interface design.Random Preference Combinations make the construction process highly scalable.
3 UserBench
UserBench is a Gymnasium-based travel-planning environment built from implicit preferences, controlled search tools, and multi-turn oracle-user interactions. Its design supports scalable scenario generation, user-centric evaluation, and extensions for noisy tools and budget-aware decisions.
- Environment: UserBench builds on the standardized Gymnasium environment and focuses on travel planning with gradually revealed implicit preferences.Compared with prior work, it emphasizes diverse, grounded preferences and tool-augmented search with noise and cost awareness.
- Data construction: The benchmark covers five travel aspects—flight, hotel, apartment, car rental, and restaurant—and collects realistic, diverse, implicitly expressed preferences.Each preference is paired with multiple naturally phrased statements that imply rather than explicitly state the intended constraint.
- Data construction: Randomly combined preferences across five aspects form scenarios in three difficulty tiers, each paired with an ideal arrangement statement.GPT-4o generates the arrangement statement incorporating the relevant preferences.
- Tools: Aspect-specific search tools return pre-generated options, providing stable and controlled outputs instead of querying real-time data.The controlled design keeps the focus on user-centric reasoning rather than online-data challenges.
- Tools: Each scenario mixes correct, wrong, and noisy options, with databases containing up to over 10 options per aspect.Wrong options violate at least one preference, while noise options may be incomplete, irrelevant, or mismatched to the query.
- Interaction: At reset, the environment reveals only basic trip information; an oracle user then elicits hidden preferences across multi-turn interaction.This setup exposes underspecification, incrementality, and indirectness as explicit reasoning challenges.
- Interaction: Agents use action, search, and answer operations to communicate, query travel options, and provide final responses.Search arguments are matched against ground truth, while action typically supports clarifying questions.
- Extensions: UserBench supports configurable search noise and budget-aware rewards that favor correct and cost-effective choices.A hyperparameter controls the strength of the practical-utility reward.
4 Experiments
UserBench evaluates open- and closed-source models in single- and multi-choice travel-planning settings using task scores and interaction metrics that measure search validity and preference elicitation. Results show that multiple answer opportunities improve scores, but models still struggle to uncover preferences proactively, despite stronger performance on valid search attempts.
- Experimental setup: Models are evaluated in single-choice and multi-choice settings across open- and closed-source model families, with responses generated deterministically.The evaluation includes GPT, Claude, Deepseek, Gemini, Qwen3, and Llama3 models ranging from 8B to 70B, all using temperature 0.0.
- Metrics: The normalized score averages the reward of selected travel options, assigning 1.0 to the best option, 0.8 to another correct option, and 0.0 otherwise.In multi-choice evaluation, the highest reward among selected options is used for each aspect; single-choice considers only the first option.
- Metrics: Auxiliary metrics measure whether selected options include the best or a correct choice, whether search queries are valid, and whether actions successfully probe user preferences.Preference Elicited measures the percentage of ground-truth preferences revealed, including active and passive elicitation.
- Main findings: Approximately 40%: average scores drop when switching from multi-choice to single-choice evaluation, showing that selecting one strong answer is substantially harder.Allowing multiple answers generally improves performance by increasing the chance of including a correct option.
- Main findings: Preference elicitation does not significantly improve in multi-choice evaluation and sometimes declines, indicating that higher scores may reflect guessing or brute-force coverage rather than active reasoning.Overall preference elicitation remains low, especially for preferences revealed through active queries.
- Main findings: Top-performing models exceed 80% on valid search attempts, while valid action attempts are much lower, indicating that understanding users is harder than executing tool use.Valid action attempts are computed only over action steps and reflect whether questions are precise and preference-relevant.
5 Analysis
UserBench performance declines as preference complexity increases, especially when multiple preferences are concentrated within an aspect. Models also struggle to balance timely preference elicitation with reliable, user-aligned decisions.
- Difficulty and preference complexity: Scores generally decline as scenario difficulty increases from Easy to Hard, indicating weaker robustness in complex user interactions.
- Difficulty and preference complexity: More preferences per aspect consistently reduce model scores, making preference density a main difficulty driver.
- Difficulty and preference complexity: With total preferences fixed, distributing fewer preferences across more aspects improves performance compared with concentrating them within fewer aspects.
- Interaction and sampling: Models face a trade-off between discovering valid answers quickly and investing enough turns to understand evolving preferences.
- Interaction and sampling: More interaction turns do not guarantee better performance and can sometimes degrade results when models fail to use the extra turns productively.
- Interaction and sampling: More samples increase maximum score, but average score remains flat or declines, indicating reliance on sampling luck rather than robust reasoning.
- Choice distribution: Reducing wrong and noise options generally improves scores, but removing about 30% of distractors yields only modest gains for strong models.
6 Discussions
UserBench is designed as a flexible Gym-based environment for evaluating and training agents on user-aligned interaction. Its analyses emphasize the need to balance prompt responses with deeper preference understanding, while exposing behaviors that targeted reward design can address.
- Environment and training: UserBench’s standard Gym interface supports modular evaluation and training while modeling user behavior as a black-box environment.
- Environment and training: Customizable options, rewards, feedback, and turn-wise signals support fine-grained behavioral analysis and supervised or reinforcement learning.
- Efficiency and effectiveness: Weighted timing analysis shows that agents may guess early and appear helpful while overlooking deeper user intent, producing repeated prompt revisions and dissatisfaction.
- Efficiency and effectiveness: UserBench does not explicitly quantify the efficiency-effectiveness balance, but it surfaces behaviors that targeted training can address.
- Efficiency and effectiveness: Penalizing late preference discovery and rewarding partial preference elicitation can discourage guessing and encourage deliberate, user-aware interaction.
7 Conclusion
UserBench evaluates whether agents can understand, interact with, and adapt to realistic user communication, exposing a gap between tool-based task execution and communicative intelligence.
- UserBench targets user communication involving underspecification, incrementality, and indirectness.
- The benchmark shows that tool proficiency does not ensure that agents uncover evolving intent or respond effectively to implicit signals.
- UserBench is intended to support agents that actively collaborate, interpret subtle cues, and align with users through adaptive interactions.
A Comparison Traits Details
UserBench combines user-centered interaction requirements with infrastructure features for configurable, scalable evaluation across realistic multi-step tasks.
- Interaction traits: UserBench requires extended conversations in which agents clarify, refine, or improve responses through feedback.
- Interaction traits: Tasks begin with vague or indirect goals, requiring agents to ask questions, infer missing information, and resolve ambiguity over time.
- Agent capabilities: Agents use external tools while coordinating decisions across multiple interdependent subgoals.
- Environment capabilities: The environment updates an internal state across time and includes simulated users that respond to agent queries or actions.
- Benchmark infrastructure: UserBench spans multiple domains, tools, and task structures to support generalization beyond a fixed scenario.
- Benchmark infrastructure: The framework is customizable and scalable, allowing researchers to adjust settings and generate large-scale reproducible evaluations.
B Gym Construction Details
UserBench constructs travel-planning interactions with curated options, simulated users, and controlled tool behavior to test preference-aware search and decision-making.
- Option generation: UserBench generates correct, wrong, and noise options from the same user scenario.
- Option generation: Under budget constraints, the best option is the one with the lowest total cost after all relevant charges, including upgrades.
- Option generation: Wrong options violate user preferences, whereas noise options are unrelated to the search intent or unrealistic.
- Environment setting: The environment simulates users who gradually and implicitly reveal preferences and evaluates whether search queries match ground-truth arguments.
- Environment setting: Model utterances are classified into four intent types, including targeted preference questions, vague preference queries, and ordinary conversation.
- Environment setting: Targeted questions can reveal preferences, while vague questions receive corrective responses and off-topic behavior can trigger passive preference elicitation.
- Evaluation: Submitted answers are parsed with rules, compared against ground truth, and mapped to rewards.
C Experiment Details
The experiments standardize model generation and tool interaction while using configurable rewards, choice settings, and metrics for search, clarification, and preference elicitation.
- Settings: Evaluations use temperature 0, one sample, a 2048-token response limit, and at most 20 interaction turns.
- Tool interaction: Models interact through the required interact_with_env tool using search, action, or answer choices.
- Environment configuration: The basic configuration uses 20 maximum steps, search failures every fifth search, and proactive elicitation after three off-topic turns.
- Reward configuration: Rewards distinguish aligned searches, successful preference questions, best choices, and correct but suboptimal choices.
- Metrics: Best Exist Rate measures selection of best options, while Correct Exist Rate counts any option that satisfies user preferences.
- Metrics: Valid Search Attempt measures effective search queries, while Valid Action Attempt measures targeted clarifying questions.
- Metrics: Valid Action Attempt is relative, so a high rate does not necessarily mean that many preferences were elicited.
- Metrics: Preference Elicited counts preferences obtained through both valid questions and passive environment release.
D Analysis Details
The analysis details examine positive and negative UserBench interactions, alongside prompts for judging search alignment, utterance types, preference revelation, and agent behavior. Case studies show that effective agents clarify evolving preferences and refine searches, while weaker behavior can remain structurally valid yet poorly aligned.
- Case studies: Qwen3-32B recovers from a vague initial action by asking targeted follow-up questions, conducting precise searches, and comparing viable options.The case study states that its reasoning aligns with user preferences and budget constraints before the correct selection.
- Case studies: Claude-4-Sonnet elicits nuanced preferences through stepwise clarification, adapts inquiries using prior feedback, and recommends an option matching stated and implied travel needs.The interaction is presented as evidence of adaptability and goal-oriented reasoning in multi-aspect travel planning.
- Case studies: Deepseek-V3 issues structurally valid searches but fails to elicit preferences, narrows constraints inadequately, repeats selections, and chooses suboptimal options.The case attributes these behaviors to shallow user understanding and poor state tracking.
- Evaluation prompts: The evaluation setup uses prompts to judge search-argument alignment, classify utterances and preference behavior, and generate responses that implicitly reveal user preferences.Additional prompts support proactive preference introduction, neutral responses without preferences, and separate single-choice and multi-choice agent instructions.