Source-linked AI summary
VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applications
Wei He, Yueqing Sun, Hongyan Hao, Xueyuan Hao, Zhikang Xia, Qi Gu, Chengcheng Han, Dengchang Zhao, Hui Su, Kefeng Zhang, Man Gao, Xi Su, Xiaodong Cai, Xunliang Cai, Yu Yang, Yunke Zhao
TL;DR
Existing benchmarks inadequately capture the information, tool, and interaction complexity of real-world LLM-agent tasks. VitaBench addresses this gap with a multi-domain benchmark and rubric-based sliding-window evaluator, finding that advanced models achieve only 30% success on cross-scenario tasks and less than 50% on single-scenario tasks.
Problem
Existing benchmarks do not adequately capture agents’ need to handle extensive information, diverse tools, inter-tool dependencies, and dynamic user interactions in real-world applications.
Method
VitaBench composes 66 tools across delivery, in-store consumption, and online travel scenarios, using multi-request environments and a rubric-based sliding-window evaluator.
Results
30% success rate is achieved on cross-scenario tasks, while advanced models achieve less than 50% on single-scenario tasks.
Takeaways & Limitations
VitaBench provides a challenging testbed and actionable insights for advancing agents in practical real-world applications.
Takeaways & Limitations
Some models could not be evaluated because of API stability concerns, and the authors are working to include the latest models.
Abstract
from arXiv · showhide
As LLM-based agents are increasingly deployed in real-life scenarios, existing benchmarks fail to capture their inherent complexity of handling extensive information, leveraging diverse resources, and managing dynamic user interactions. To address this gap, we introduce VitaBench, a challenging benchmark that evaluates agents on versatile interactive tasks grounded in real-world settings. Drawing from daily applications in food delivery, in-store consumption, and online travel services, VitaBench presents agents with the most complex life-serving simulation environment to date, comprising 66 tools. Through a framework that eliminates domain-specific policies, we enable flexible composition of these scenarios and tools, yielding 100 cross-scenario tasks (main results) and 300 single-scenario tasks. Each task is derived from multiple real user requests and requires agents to reason across temporal and spatial dimensions, utilize complex tool sets, proactively clarify ambiguous instructions, and track shifting user intent throughout multi-turn conversations. Moreover, we propose a rubric-based sliding window evaluator, enabling robust assessment of diverse solution pathways in complex environments and stochastic interactions. Our comprehensive evaluation reveals that even the most advanced models achieve only 30% success rate on cross-scenario tasks, and less than 50% success rate on others. Overall, we believe VitaBench will serve as a valuable resource for advancing the development of AI agents in practical real-world applications. The code, dataset, and leaderboard are available at https://vitabench.github.io/
1 Introduction
VitaBench addresses the gap between controlled benchmarks and real-world agent deployment by modeling reasoning, tool, and interaction complexity. It evaluates agents across composed scenarios and finds substantial performance weaknesses, especially in cross-scenario tasks.
- Existing benchmarks often emphasize function-calling accuracy or rigid domain policies while overlooking inter-tool dependencies and autonomous exploration.VitaBench is motivated by the gap between controlled laboratory settings and complex real-world deployments.
- The benchmark targets reasoning, tool, and interaction complexity, including environmental information, inter-tool dependencies, and diverse multi-turn user behavior.These dimensions are intended to characterize the complexity agents face in real-world applications.
- VitaBench composes 66 tools across delivery, in-store consumption, and online travel domains into flexible single- and cross-scenario tasks.Tasks derive from multiple authentic user requests and use independent environments with user profiles, spatiotemporal contexts, and service databases.
- 30.0% success rate is achieved by the best-performing model on cross-scenario tasks, versus 48.3% on single-scenario tasks.Cross-scenario settings require navigating different domain contexts and expanded action spaces.
- Reasoning errors dominate failures at 61.8%, followed by tool usage errors at 21.1% and interaction management failures at 7.9%.The analysis also reports poor self-awareness and limited error recovery capabilities.
2 Related Work
Prior benchmarks increasingly address tool use, reasoning, and user interaction, but commonly constrain autonomy or evaluate these complexity dimensions separately. VitaBench compares existing interaction benchmarks across reasoning, tool, and interaction coverage.
- Table 1 marks whether prominent user-interaction benchmarks fully, partially, or do not address reasoning, tool, and interaction complexity.The table uses ✓, ✓–, and ✗ to indicate full, partial, and absent coverage.
- Existing benchmarks typically address reasoning, tool manipulation, and interaction abilities in isolation rather than comprehensively.Table 1 organizes the comparison across the three task-complexity dimensions.
- ToolTalk supports multi-step conversational tool execution but relies on predefined dialogue trajectories, limiting agent autonomy.MINT and IN3 likewise operate in relatively constrained agentic settings.
3 VitaBench: A Benchmark for Versatile Interactive Tasks
VitaBench formalizes real-world agent tasks as partially observable, interactive environments and constructs them across three complexity dimensions. Its benchmark pipeline combines scenario tools, evolving user behavior, multi-request tasks, and rubric-based sliding-window evaluation.
- 3.1 Formulation: VitaBench models each task as a POMDP with tool and dialogue actions, database and user state, partial observations, and stochastic user transitions.API calls use deterministic database transitions, while user interactions use stochastic language-model transitions.
- 3.1 Formulation: Task complexity is formalized as reasoning, tool, and interaction complexity, covering partial observability, dependency-graph structure, and dynamic multi-turn conversations.The three dimensions collectively guide benchmark design and evaluation.
- 3.2 Benchmark Construction: VitaBench constructs delivery, in-store consumption, and online travel scenarios using simplified APIs whose interdependencies are represented as a directed graph.Tool descriptions include preconditions and related state information to capture real-world functionality.
- 3.2 Benchmark Construction: Simulated users reveal multi-requirement instructions progressively and expose implicit constraints only when agents inquire, while profiles encode varied attributes and behaviors.Agents cannot directly access some information, such as dietary restrictions, and must infer it from history or user responses.
- 3.2 Benchmark Construction: The benchmark pipeline combines anonymized user profiles, multi-request instructions, environmental information, and task rubrics.These components support diverse conversational dynamics and evaluation of multiple valid solution paths.
- 3.3 Rubric-based Sliding Window Evaluator: The sliding-window evaluator processes long trajectories in overlapping segments while persisting rubric state across windows.Atomic criteria support detailed scoring, and human evaluation reports Cohen’s κ ≥0.81 inter-rater agreement.
4 Experiments
VitaBench evaluates function-calling agents across diverse real-world domains using repeated stochastic trials and complementary effectiveness, consistency, and efficiency measures. Results show severe cross-scenario difficulty, unstable success across samples, and advantages for thinking models.
- 4.1 Experimental Setups: Each main-result task is run four times at temperature 0.0, with Avg@4, Pass@4, and Passˆ4 reported across tasks.Agents use OpenAI tool schemas, while separate models implement the user simulator and evaluator.
- 4.1 Experimental Setups: API stability concerns prevent evaluation of some models, and the project reports that updated results may appear on its website.The limitation specifically affects current evaluation coverage.
- 4.2 Main Results: 30.0% Avg@4 is achieved by top models on cross-scenario tasks, versus over 50% in single-domain settings, exposing major difficulty in expanded action spaces.The leaderboard compares non-thinking and thinking models across domains, with cross-scenario performance as the sorting criterion.
- 4.2 Main Results: Pass@4 improves with sampling, but Passˆ4 falls to near-zero consistency for even top models, revealing unstable trajectories in complex environments.Evaluations with k = 32 samples confirm only marginal exploration gains and persistent stability challenges.
- 4.2 Main Results: Thinking models improve both performance and interaction efficiency, averaging 23.8% versus 17.9% while using 61.1 versus 69.9 turns.Claude-4.1-Opus rises from 21.8% to 29.0%, and GLM-4.5 rises from 20.0% to 22.8%.
5 Discussion
Discussion analyses evaluator reliability, environmental complexity, interaction effects, and failure patterns. Reasoning and tool complexity strongly constrain performance, while reasoning errors dominate observed failures.
- 5.1 Reliability Analysis: The user simulator scores 9.48/10 for information fidelity and 9.34/10 for persona-behavior alignment across evaluated conversations.Cooperative personas show the highest consistency, while scattered personas show lower controllability.
- 5.1 Reliability Analysis: The rubric-based sliding-window evaluator is compared with full-trajectory and no-rubric configurations against human-annotated ground truth.The ablation includes an external memory module for the sliding-window, no-rubric configuration.
- 5.1 Reliability Analysis: Repeated-run resampling is used to assess evaluation stability because small perturbations can diverge across multi-turn trajectories.The analysis uses 32 independent trials and computes MSE for run counts from 1 to 20.
- 5.2 Task Complexity Analysis: 16.2% performance accompanies the highest tool complexity in cross-scenario tasks, which use 66 tools and 512 dependency edges.Cross-scenario tasks also require 10.3 reasoning points, while the in-store domain reaches 42.1% with fewer reasoning points.
- 5.2 Task Complexity Analysis: Interaction complexity affects models differently: Claude-4-Sonnet benefits more from solo instructions, whereas conversational styles challenge GPT-4.1-Mini more strongly.The ablation compares default, neutral-user, and solo-agent conditions.
- 5.3 Error Pattern Analysis in VitaBench: 61.8% of failures are reasoning errors, followed by tool-use errors at 21.1%, interaction errors at 7.9%, and user-simulator errors at 9.2%.The categories cover decision-making, tool selection and recovery, dialogue management, and simulator stochasticity.
6 Conclusion
VitaBench reframes agent evaluation around real-world task complexity across reasoning, tool use, and interaction. Its results show that advanced models still struggle especially with cross-scenario tasks.
- 6 Conclusion: 30% success is achieved under cross-scenario settings, while single-scenario settings remain below 50% for advanced models.VitaBench provides a life-serving simulation environment with 66 tools and 400 tasks spanning both settings.
Contributions
The supplied contribution passages identify the paper’s author-list and affiliation notes. They provide no substantive methodological or empirical contribution claim.
- Contributions: Authors without explicit affiliations are identified as belonging to Meituan, and Wei He is noted as an intern during the work.The author list is stated to be alphabetical.
A Comparison Traits Details
VitaBench-related prompts specify how agents, simulated users, and evaluators should conduct and assess multi-turn tool-based tasks. The templates enforce gradual information disclosure, rubric tracking, and completion checks.
- Tool Usage Guidelines: Tool-based evaluation requires checking parameters, using tools to obtain missing information, and satisfying preconditions and postconditions.The guidelines distinguish tool results from assistant recommendations and warn that apparent ordering success may be incorrect.
- User Simulation System Prompt: The user simulation prompt models a persona-driven user who conveys instructions through dialogue.It requires the simulator to reflect the persona while communicating the content of the instructions.
- Information Disclosure Rules: User conversations disclose requirements gradually across rounds rather than revealing every need immediately.The simulator breaks information into separate points and avoids disclosing all needs in the first round.
- Conversation Guidelines: Conversations end only after all stated needs and tasks are completed correctly, or the system explicitly cannot complete them.After completion, the assistant asks whether there are additional needs and emits a stop mark when the user indicates there are none.
- Sliding Window Evaluator System Prompt: The evaluator updates rubric status from conversation windows, allowing completed goals to be marked true and overturned conclusions to revert to false.Each window contains 10 turns with 2 overlapping turns, and rubric states are updated from the current window.
C An Example Trajectory
The example trajectory spans restaurant booking, delivery coordination, and train booking for a family trip. It combines persona constraints, implicit requirements, and temporal and spatial coordination across 66 tools.
- C An Example Trajectory: The trajectory coordinates restaurant reservation, delivery, and train booking across multiple domains for family logistics.The tasks must be handled together rather than as an isolated single-domain interaction.
- C An Example Trajectory: The scenario combines explicit and implicit constraints, including dietary restrictions, accessibility, and suitability for three generations.Some requirements require inference about the needs of elderly family members and children.
- C An Example Trajectory: The schedule requires aligning train arrival before 11 AM, delivery at noon, and boarding at 3 PM.These time-sensitive activities must be coordinated within the same family itinerary.
- C An Example Trajectory: The agent must navigate 66 tools while satisfying temporal and spatial constraints through multi-turn dialogue, illustrating VitaBench’s realistic complexity.The example includes distance calculations between locations and coordination of delivery logistics.
User Profile
The example user is a married 30–35-year-old blue-collar worker in Harbin preparing a three-generation cruise trip. The request combines dietary, accessibility, timing, and booking constraints.
- User Profile: The user is a married male blue-collar worker aged 30–35 who resides and works in Harbin.The profile includes home and work locations in Harbin.
- User Profile: The user avoids high-purine and fried foods because of dietary restrictions.The listed restrictions exclude organ meats, seafood soup, and fried foods.
- User Profile: The family plans to board a cruise in Dalian on the 27th at 3 PM and wants a nearby restaurant beforehand.The restaurant must accommodate three generations, provide accessibility facilities, and support a table reservation for six at noon.