PaperlayerView exact revision on arXiv

Source-linked AI summary

VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applications

Wei He, Yueqing Sun, Hongyan Hao, Xueyuan Hao, Zhikang Xia, Qi Gu, Chengcheng Han, Dengchang Zhao, Hui Su, Kefeng Zhang, Man Gao, Xi Su, Xiaodong Cai, Xunliang Cai, Yu Yang, Yunke Zhao

arXiv:2509.26490v2cs.CLcs.AI

TL;DR

Existing benchmarks leave a gap between controlled settings and real-world agent deployments; VitaBench evaluates interactive tasks across composed application scenarios. Even advanced models achieve only 30% success rate under cross-scenario settings.

  • Problem

    Existing benchmarks leave a gap between controlled settings and real-world deployments, with limited coverage of complex tool interdependencies and unconstrained agent exploration.

    Source: §1 · p. 2

  • Method

    VitaBench models task complexity across reasoning, tool, and interaction dimensions and builds a benchmark with 66 tools and 400 single- and cross-scenario tasks.

    Source: §1 · p. 2

  • Results

    Even advanced models achieve only 30% success rate under cross-scenario settings.

    Source: §6 · p. 10

  • Takeaways & Limitations

    VitaBench offers a challenging testbed for evaluating progress on real-world agent applications.

    Source: §6 · p. 10

    API stability concerns currently prevent evaluation of some models on VitaBench.

    Source: §4.1 · p. 6

Abstract

from arXiv · show

As LLM-based agents are increasingly deployed in real-life scenarios, existing benchmarks fail to capture their inherent complexity of handling extensive information, leveraging diverse resources, and managing dynamic user interactions. To address this gap, we introduce VitaBench, a challenging benchmark that evaluates agents on versatile interactive tasks grounded in real-world settings. Drawing from daily applications in food delivery, in-store consumption, and online travel services, VitaBench presents agents with the most complex life-serving simulation environment to date, comprising 66 tools. Through a framework that eliminates domain-specific policies, we enable flexible composition of these scenarios and tools, yielding 100 cross-scenario tasks (main results) and 300 single-scenario tasks. Each task is derived from multiple real user requests and requires agents to reason across temporal and spatial dimensions, utilize complex tool sets, proactively clarify ambiguous instructions, and track shifting user intent throughout multi-turn conversations. Moreover, we propose a rubric-based sliding window evaluator, enabling robust assessment of diverse solution pathways in complex environments and stochastic interactions. Our comprehensive evaluation reveals that even the most advanced models achieve only 30% success rate on cross-scenario tasks, and less than 50% success rate on others. Overall, we believe VitaBench will serve as a valuable resource for advancing the development of AI agents in practical real-world applications. The code, dataset, and leaderboard are available at https://vitabench.github.io/

1 Introduction

VitaBench frames real-world agent task complexity across reasoning, tools, and interaction, and evaluates agents on composed, multi-turn tasks; even the best-performing model remains far from reliable success.

Motivation

  • Prior benchmarks often miss tool interdependencies or user uncertainty, while recent real-world efforts impose rigid domain policies and constrained actions.The introduction contrasts early tool-use benchmarks [Qin et al., 2024, Patil et al., 2025] with recent efforts [Yao et al., 2024, Barres et al., 2025].Source: §1 · p. 2

Complexity framework

  • The benchmark characterizes task complexity through reasoning, tool, and interaction dimensions, with tool dependencies modeled as graphs.Reasoning complexity reflects environmental information volume, tool complexity reflects graph node cardinality and edge density, and interaction complexity reflects user attributes and conversational patterns across turns.Source: §1 · p. 2

Benchmark design

  • VitaBench combines 66 tools across delivery, in-store consumption, and online travel to create 400 single- and cross-scenario evaluation tasks.Tasks draw on multiple authentic user requests and independent environments with annotated profiles, spatiotemporal contexts, and service databases; a rubric-based sliding-window evaluator scores long-horizon trajectories with multiple valid pathways.Source: §1 · p. 2, Figure 2 · p. 2

Results

  • 48.3% success on single-scenario tasks fell to 30.0% on cross-scenario tasks for the best-performing model.In cross-scenario settings, agents must navigate between domain contexts and select tools from expanded action spaces.Source: §1 · p. 2

Failure analysis

  • Reasoning errors account for 61.8% of failures, tool-use errors 21.1%, and interaction-management failures 7.9%.The analysis reports strong correlations between complexity metrics and task difficulty, alongside poor agent self-awareness and limited error recovery; validation supports evaluator reliability.Source: §1 · pp. 2–3

2 Related Work

Prior benchmarks address tool use, reasoning, and interaction incompletely or in isolation, while VitaBench targets their integration in real-world tasks requiring autonomous exploration and dynamic user interaction.

2 Related Work

  • Table 1 compares prominent user-interaction benchmarks across reasoning, tool, and interaction complexity, marking each dimension as fully, partially, or not addressed.The table caption defines ✓ as fully addressed, ✓ – as partially addressed, and ✗ as not addressed.Source: Table 1 · p. 3
  • Early tool-use benchmarks [Huang et al., 2024, Qin et al., 2024, Patil et al., 2025] emphasized single-turn API accuracy, while newer benchmarks often address capabilities in isolation.This leaves inter-tool dependencies and dynamic user interactions—features of real-world applications—underrepresented.Source: §2 · p. 3, Table 1 · p. 3
  • ToolTalk [Farn and Shin, 2023] uses predefined dialogue trajectories; MINT [Wang et al., 2024a] and IN3 [Qian et al., 2024] address feedback and implicit intent in constrained settings.ToolTalk introduces multi-step tool execution through conversational interfaces.Source: §2 · p. 3
  • ToolSandbox [Lu et al., 2025] and the τ-bench family [Yao et al., 2024, Barres et al., 2025] model stateful execution and tool interdependencies but constrain agents with verbose policies.DialogTool [Wang et al., 2025] focuses primarily on agent-side capabilities, whereas UserBench [Qian et al., 2025] captures preference-driven interaction but has limited task complexity.Source: §2 · p. 3
  • Other work [Yang et al., 2024, Wang et al., 2024b] studies recognition of incomplete conditions and proactive information seeking, but no benchmark combines all complexity dimensions.VitaBench aims to fill this gap with information-rich environments for autonomous exploration, dynamic interaction with diverse users, and navigation of intricate tool dependencies.Source: §2 · p. 3

3 VitaBench: A Benchmark for Versatile Interactive Tasks

VitaBench formalizes agents’ interactions with databases and simulated users, then builds composite real-world tasks and evaluates trajectories with a rubric-based sliding-window method. Its design targets reasoning, tool-use, and interaction complexity across delivery, in-store consumption, and online travel scenarios.

3.1 Formulation

  • VitaBench models each environment as a POMDP in which agents act through tool calls and user dialogue, observe database feedback and conversation history, and face deterministic API but stochastic user transitions.The state includes both database and user state.Source: §3.1 · pp. 3–4
  • Task difficulty is characterized along reasoning, tool, and interaction dimensions, covering partial observability, tool-graph structure and cross-domain action spaces, and evolving user states and ambiguity.These dimensions guide benchmark design and evaluation.Source: §3.1 · p. 4

3.2 Benchmark Construction

  • VitaBench models tool dependencies as a directed graph with pre- and post-conditions, encoding domain rules in tool structures instead of verbose policies and enabling cross-domain composition.For example, modifying an order requires first retrieving its details.Source: §3.2 · p. 4
  • Its user simulator progressively reveals multi-requirement instructions and implicit constraints upon inquiry, while persona knowledge boundaries require agents to infer some needs from history or responses.The simulator follows Yao et al. [2024].Source: §3.2 · pp. 4–5
  • The task pipeline constructs 400 composite tasks from real requests and platform data, with distractors, transaction histories, and typically 5–20 providers per task.Some tasks include over 100 products; human-verified refinement preserves multiple valid solution pathways.Source: §3.2 · p. 5

3.3 Rubric-based Sliding Window Evaluator

  • To handle long trajectories and multiple valid paths, VitaBench evaluates atomic rubrics over overlapping windows, persistently tracks satisfied criteria, and requires all criteria for success.Human evaluation reaches Cohen’s κ ≥0.81, and fine-grained rubric scores provide dense signals for reinforcement learning.Source: §3.3 · pp. 5–6

4 Experiments

Experiments evaluate diverse language agents in a tool-calling setup and show that cross-scenario tasks remain difficult, sampling exposes stability problems, and thinking models generally improve performance and efficiency.

4.1 Experimental Setups

  • The evaluation compared proprietary and open models in thinking and non-thinking categories, excluded models under 32B parameters, and used OpenAI-schema function-calling agents.Each main-results task ran four times at temperature 0.0; the user simulator was gpt-4.1-2025-04-14 and the evaluator claude-3.7-sonnet. Some models could not be evaluated because of API stability concerns.Source: §4.1 · p. 6
  • Results averaged across tasks report Avg@4, Pass@4, and Passˆ4, representing average performance, at-least-one success across four trials, and success in all four trials, respectively.The Pass@k and Passˆk definitions follow Yao et al. [2024].Source: §4.1 · p. 6

4.2 Main Results

  • Cross-scenario tasks capped at 30.0% Avg@4 for top models versus over 50% in single-domain settings; in-store tasks were easier than delivery despite larger product pools.The results indicate that delivery tasks’ coordination and strict constraints, rather than database size, contribute to difficulty.Source: Table 3 · p. 7, §4.2 · p. 7
  • Pass@4 sampling improved completion, but Passˆ4 consistency neared zero for top models; k = 32 tests found marginal gains and persistent instability, including for Claude-4-Sonnet.Pass@k measures whether at least one trial succeeds, whereas Passˆk measures whether all trials succeed.Source: §4.2 · pp. 7–8, Figure 4 · p. 8
  • Thinking models generally outperformed non-thinking variants: Claude-4.1-Opus rose from 21.8% to 29.0%, and GLM-4.5 from 20.0% to 22.8%.Average performance was 23.8% for thinking models versus 17.9% for non-thinking models, with 61.1 versus 69.9 turns; thinking aids decomposition and targeted clarification.Source: §4.2 · p. 8, Figure 5 · p. 8

5 Discussion

The discussion examines benchmark-component reliability, how reasoning, tool, and interaction complexity shape task difficulty, and the recurring failures of current agents. It finds reasoning errors dominate failures and recommends four evaluation runs to balance precision and cost.

5.1 Reliability Analysis of VitaBench Components

  • The user simulator scores 9.48/10 for information fidelity and 9.34/10 for persona alignment, with scattered personas less controllable than cooperative ones.It responds “I don’t know” to questions about unprovided information; minor conversational variations do not compromise task requirements.Source: §5.1 · pp. 8–9
  • The sliding-window rubric evaluator achieves Cohen’s κ = 0.828 and 95% task-level accuracy, outperforming methods without rubric structure (κ < 0.07).Full-trajectory evaluation with a rubric yields similar final scores (19% vs. 20%), but limited long-context capability hinders accurate assessment of all rubrics.Source: §5.1 · p. 9, Table 4 · p. 9
  • Four evaluation runs reduce MSE by 77.5% versus one run; eight runs offer only marginal further reduction despite doubling computational overhead.The authors select four runs based on resampling 32 independent trials, balancing statistical precision and cost.Source: §5.1 · p. 9

5.2 Task Complexity Analysis

  • Cross-scenario tasks combine 10.3 reasoning points with 66 tools and 512 dependency edges and have the lowest performance (16.2%), while In-store tasks reach 42.1%.OTA tasks require 9.7 reasoning points and perform at 20.7%; their 22% graph density reflects complex inter-tool dependencies.Source: §5.2 · p. 9, Table 5 · p. 9
  • User interaction adds task difficulty: conversational styles challenge weaker models more, while Claude-4-Sonnet benefits more from receiving complex instructions upfront in solo mode.These effects vary by model capability, supporting interaction complexity as a distinct dimension of task difficulty.Source: §5.2 · p. 10, Figure 8 · p. 10

5.3 Error Pattern Analysis in VitaBench

  • Reasoning errors account for 61.8% of failed rubrics, followed by tool-use errors (21.1%), simulator errors (9.2%), and interaction errors (7.9%).Simulator errors reflect inherent stochastic behavior, which the authors mitigate through multiple runs.Source: §5.3 · p. 10, Figure 9 · p. 10

6 Conclusion

VitaBench evaluates LLM agents on real-world task complexity across reasoning, tool use, and interaction. Even advanced models achieve only 30% success in cross-scenario settings and less than 50% in single-scenario settings.

6 Conclusion

  • 30% success rate under cross-scenario settings and less than 50% under single-scenario settings show that even advanced models struggle on VitaBench.The benchmark is intended as a challenging testbed offering actionable insights for advancing real-world agent applications.Source: §6 · p. 10
  • VitaBench provides a life-serving simulation environment with 66 tools and 400 tasks spanning single- and cross-scenario settings.Source: §6 · p. 10
  • VitaBench bridges controlled benchmarks and practical deployments by formalizing agentic task complexity across reasoning, tool use, and interaction.Source: §6 · p. 10

A Comparison Traits Details · B Prompt Templates

The appendices define nine benchmark-comparison traits spanning task complexity, tool use, cross-scenario capacity, interaction length, and user modeling, then specify prompts for agent behavior, user simulation, and sliding-window evaluation.

A Comparison Traits Details

  • Nine traits characterize benchmark complexity: information integration, composite goals, ambiguity, tool count and dependencies, cross-scenario coverage, turn count, user profiles, and behavior attributes.The traits span task requirements, interaction and tool complexity, and user modeling.Source: A Comparison Traits Details · p. 14

B Prompt Templates

  • The agent prompt directs agents to check whether tool parameters are known, ask users for missing values, retrieve information when users cannot provide it, and respect tool preconditions and postconditions.It also restricts responses to grounded context and focuses the agent on completing the user’s needs.Source: B Prompt Templates · p. 14
  • The user-simulation prompt casts the model as a persona conveying instructions through dialogue, with information disclosed across separate turns rather than all at once.It requires every instruction detail, including background information, to surface over the conversation.Source: B Prompt Templates · pp. 14–15
  • The simulator answers from the persona and instructions without fabrication, sustains the exchange until task completion, and may show impatience after the agent repeats a question three times.It should not end the conversation before needs and constraints are expressed and tasks are correctly completed.Source: B Prompt Templates · p. 15
  • The evaluator processes conversations in 10-turn windows with 2 overlapping turns, updating rubric statuses from the current window and allowing a satisfied status to revert if a correct conclusion is overturned.Rubrics begin false and can become true when the assistant completes a goal in the window.Source: B Prompt Templates · p. 16
  • Rubric checks distinguish successful order placement from an assistant’s mistaken belief that an order succeeded, and require order details to match the stated requirements.Address and order-note matching may use functional equivalence.Source: B Prompt Templates · p. 16

C An Example Trajectory

The example trajectory illustrates how VitaBench tasks combine cross-domain coordination with user-specific, implicit, temporal, and spatial constraints. It requires an agent to manage family logistics through multi-turn dialogue while navigating 66 tools.

C An Example Trajectory

  • The example coordinates restaurant reservation, delivery of elderly-care items, and train booking for family logistics across multiple domains.The task spans restaurant reservation for family dining, delivery for elderly care items, and train booking for family coordination.Source: C An Example Trajectory · p. 17
  • The user’s cold personality and dietary restrictions shape interaction, while requirements such as suitability for three generations and accessibility require inference about elderly people and children.Source: C An Example Trajectory · p. 17
  • The agent must align train arrival before 11 AM, noon delivery, and 3 PM boarding, while reasoning about distances between the port and restaurant and coordinating delivery logistics.Source: C An Example Trajectory · p. 17
  • Managing these family logistics through multi-turn dialogue across 66 tools illustrates the realistic complexity that even state-of-the-art models struggle with.Source: C An Example Trajectory · p. 17

User Profile … Assistant

The profile describes a married man with children in Harbin who avoids certain foods and communicates tersely; the instruction sets out preparations for a three-generation cruise departure from Dalian.

User Profile

  • The user is a 30–35-year-old male blue-collar worker in Harbin, married with children, who avoids high-purine and fried foods and is described as cold, concise, and impatient.The profile also provides his home and work addresses.Source: User Profile · p. 17

Instruction

  • For the family’s cruise departure from Dalian on the 27th, the user seeks an accessible restaurant suitable for three generations and a six-person table at noon.The ship boards at 3 PM; the restaurant should offer dishes suitable for elderly people and children.Source: Instruction · p. 17
  • The user plans to have a walking cane and adult diapers delivered to the restaurant around noon for the elderly relatives to take aboard.Bringing the items to the restaurant themselves would be too troublesome.Source: Instruction · p. 17
  • The user must find his aunt a morning high-speed train ticket from Beijing to Dalian, preferably arriving before 11 AM, with first-class seating.Source: Instruction · p. 17

Related papers

Loading 2509.26490v2…