Source-linked AI summary
Benchmarking LLM Tool-Use in the Wild
Peijie Yu, Wei Liu, Yifan Yang, Jinjian Li, Zelong Zhang, Xiao Feng, Feng Zhang
TL;DR
Existing tool-use benchmarks underrepresent compositional tasks, hidden intent, and instruction transitions found in real user interactions. WildToolBench addresses this gap with a human-verified benchmark grounded in real user behavior, and evaluation of 57 LLMs finds no mainstream model above 15% session accuracy. The results identify realistic user behavior as a central difficulty for current tool-use systems.
Problem
Existing benchmarks overlook compositional tasks, implicit intent across dialogue, and instruction transitions that characterize real user behavior.
Method
WildToolBench constructs human-verified scenarios and evaluates tool use in dialogues grounded in real user behavior patterns.
Results
15% session accuracy is the highest level reached by mainstream LLMs, while multi-step orchestration and later dialogue tasks remain difficult.
Takeaways & Limitations
Effective tool use requires models to understand users, not merely execute tools, including compositional instructions, hidden intent, and instruction transitions.
Takeaways & Limitations
Human annotation improves quality and alignment with real user behavior but limits dataset scaling and feasible task length.
Abstract
from arXiv · showhide
Fulfilling user needs through Large Language Model multi-turn, multi-step tool-use is rarely a straightforward process. Real user interactions are inherently wild, being intricate, messy, and flexible. We identify three key challenges from user behaviour: compositional tasks that demand efficient orchestration of tool-call topologies, implicit intent spread across dialogue turns that require contextual inference, and instruction transition, which mixes task queries, clarifications, and casual conversation, forcing LLMs to adjust their policies on the fly. Existing benchmarks overlook these behaviors, making the apparent progress of LLMs on tool-use spurious. To address this, we introduce WildToolBench, an LLM tool-use benchmark grounded in real-world user behavior patterns. Comprehensive evaluations of 57 LLMs reveal that no model achieves an accuracy of more than 15%, indicating a substantial gap in the robustness of LLMs' agentic ability. Controlled experiments and in-depth analyses further indicate that the real challenge for LLM tool-use lies not in artificially complex tasks, but in the wild nature of user behavior, emphasizing the need to reconsider the interactions among LLMs, users, and tools.
1 Introduction
WildToolBench targets the mismatch between idealized tool-use benchmarks and real user behavior, which combines compositional tasks, contextual intent, and shifting instruction types. Across 57 LLMs, most models achieve no more than 15% session accuracy, showing that realistic interactions remain difficult.
- Motivation: Existing benchmarks overlook the complexity of real-world multi-turn, multi-step tool-use interactions.Real user logs motivate a benchmark grounded in how people actually employ LLMs with tools.
- Three challenges: Users combine multiple simple requirements, requiring tool orchestration beyond simple chaining.These compositional tasks may demand efficient coordination of several tool calls.
- Three challenges: Implicit intentions spread across dialogue turns, requiring LLMs to infer context.The relevant intent may not be fully specified in any single user message.
- Three challenges: Users transition among task-giving, follow-up, explanation, and casual-chat instructions, requiring policy adaptation.The assistant must adjust its response strategy as instruction types change.
- Results: 15% session accuracy is the upper bound reached by most models evaluated on WildToolBench.The benchmark contains 256 scenarios with 1,024 tasks and remains challenging where prior benchmarks tend toward saturation.
2 Related Work
Prior tool-use benchmarks increasingly model multi-turn and multi-step interaction, but important approaches still simplify or simulate user behavior. WildToolBench addresses this gap through human-in-the-loop incorporation of real user behaviors.
- Existing benchmarks: Earlier benchmarks assess tool-use sub-capabilities but can treat invocation as question answering rather than interactive agent behavior.This misses aspects of the multi-turn agent loop.
- User modeling: LLM-based user simulators approximate real environments but diverge from real users through unrealistically flawless behavior and unstable evaluation.WildToolBench instead incorporates three real user behaviors through human-in-the-loop annotation.
- WildToolBench: WildToolBench focuses on compositional tasks, hidden intent, and instruction transitions as real-user behaviors absent from simplified evaluations.These behaviors are used to make benchmark interactions more realistic.
3 WildToolBench
WildToolBench formalizes realistic dialogue as mixed user tasks requiring different response policies, then constructs and evaluates scenarios from real-user behavior patterns. Its curation combines sampled behaviors, verified APIs, generated trajectories, and manual ground-truth annotation.
- 3.1 Formulation: A dialogue contains user messages, assistant responses, scattered tasks, and possible multi-step tool interactions with an external environment.The formulation represents tool calls and environment feedback as part of the interaction.
- 3.1 Formulation: Different task types require no tool use, single-tool invocation, or multi-step tool invocation.WildToolBench models clarification, chat, simple tasks, and hard tasks as requiring distinct policies.
- 3.2 Data Curation: The pipeline analyzes real user logs and samples behavior patterns for scenario construction without leaking real user data.The three identified challenges guide the prompts and scenario distribution.
- 3.2 Data Curation: The benchmark uses 400 tool lists covering around 1,600 APIs, selecting a corresponding subset for each seed scenario.The APIs were publicly collected, verified, and cleaned before task generation.
- 3.2 Data Curation: Generated multi-agent trajectories are manually examined and annotated at each tool invocation to produce ground truth.Manual annotation and validation support accuracy and diversity throughout the curation process.
- 3.3 Challenge 1: Tool Orchestration for Compositional Task: Compositional tasks combine simple requirements in realistic scenarios, requiring legal tool execution paths rather than a single fixed chain.Human-labeled dependencies support enumeration of all legal paths through depth-first topological sorting.
- 3.4–3.5 Hidden Intention and Instruction Transition: Users distribute intent across turns and shift among task, follow-up, explanation, and casual-chat modes, requiring contextual policy changes.The benchmark includes partial information, interruptions, and multiple instruction types to test these behaviors.
- 3.5 Instruction Transition: WildToolBench contains four task types: single-tool, multi-tool, conversational or tool-free, and clarification tasks.Their proportions and switching frequency are curated to reflect instruction transitions.
4 Experiments
Experiments across 57 LLMs show that realistic multi-turn tool-use remains difficult, especially when tasks require compositional orchestration, hidden-intent inference, or frequent instruction transitions. Errors increasingly reflect semantic reasoning and long-range planning rather than basic parameter syntax.
- Experimental setup: 57 LLMs were evaluated across proprietary, open-source, general, specialized, instruction-tuned, and reasoning models.Models used their native function-call formats, with full results reported for all 57 models.
- Overall performance: No mainstream model exceeded 15% session accuracy, while most models remained below 60% task accuracy.Proprietary models generally outperformed open-source models, and reasoning models generally outperformed non-reasoning models.
- Tool orchestration: 43.75% was the highest task accuracy for tool orchestration, falling to 25% for mixed sequential-and-parallel gS+P_multi tasks.The peak optimal-path rate was 42.74%, indicating difficulty executing efficient tool-call structures.
- Hidden intention: Long-range dependency tasks stayed below 50% accuracy, with scores clustered between 30 and 45 and a 17.3-point performance gap.Reasoning models were stronger at omitted-information and hidden-intent inference, but no single model led across all task types.
- Instruction transitions: Task accuracy decreased as instruction transitions increased, with drops reaching 30% in some cases.The benchmark includes single-tool, multi-step, chat, and clarification tasks, and defines a transition as a task-type change between consecutive dialogue tasks.
- Error analysis: Error analysis indicates that higher-order planning and reasoning are more problematic than parameter-level syntax, with redundant calls revealing weak long-range planning.Models show divergent cautious and eager failure profiles, trading refusals against incorrect actions.
5 Conclusion
WildToolBench frames realistic tool use as a combination of tool execution and user understanding. Its findings identify compositional instructions, hidden intent, and instruction transitions as central challenges and provide rubrics for interpreting user behavior.
- Conclusion: WildToolBench identifies compositional instructions, hidden intent, and instruction transitions as three major challenges in realistic multi-turn, multi-step tool use.The benchmark emphasizes realistic user scenarios rather than only increasing tool-call procedure complexity.
- Conclusion: Effective tool use requires models to understand users rather than merely execute tools.The paper connects this capability to instruction following, long-context comprehension, and theory of mind.
- Conclusion: Structured rubrics support model developers in interpreting user behaviors from multiple perspectives and facilitating model iteration.This extends WildToolBench beyond a leaderboard.
6 Reproducibility statement
The authors provide materials intended to support full reproduction of the WildToolBench evaluation.
- Reproducibility statement: WildToolBench provides the datasets, evaluation scripts, and trajectories for all 57 evaluated LLMs.The materials are available in the submitted Supplementary Material zip file.
7 Limitations
WildToolBench prioritizes data quality, diversity, and alignment with real user behavior through human annotation, but this constrains dataset scaling and task length.
- Human annotation supports data quality, diversity, and alignment with real user behavior, but limits the potential scale of the dataset.
- Maintaining data quality while covering all policy transition types constrains the feasible length of tasks.
- The authors are combining human-annotated rubrics with an automated synthetic-environment scaling pipeline for future training and evaluation.
B Benchmark Comparison
The benchmark comparison indicates that prior LLM tool-use benchmarks tend toward saturation, whereas WildToolBench remains challenging.
- Prior LLM tool-use benchmarks tend toward saturation, while WildToolBench remains challenging.
C Data Curation
WildToolBench curates scenarios from real user logs, generates diverse tasks and tool-use trajectories, and applies repeated expert review to improve realism and data quality.
- Real user logs provide seed scenarios and behavioral patterns, while sampled examples help preserve their distribution without leaking user data.
- Controlled generation creates four task types across varied sentence structure, style, background, length, and difficulty dimensions.
- Assistant trajectories are automatically checked for tool-call errors, then reviewed by five experts who correct trajectories and annotate tool-call dependencies.
- Experts rewrite overly organized synthetic dialogue to better resemble natural user utterances and enrich policy-switch types.
- After four review iterations, reported data quality increased from 62%, 78%, 86%, and 94% to a final 100%.Nine human experts completed the curation process in one month.
D Details of Tool Orchestration Evaluation
The tool-orchestration evaluation enumerates serial and parallel execution paths in a DAG, using depth-first search with backtracking and configurable path-length stopping conditions.
- Algorithm 1 enumerates possible serial and parallel tool-execution paths in a directed acyclic graph.
- The procedure selects zero-indegree nodes, generates all non-empty subsets to model parallel execution, updates successor indegrees, and recurses.
- Paths are added when all graph nodes are visited or the annotated length limit is reached.
- Inference uses official default hyperparameters for proprietary models and default settings with max_new_tokens set to 512 for open-source models without official APIs.
E.2 WildToolBench Full Results
The appendix provides full WildToolBench results for 57 models, including tool-orchestration and error analyses, alongside task-generation prompts. These prompts cover single and multi-tool calls, clarification, chat, and context-dependent tasks.
- Full Results: 57 models are evaluated across proprietary general, open-source general, and open-source tool-use-specialized categories.The full results are reported in Table 6, with 16 proprietary general, 30 open-source general, and 11 open-source specialized models.
- Full Results: Tool-orchestration results and full error distributions are reported separately for the same 57-model evaluation.Table 7 covers tool orchestration, while Table 8 covers error distribution analysis.
- Task Generation: Task-generation prompts cover single-tool, sequential multi-tool, parallel multi-tool, and mixed multi-tool calls.The prompts instruct a simulated user to propose tasks for an agent with access to external tools, including tasks requiring combined tool use.
- Task Generation: Additional prompts generate clarification, chat, and context tasks involving dialogue-dependent user interactions.Context-task generation includes partial-information and coreferential-reference task types based on previous conversations.
- Error Analysis: The appendix also presents typical error examples discussed in the main text.These examples are collected in Figure 17.