Source-linked AI summary
ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities
Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, Ruoming Pang
TL;DR
Tool-use benchmarks need to evaluate stateful execution, conversational interaction, and flexible trajectories rather than only stateless APIs or fixed dialogs. ToolSandbox provides a stateful, conversational, interactive benchmark with simulated users and milestone-based evaluation, revealing performance gaps between model types and persistent challenges in complex task categories. Its milestone and minefield authoring limits scalability.
Problem
Existing tool-use benchmarks do not provide an all-encompassing evaluation of stateful, conversational, interactive tasks, often relying on stateless tools, single-turn queries, or predefined trajectories.
Method
ToolSandbox combines stateful tools with implicit dependencies, an LLM-simulated user for on-policy conversations, dynamic trajectories, and human-authored milestones and minefields for intermediate and final evaluation.
Results
ToolSandbox reveals a significant performance gap between open-source and proprietary models, while State Dependency, Canonicalization, and Insufficient Information challenge even capable SOTA models.
Takeaways & Limitations
The benchmark provides new insights into LLM tool-use capabilities by evaluating flexible, state-dependent, conversational behavior across complex scenarios.
Takeaways & Limitations
Authoring milestones and especially mandatory intermediate milestones requires deep tool knowledge and many iterations, hindering scalability.
Abstract
from arXiv · showhide
Recent large language models (LLMs) advancements sparked a growing research interest in tool assisted LLMs solving real-world challenges, which calls for comprehensive evaluation of tool-use capabilities. While previous works focused on either evaluating over stateless web services (RESTful API), based on a single turn user prompt, or an off-policy dialog trajectory, ToolSandbox includes stateful tool execution, implicit state dependencies between tools, a built-in user simulator supporting on-policy conversational evaluation and a dynamic evaluation strategy for intermediate and final milestones over an arbitrary trajectory. We show that open source and proprietary models have a significant performance gap, and complex tasks like State Dependency, Canonicalization and Insufficient Information defined in ToolSandbox are challenging even the most capable SOTA LLMs, providing brand-new insights into tool-use LLM capabilities. ToolSandbox evaluation framework is released at https://github.com/apple/ToolSandbox
1 Introduction
Tool-use evaluation remains incomplete because existing benchmarks often use stateless tools, single-turn prompts, fixed trajectories, or final LLM-judged outcomes. ToolSandbox addresses this gap with stateful tools, conversational on-policy interaction, and dynamic milestone-based evaluation.
- Existing benchmarks cover only parts of this challenge, relying on stateless tools, single-turn queries, predefined trajectories, or fixed action sequences.ToolEval supports multiple interaction rounds but relies solely on an LLM evaluator for final pass and win rates.
- State dependencies require agents to track mutable world states because some tools alter state while others cannot operate until prerequisites are satisfied.For example, searching for a nearby restaurant may require an enabled internet connection, even when the user does not mention that dependency.
- Interactive real-world scenarios require evaluation of tool errors, execution exceptions, user corrections, and key intermediate interactions across multi-turn sessions.
- ToolSandbox includes implicit state dependencies between stateful tools, enabling agents to track and alter world state using information implicit in user requests.
- An LLM-simulated user enables realistic on-policy conversational evaluation of implicit dialog-state tracking.
- Dynamic trajectories support multiple valid tool sequences and are evaluated through human-authored milestones and minefields at intermediate and final stages.
2 TOOLSANDBOX Design
TOOLSANDBOX combines stateful tools, simulated on-policy conversation, and flexible milestone/minefield evaluation for dynamic tool-use trajectories.
- Core environment: TOOLSANDBOX models User, Agent, and Execution Environment interactions across multiple rounds through a message bus.The Agent can request information from the User or invoke tools through the Execution Environment.
- Stateful tools: Stateful tools inspect, depend on, or manipulate world states, creating implicit dependencies that agents must track and resolve.For example, a failed send_message call caused by disabled cellular service should prompt enabling the service before retrying.
- Conversational evaluation: On-policy conversational evaluation uses a GPT-4o user simulator with Knowledge Boundary and Demonstration prompt components.These components address hallucination and instruction-following failures that arise when the simulator receives only the user goal.
- Interactive evaluation: Milestones and Minefields evaluate dynamic trajectories through required and prohibited events, respectively.Milestones accommodate different tool choices, orderings, and trial-and-error paths, while minefield violations force the whole-trajectory score to 0.
- Interactive evaluation: Milestone matching finds the highest averaged similarity across turn-to-milestone mappings whose chronological order is a topological sort of the milestone DAG.The framework combines explainable tool-call and execution-result matching with flexible evaluation of possible trajectories.
3 Test Scenarios
TOOLSANDBOX organizes 1,032 expert-crafted scenarios around composable tools and challenging conditions requiring state tracking, canonicalization, clarification, and resistance to hallucinated actions.
- Scenario construction: TOOLSANDBOX contains 1,032 test scenarios created by two domain experts with human-authored milestones and minefields.One annotator creates scenarios, while another validates the evaluation criteria.
- Scenario construction: The benchmark provides 34 representative, diverse, and composable tools spanning 11 domains.Tools use Python-native implementations when possible and selected RapidAPI endpoints when necessary.
- Scenario categories: State Dependency scenarios require successful tool execution to follow world-state changes, including nested dependencies and implicit call-stack tracking.For example, sending a message may require cellular service, which may itself require low battery mode to be disabled.
- Scenario categories: Canonicalization scenarios test transforming natural-language surface forms into canonical arguments, sometimes using tools for dates, locations, or other external knowledge.Examples include converting 1B to 1_000_000_000 and resolving “this Friday” or the Golden Gate Bridge to canonical values.
- Scenario categories: Insufficient Information scenarios withhold a necessary tool and use minefields to test whether agents recognize impossibility instead of hallucinating tools or arguments.This is more fine-grained than treating every unsolvable task as fully correct.
- Scenario coverage: 85% of scenarios include Multiple Tool Call, Multiple User Turn, State Dependency, or Insufficient Information challenges.Tool augmentations additionally vary distraction tools, informativeness of names, argument descriptions, and type hints.
4 Evaluation Results
TOOLSANDBOX evaluates models with a minimalist prompt across scenario categories and tool-schema augmentations, revealing substantial differences between model families and persistent challenges in state dependency, canonicalization, and insufficient information.
- Overall comparison: The best open-source model, Hermes, trails the second-lowest proprietary model, Claude-3-Haiku, by more than 20 points.The paper attributes part of this gap to some models’ inability to consume tool responses.
- Overall comparison: GPT-4o achieves the highest proprietary-model similarity score, while Claude-3-Opus uses fewer turns to reach the user goal.Multiple Tool Call and Multiple User Turn performance deteriorates faster than their single-step counterparts across model sizes.
- Scenario results: Larger models such as GPT-4 and Claude-3-Opus perform worse than smaller or mid-sized models on State Dependency because of erroneous parallel calls to dependent tools.The execution environment surfaces race conditions, exposing these parallel-call errors.
- Scenario results: Canonicalization remains difficult, with larger models favoring memorized world knowledge and smaller models more often using tools.Time-related arguments are especially challenging: models hallucinate timestamps and incorrectly resolve relative dates and times.
- Scenario results: Models can make premature choices when tool responses contain multiple location entities instead of requesting user disambiguation.This behavior produces canonicalization errors.
- Scenario results: Insufficient Information performance can expose hallucinated tool names or arguments even in simple tasks, and stronger complex-task performance correlates with worse performance in this category.The category tests whether models recognize when required capabilities are unavailable.
- Tool augmentations: Tool-augmentation robustness varies by model: Claude-3-Sonnet loses almost 10 points with extensive distraction tools, while other models are sensitive to scrambled descriptions or types.GPT-4o is especially susceptible to Tool Description Scrambling; GPT-4 relies heavily on argument descriptions; Gemini-1.5 struggles with Argument Type Scrambling.
5 Related Work
Related work spans benchmarks for function calling, web and embodied interaction, tool-using agents, dialogue state tracking, and simulated-user evaluation.
- Tool-use benchmarks: Function-calling benchmarks such as BFCL, ToolBench, StableToolBench, NexusRaven V2, and API-BLEND assess planning and execution of function calls.Other benchmarks evaluate web, browser, operating-system, or embodied interaction settings.
- Tool-use agents: Tool-use agents including Toolformer, Gorilla, ToolLLM, and CodeACT target increasingly broad or complex real-world tool-use scenarios.ToolLLM supports over 16,000 real-world APIs through generated instructional data and neural API retrieval.
- Dialogue state tracking: Dialogue State Tracking research requires agents to maintain and update dialogue states and actions across domains.MultiWOZ provides multi-domain dialogues, while schema-guided DST addresses scalability and adaptability.
- User simulation: User-simulator studies support automated conversational evaluation when multi-turn dialogues make off-policy evaluation insufficient and human evaluation costly.TOOLSANDBOX is situated within this line of work on simulated users.
6 Conclusion
TOOLSANDBOX is a stateful, conversational, interactive benchmark for LLM tool-use capabilities. It reports gaps between open-source and proprietary models and identifies difficult scenarios that challenge even leading models.
- Conclusion: TOOLSANDBOX combines stateful tools, implicit dependencies, an LLM-simulated user, and milestone-and-minefield evaluation.The benchmark is intended to provide additional insight into tool-use capabilities.
- Conclusion: The benchmark reveals a significant performance gap between open-source and proprietary models.It also identifies State Dependency, Canonicalization, and Insufficient Information as challenging categories for SOTA models.
7 Limitations
TOOLSANDBOX remains constrained by authoring scalability, simulated-user reliability, uncovered confirmation and authentication, unsupported daemon interruptions, and external-service dependencies.
- Annotation scalability: Authoring mandatory intermediate milestones requires deep tool knowledge and many iterations, hindering benchmark scalability.The paper suggests simplified or automatic milestone and minefield identification to scale data volume.
- User simulation: The user simulator still exhibits non-negligible hallucination and instruction-following errors despite behavioral controls.Providing the simulator with an end_conversation tool noticeably improved instruction following for dialog termination.
- Uncovered interaction requirements: Mandatory confirmation and authentication are not addressed, leaving confirmation decisions to the tool-use model.The paper points to orchestration-level enforcement as a possible direction.
- Execution scope: Tools that spawn daemon processes, such as timers, are not supported because future interruptions create challenges for orchestration and the model.These tools return after spawning a daemon that later interrupts the main process.
- Execution scope: Some external-knowledge tools, such as weather search, remain backed by external web services.
A.7 Evaluation
The evaluation matches trajectory snapshots to milestone DAGs using flexible similarity functions while preserving milestone ordering, exposing both intermediate progress and final outcomes.
- Milestone matching: Evaluation searches for the highest-average-similarity mapping between trajectory snapshots and milestone nodes subject to a valid topological order.The milestone DAG contains nodes and edges, while the trajectory is represented as database snapshots across conversation turns.
- Milestone matching: The resulting maximum similarity value is used as the trajectory’s similarity score against the milestone DAG.
- Similarity design: Column-wise similarity functions can use exact matching, ROUGE-L, AST matching, and other measures to compare snapshots with milestone targets.The flexible functions produce values in the [0, 1] range.
- Similarity design: Geometric aggregation makes any violated mandatory column constraint reduce the overall similarity to zero.
- Similarity design: Reference milestones enable guardrail checks and tool-trace-dependent similarities conditioned on earlier milestone states.
- Interpretation: Intermediate milestones showed GPT-4o could solve state-dependency challenges despite failing to finish within the maximum turns.The example indicates that improving turn efficiency is the relevant next step for that test case.
B.1 Tool-use Benchmark Comparisons
TOOLSANDBOX compares tool-use behavior through diverse composable tools, branched conversational scenarios, and trajectory statistics, while illustrating errors in state tracking, time reasoning, disambiguation, and task framing.
- Benchmark comparisons: TOOLSANDBOX statistics are calculated from GPT-4o trajectories, while comparison benchmarks use their specified datasets or trajectories and differing turn conventions.BFCL treats a single-prompt tool-call evaluation as two turns.
- Observed trajectories: GPT-4o spent most of an example trajectory resolving state dependencies but exceeded the maximum turns before completing the task.Intermediate milestones still exposed the reason for the final failure.
- Benchmark design: Scenarios branch from simple seed requests into multiple-tool, multiple-user-turn, state-dependency, insufficient-information, and combined cases.Alternative phrasings reuse milestone definitions, reducing annotation workload.
- Benchmark design: TOOLSANDBOX defines 34 tools across 11 domains, balancing representative coverage and manageable milestone annotation.Domains include contacts, messaging, reminders, system settings, time, maps, weather, stocks, conversion, math, and holidays.
- Evaluation coverage: Milestones track world-state databases and tool traces, with state-modification calls primarily evaluated through database milestones.A scenario can define multiple milestones and multiple database or tool-trace constraints.
- Observed trajectories: Example trajectories show errors involving code-generation framing, relative-time inference, hallucinated time, location disambiguation, and inefficient nested state-dependency handling.GPT-4o selected an unannounced first location among multiple entities, while GPT-4 incorrectly shifted the timestamp instead of deducing the requested time.
C.5 Insufficient Information
The insufficient-information category tests whether an agent recognizes that a task cannot be completed when a necessary capability or fact is unavailable.
- With only search_contacts available, GPT-3.5 hallucinated a remove_contact tool.
D.1 Prompting Experiment
The experiment compares default minimal prompting with ReAct prompting across four representative models. ReAct has minimal effect on Claude models and produces a small gain for GPT-4 models.
- The experiment compares ReAct prompting with the minimal prompting design across four representative models.The comparison is reported in Table 9.
- ReAct has minimal effect on the Claude model family and provides a small gain for the GPT-4 model family.
- The baseline setup may already provide a reasoning process through interactions with the execution environment, error recovery, and simulated user confirmation.Under this explanation, additional natural-language reasoning contributes only marginal gains.
D.2 Model Feature Comparison
The feature comparison documents whether models support the capabilities required for conversational, interactive tool use. Some open-source models cannot consume tool responses, limiting them to simpler single-tool-call scenarios.
- Table 10 compares model support for features required in a conversational, interactive tool-use workflow.Command R models were tested using Hugging Face weights, which reportedly lack a prompt template for consuming tool responses.
- A complete workflow requires multiple user-input rounds, choosing between tool calls and user responses, and consuming tool responses to determine the next step.
- Gorilla and Command-R cannot consume tool responses, so they may produce reasonable results on Single Tool Call scenarios but cannot complete scenarios requiring further interaction.