PaperlayerView exact revision on arXiv

Source-linked AI summary

ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities

Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, Ruoming Pang

arXiv:2408.04682v2cs.CLcs.AIcs.LG

TL;DR

Existing benchmarks lack comprehensive coverage of stateful, conversational, and interactive tool use. ToolSandbox evaluates these capabilities using state-dependent tools, a simulated user, and milestone-and-minefield evaluation. Proprietary models outperform open-source models, yet even state-of-the-art models struggle with State Dependency, Canonicalization, and Insufficient Information.

  • Problem

    Existing benchmarks lack comprehensive coverage of the stateful, conversational, and interactive nature of task-oriented dialogue, which challenges systematic evaluation of tool-using LLMs.

    Source: §1 · p. 1

  • Method

    ToolSandbox combines state-dependent tools, an LLM-simulated user, and flexible milestone-and-minefield evaluation in a stateful, conversational, interactive benchmark.

    Source: §6 · p. 9

  • Results

    Proprietary models substantially outperform open-source models, while State Dependency, Canonicalization, and Insufficient Information challenge even state-of-the-art models.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Larger models can underperform smaller models on state-dependent tasks because they issue erroneous parallel calls to dependent tools.

    Source: §4 · p. 7

    Authoring milestones and minefields requires deep knowledge of tool capabilities and repeated iteration, limiting benchmark scalability.

    Source: §7 · p. 10

Abstract

from arXiv · show

Recent large language models (LLMs) advancements sparked a growing research interest in tool assisted LLMs solving real-world challenges, which calls for comprehensive evaluation of tool-use capabilities. While previous works focused on either evaluating over stateless web services (RESTful API), based on a single turn user prompt, or an off-policy dialog trajectory, ToolSandbox includes stateful tool execution, implicit state dependencies between tools, a built-in user simulator supporting on-policy conversational evaluation and a dynamic evaluation strategy for intermediate and final milestones over an arbitrary trajectory. We show that open source and proprietary models have a significant performance gap, and complex tasks like State Dependency, Canonicalization and Insufficient Information defined in ToolSandbox are challenging even the most capable SOTA LLMs, providing brand-new insights into tool-use LLM capabilities. ToolSandbox evaluation framework is released at https://github.com/apple/ToolSandbox

1 Introduction

ToolSandbox addresses stateful, conversational, and interactive tool-use evaluation through implicit tool dependencies, an LLM-simulated user, and dynamic milestone/minefield checks of intermediate and final execution.

Stateful

  • ToolSandbox models implicit dependencies between stateful tools, requiring agents to track and modify world state using knowledge not explicit in user requests.BFCL and ToolEval (Qin et al., 2024) use stateless RESTful tools. API-Bank (Li et al., 2023), ToolTalk (Farn and Shin, 2023), and τ-bench (Yao et al., 2024) modify world states without studying state-dependency effects.Source: §1 · pp. 1–3

Conversational

  • An LLM-simulated user enables realistic on-policy conversational evaluation of agents’ implicit dialog-state tracking.Conversational evaluation must account for user–policy interdependence and natural-language ambiguity. Existing benchmarks use self-contained, unambiguous single-turn queries or predefined off-policy dialogues, with the latter unable to assess agents following their own policies.Source: §1 · pp. 2–3

Interactive

  • ToolSandbox combines dynamic interactive trajectories, highly composable tools, and human-authored milestone/minefield checks for intermediate and final execution evaluation.Interactive evaluation must accommodate erroneous calls, execution exceptions, and user corrections throughout multi-turn sessions. Existing approaches constrain trajectories or action sequences, limiting error correction and alternative solutions, or rely solely on LLM judgments of final pass and win rates, raising reliability and interpretability concerns.Source: §1 · pp. 2–3

2 TOOLSANDBOX Design

ToolSandbox evaluates stateful tool use through conversations among a user simulator, an agent, and an execution environment. Its Milestones and Minefields accommodate varied trajectories while checking required events and detecting prohibited actions.

2 TOOLSANDBOX Design

  • ToolSandbox connects User, Agent, and Execution Environment through a message bus, letting agents request clarification or invoke Python-function tools that can modify world state.Execution Context stores world state, and tools execute in code.InteractiveConsole. The addressed role speaks next, and the user ends the conversation through end_conversation.Source: §2 · p. 3, Figure 2 · p. 3

2.1 Stateful

  • 44% of ToolSandbox’s toolbox is stateful, creating implicit dependencies that test agents’ tracking of world state and tool-call stacks.Messaging, RapidAPI, and current-location tools depend on cellular service, wifi, and location service respectively. Enabling those services requires low battery mode to be off, creating nested dependencies.Source: §2.1 · p. 3

2.2 Conversational

  • A GPT-4o user simulator supports on-policy conversations using Knowledge Boundary prompts with partial expected results and simulator-only few-shot Demonstrations.Goal-only prompts can produce hallucination or agent-induced instruction-following failures; the simulator can terminate completed or uncompletable tasks. Combining both prompting components produces the lowest error rate in every category, with error rates consistent across agents.Source: §2.2 · p. 4, Table 2 · p. 4, Table 3 · p. 4

2.3 Interactive

  • Milestones define required events and score trajectories by maximizing average turn-to-milestone similarity over mappings that respect a temporal-dependency DAG.Similarity ranges from 0 to 1, combining explainable tool-call AST matching and execution-result exact matching with flexibility across different trajectories. Efficiency is measured separately through turn counts.Source: §2.3 · pp. 4–5
  • Minefields define prohibited events, testing whether agents recognize tasks they cannot complete rather than hallucinating; any nonzero minefield similarity makes the entire trajectory score zero.Minefields otherwise use the same evaluation machinery as Milestones.Source: §2.3 · p. 5

3 Test Scenarios

ToolSandbox uses expert-authored scenarios to test tool-call complexity, conversational clarification, state dependencies, canonicalization, and recognition of insufficient information. Tool-schema augmentations additionally support ablations of representation effects on agent accuracy.

Single / Multiple Tool Call

  • Single- and multiple-tool-call categories distinguish whether fulfilling the user’s task requires one tool call or several.Multiple Tool Call is among the categories identified as requiring complex reasoning.Source: §3 · pp. 5–6

Single / Multiple User Turn

  • Multiple-user-turn scenarios require clarification of ambiguous or incomplete requests and belong to ToolSandbox’s complex-reasoning challenges.Single-user-turn scenarios instead provide all necessary information in the initial user message.Source: §3 · p. 6

State Dependency

  • State-dependent scenarios require agents to discover implicit tool dependencies through trial and error, track nested prerequisites, and backtrack when necessary.For example, sending a message requires cellular service, but enabling cellular service first requires disabling low battery mode.Source: §3 · p. 6

Canonicalization

  • Canonicalization scenarios test conversion of natural-language surface forms into API-required canonical arguments, either directly by the model or with tool assistance.Examples include converting 1B to 1_000_000_000 directly, resolving relative dates using the current date, and looking up geographic coordinates.Source: §3 · p. 6

Insufficient Information

  • Insufficient-information scenarios deliberately withhold a necessary tool, testing whether agents recognize impossibility instead of hallucinating tools or arguments.Minefields detect tool calls implying hallucination. Unlike BFCL’s often irrelevant tools and ToolEval’s full credit for tasks deemed unsolvable, these scenarios require identifying missing capabilities among relevant tools and evaluate hallucination.Source: §3 · p. 6

Tool Augmentation

  • Tool augmentations enable accuracy ablations by adding distraction tools, obscuring tool or argument names, and removing argument descriptions or type hints.These augmentations are orthogonal to the scenario categories.Source: §3 · p. 6

3 Test Scenarios

  • 85% of ToolSandbox’s 1032 scenarios involve multiple tool calls, multiple user turns, state dependency, or insufficient information.Two internal experts create and validate scenarios defined by initial world states, messages, available tools, and calibrated milestones and minefields. Its 34 tools span 11 domains, with scenarios averaging more tool calls and dialog turns than other benchmarks.Source: §3 · pp. 5–6

4 Evaluation Results

Proprietary models outperform open-source models, with GPT-4o achieving the highest similarity score. State dependencies, canonicalization, insufficient information, and tool augmentations expose different weaknesses across models.

Open Source Models

  • More than 20 points separate best-performing open-source Hermes (interstellarninja et al.) from Claude-3-Haiku, the second-to-last proprietary model.Gorilla (Patil et al., 2023) and Command-R (Cohere and for AI, 2024) cannot consume tool responses, preventing multi-call solutions; Hermes and Mistral struggle to identify when tools are needed. These failures inflate Insufficient Information ratings by avoiding hallucinated calls, rather than demonstrating stronger reasoning.Source: §4 · pp. 6–7

Proprietary Models

  • GPT-4o achieves the highest similarity score, closely followed by Claude-3-Opus, which achieves user goals with fewer turns.Models receive the same minimalist prompt without additional prompt engineering. Comparing large and small models within GPT, Claude, and Gemini families (Reid et al., 2024), performance deteriorates faster on multiple-call and multiple-turn tasks than single-call and single-turn tasks.Source: §4 · pp. 6–7, Table 5 · p. 7

State Dependency

  • GPT-4 (Achiam et al., 2023) and Claude-3-Opus underperform GPT-3.5-Turbo and Claude-3-Sonnet on State Dependency because they issue erroneous parallel calls to dependent tools.The execution environment always surfaces race conditions when present. Nested dependencies also cause forgotten unresolved issues and inefficient backtracking, producing repeated errors and excessive turns.Source: §4 · p. 7

Canonicalization

  • Canonicalization challenges all models, particularly tool-assisted canonicalization and time arguments, with hallucinated timestamps, incorrect relative dates, and premature choices under ambiguity.Larger models tend to rely on memorized stable world knowledge, while smaller models more readily use tools. Models sometimes select the first returned location rather than asking the user to disambiguate.Source: §4 · p. 7

Insufficient Information

  • Insufficient Information performance negatively correlates with other categories: models stronger on complex tasks perform worse when information or tools are insufficient.Even on simple tasks with few tools, GPT-3.5-Turbo and GPT-4 can hallucinate tool names or arguments. Difficulty increases with task steps as models focus on immediate errors and forget the main objective.Source: §4 · p. 8

Tool Augmentations

  • Almost 10 points are lost by Claude-3-Sonnet when moving from zero distraction tools to making all ToolSandbox tools available.Augmentation robustness varies by model: GPT-4o is particularly susceptible to Tool Description Scrambling, GPT-4 pays extra attention to argument descriptions, and Gemini-1.5 struggles with Argument Type Scrambling.Source: §4 · p. 8

5 Related Work

Related work covers tool-use benchmarks, tool-using agents, dialogue state tracking, and user simulators. ToolSandbox complements explicit, off-policy state-tracking datasets with implicit world states and interactive online evaluation, while aligning with simulator-based evaluation approaches.

Tool-use Benchmarks

  • Existing benchmarks evaluate function calling, web-based task solving, and tool use within broader generalist-agent assessments.Examples include ToolBench (Qin et al., 2023b) for function calling, WebArena (Zhou et al., 2023) for web-based tasks, and AgentBench (Liu et al., 2023) for generalist-agent evaluation.Source: §5 · p. 8

Tool-use agent

  • Tool-use agents employ self-supervised learning, generated instruction–API pairs, API retrieval, and executable code actions.Toolformer (Schick et al., 2023) demonstrates autonomous tool learning through self-supervision, while Gorilla (Patil et al., 2023) uses self-instruct instruction–API pairs with and without retrieval. ToolLLM (Qin et al., 2024) combines generated instructional data and neural retrieval to use over 16,000 APIs and generalize to unseen APIs; CodeACT (Wang et al., 2024a) trains with executable code actions to improve decision-making and task solving.Source: §5 · p. 8

Dialogue State Tracking

  • ToolSandbox complements explicit, off-policy dialogue state tracking with implicit world states inferred from world knowledge and an interactive environment for online evaluation.MultiWOZ (Budzianowski et al., 2018) provides multi-domain dialogues requiring complex state tracking. Rastogi et al. (2020) introduce schema-guided tracking to improve scalability and adaptability while reducing domain-specific annotation needs.Source: §5 · p. 8

User Simulator in SandBox

  • Built-in user simulators support multi-turn evaluation when off-policy assessment may be insufficient and human-in-the-loop evaluation is costly and time-consuming.DAUS (Sekulic et al., 2024) uses LLMs fine-tuned on task-oriented dialogues, while MINT (Wang et al., 2024b) uses GPT-4 to simulate natural-language user feedback. AMIE (Tu et al., 2024) includes a patient model for symptom collection, and Zhang et al. (2024) use a yes-or-no simulator for identifying unknown entities; ToolSandbox aligns with these approaches.Source: §5 · p. 8

6 Conclusion

ToolSandbox evaluates LLM tool use through stateful, conversational, interactive tasks. It reveals a significant performance gap between open-source and proprietary models and scenarios that challenge even state-of-the-art models.

6 Conclusion

  • ToolSandbox reveals a significant performance gap between open-source and proprietary models.Source: §6 · p. 9
  • State Dependency, Canonicalization, and Insufficient Information challenge even state-of-the-art models.Source: §6 · p. 9
  • ToolSandbox combines stateful and state-dependent tools, an LLM-simulated user, and flexible evaluation using milestones and minefields.Source: §6 · p. 9

7 Limitations … A.6 Execution Environment Role

ToolSandbox’s limitations include labor-intensive evaluation authoring, simulator errors, reproducibility risks, and unsupported confirmation and asynchronous interruptions. Its implementation combines shared state, composable tools, schema augmentations, role-specific messaging, simulated users, and Python execution.

7 Limitations

  • ToolSandbox remains limited by costly metric authoring, simulator errors, external-service reproducibility, and unsupported mandatory confirmation, authentication, and daemon-triggered interruptions.Automating Milestone/Minefield creation could improve scalability; end_conversation improved termination instruction following, suggesting broader simulator tool support. Caching external-service responses could stabilize results, while enforcing confirmation and handling future daemon interruptions remain open orchestration problems.Source: §7 · p. 10

A Implementation Details / A.1 Execution Context

  • The Execution Context stores tool databases and dialog history, preserving per-turn snapshots for introspection and evaluation while preventing direct agent manipulation.Roles and tools access this global state, allowing stateful tools to operate on databases without receiving them as explicit function arguments.Source: §A.1 · p. 13

A.2 Tools

  • Composable Python tools manipulate shared state and expose dependency failures through informative exceptions, enabling agents to reason about corrective tool calls.Tools include native implementations and wrapped RapidAPI services; type hints and docstrings become JSON API schemas. Execution records each tool’s name, inputs, and outputs in the Execution Context for automatic evaluation.Source: §A.2 · p. 13

A.2.1 Tool Augmentations

  • 0, 3, 10, or all remaining tools are added as distractors to test tool selection, prioritizing domain overlap and textual similarity.Scrambling obscures tool names or removes tool summaries, argument descriptions, or type hints; every scrambling condition includes three distraction tools. Argument-type exceptions reveal expected types, preserving solvability, while missing argument descriptions permit discovery through trial and error.Source: §A.2.1 · pp. 13–14

A.3 Roles and Message Bus

  • User, Agent, and Execution Environment communicate through a shared Message Bus with role-specific visibility rather than an unrestricted conversation thread.Messages specify sender, recipient, content, and visibility; the orchestrator makes the most recent recipient the next sender. Each role reads only its permitted subview of the linear transaction history.Source: §A.3 · p. 14

A.4 User Role

  • User simulator prompts combine a User Goal, Knowledge Boundary, and private demonstrations to reduce hallucination and instruction-following errors.Calling the agent “User B” improves role adherence. Demonstrations are visible only to the simulator, and omitting Knowledge Boundary and Demonstration makes errors more frequent.Source: §A.4 · pp. 14–15

A.5 Agent Role

  • The Agent responds to natural-language requests by asking for clarification or selecting tools and arguments, with JSON calls converted into executable Python.Agent prompts are consistent across models and do not disclose testing-environment specifics.Source: §A.5 · p. 15

A.6 Execution Environment Role

  • The Execution Environment deliberately triggers detected race conditions in dependent parallel tool calls, preventing agents from bypassing required execution order.It executes Agent and User Python snippets like an interactive console. Exceptions captured through stderr support trial-and-error refinement; independent tool calls can appropriately run in parallel.Source: §A.6 · p. 16

A.7 Evaluation · B Test Scenarios

ToolSandbox evaluates trajectories by maximizing milestone–snapshot similarity under milestone-DAG ordering constraints, using configurable database comparisons and reference milestones. Intermediate milestones can reveal capabilities and turn-efficiency problems even when a task remains incomplete.

A.7 Evaluation

  • Trajectory evaluation maximizes average milestone–snapshot similarity over mappings consistent with a topological ordering of the milestone DAG.The maximum average similarity becomes the trajectory’s milestone-DAG score.Source: §A.7 · p. 16
  • Milestones specify column-wise similarity functions, including exact matching for cellular service, ROUGE-L F measure for message content, and AST matching for tool traces.Each function returns a value in [0, 1], allowing different comparison criteria for different columns.Source: §A.7 · p. 16
  • Database similarity combines column scores geometrically and finds the row assignment maximizing the geometric mean of row similarities.For snapshot and target databases containing k rows each, pairwise row scores support the assignment. A mandatory column constraint can return zero to nullify overall similarity.Source: §A.7 · p. 16
  • Reference milestones enable guardrail similarity checks that detect database changes between two milestone events.These comparisons condition similarity on an additional milestone rather than only the current milestone and snapshot.Source: §A.7 · p. 16
  • Tool-trace-dependent similarity tracks information flow by extracting tool output from a reference milestone and incorporating it into the current milestone.This permits evaluation constraints that depend on outputs produced earlier in the trajectory.Source: §A.7 · p. 16
  • An incomplete trajectory nevertheless demonstrated state-dependency handling and current-location requests, identifying turn efficiency as an improvement target.The task was not completed before the maximum allowed number of turns.Source: §A.7 · p. 16

B.1 Tool-use Benchmark Comparisons … C.4 State Dependency

ToolSandbox documents its benchmark-statistic conventions, scenario annotation process, tool design, and dataset statistics. Example trajectories illustrate interaction patterns, tool-call detection mistakes, canonicalization errors, and state-dependency failures.

B.1 Tool-use Benchmark Comparisons

  • Benchmark statistics count each user, agent, or tool message as one turn; ToolSandbox statistics use GPT-4o trajectories.BFCL is counted as two turns; ToolEval uses ToolLlama DFS Retriever trajectories, and API-BANK uses level 1 and 2 test sets.Source: §B.1 · pp. 16–17

B.2 Annotation Process

  • Annotation expands simple seed requests into combinable scenario branches and paraphrases, reusing milestones to reduce annotation work.Variants require missing-slot clarification or recognition that information is insufficient. Another annotator validates feasibility and milestones/minefields by acting as an agent, followed by at least four rounds of model-agent testing.Source: §B.2 · pp. 17–18

B.3 Tool Design

  • 34 tools across 11 domains balance representative diversity with well-defined capacities and manageable tool counts for milestone/minefield annotation.Each domain provides an omnisearch entry point; stateful domains expose manipulation tools, while utilities support canonicalization and calculations. Implementations use native Python where possible and selected RapidAPI endpoints when necessary.Source: §B.3 · p. 18

B.4 TOOLSANDBOX Statistics

  • Dataset statistics cover scenario categories, database-column milestones, and tool-level milestones, allowing multiple categories and milestone constraints per scenario.Most state-modification calls are tracked through corresponding database milestones rather than tool traces.Source: §B.4 · pp. 18–19, Table 6 · p. 19, Table 7 · p. 19

C.1 Tool Call Detection

  • Mistral’s example trajectory mistakes a tool-use task for a code-generation task.Source: Figure 11 · p. 19

C.2 Single/Multiple Tool Call/User Turn

C.3 Canonicalization

  • Canonicalization examples show GPT-4 miscalculating relative time and GPT-4o hallucinating time or choosing an ambiguous location without asking the user.The location failure selected the first returned entity without announcing the choice or disambiguating the intended Whole Foods location.Source: Figure 14 · p. 20, Figure 15 · p. 20, Figure 16 · p. 21

C.4 State Dependency

  • State-dependency examples expose GPT-4’s race-producing parallel calls and lost nested dependencies, requiring sequential resolution and a mental call stack for backtracking.GPT-4 parallelized Wi-Fi repair and holiday search despite their dependency, and separately requested location before enabling location services. Another example requires resolving a PermissionError before a ConnectionError.Source: Figure 17 · p. 21, Figure 18 · p. 21, Figure 19 · p. 22

C.5 Insufficient Information · D Additional Evaluation Results

GPT-3.5 failed an Insufficient Information example by hallucinating remove_contact when search_contacts was the only available tool.

C.5 Insufficient Information

  • GPT-3.5 hallucinated remove_contact despite having access only to search_contacts, illustrating a failure in the Insufficient Information category.Figure 20 presents example failure trajectories.Source: Figure 20 · p. 22

D.1 Prompting Experiment

Compared with minimal prompting, ReAct has minimal effects on Claude models and produces small gains for GPT4 models, without changing relative model rankings.

D.1 Prompting Experiment

  • 4 representative models were evaluated to compare ReAct prompting against the minimal prompting design presented in Figure 8.Source: §D.1 · p. 22
  • The authors suggest environment interaction, error recovery, and simulated-user confirmation already provide a reasoning process in ToolSandbox, limiting gains from additional natural-language reasoning.This is offered as a likely explanation rather than an experimentally isolated mechanism.Source: §D.1 · p. 22
  • ReAct prompting minimally affects Claude models and yields small gains for GPT4 models, while preserving relative model rankings.The authors interpret this as supporting their claim that prompt-engineering gains are orthogonal to the innate model capabilities surfaced by simpler prompting.Source: Table 9 · p. 22, §D.1 · p. 22

D.2 Model Feature Comparison · D.3 Turn Count Comparison

Model feature gaps limit conversational, interactive tool use, including Gorilla and Command-R’s inability to consume tool responses. Turn counts should be interpreted alongside task-success similarity scores rather than treated as standalone efficiency measures.

D.2 Model Feature Comparison

  • Gorilla and Command-R cannot consume tool responses, a required capability for conversational, interactive tool use.The comparison identifies feature limitations that affect interpretation of these models’ evaluation results.Source: Table 10 · p. 23, §D.2 · p. 23
  • Command R models were tested using Huggingface-released weights, which, to the authors’ knowledge, lacked a prompt template for consuming tool responses.Source: Table 10 · p. 23
  • The authors document missing workflow features, especially among open-source models, to contextualize experimental metrics and motivate future research.Source: §D.2 · p. 23
  • Models unable to consume tool responses may theoretically handle Single Tool Call scenarios but cannot complete scenarios requiring multiple tool calls.This limitation applies to Gorilla and Command-R in the reported comparison.Source: §D.2 · p. 23
  • Conversational tool-use agents must accept repeated user input, choose between tool calls and user responses, and consume tool outputs to determine subsequent actions.Source: §D.2 · p. 23

D.3 Turn Count Comparison

  • Turn counts should compare similarly similarity-scored models, because early completion can reflect confident errors rather than efficiency.Table 11 reports average turn counts across scenario categories and tool augmentations, including distraction tools and scrambled tool or argument metadata.Source: Table 11 · p. 24

Related papers

Loading 2408.04682v2…