PaperlayerView exact revision on arXiv

Source-linked AI summary

ToolFuzz -- Automated Agent Tool Testing

Ivan Milev, Mislav Balunović, Maximilian Baader, Martin Vechev

arXiv:2503.04479v3cs.AIcs.SE

TL;DR

Tool documentation can be incomplete or misaligned, yet automated testing methods for these natural-language specifications are lacking. TOOLFUZZ combines fuzzing with LLM-generated prompts and consistency checks to detect runtime and correctness failures. It reports more than 20× the erroneous prompts uncovered by prompt-engineering approaches.

  • Problem

    Tool documentation can be underspecified, overspecified, or misaligned, and automated methods for testing these natural-language specifications are lacking.

    Source: §1 · p. 1

  • Method

    TOOLFUZZ combines fuzzing and LLM-generated queries for runtime failures with synonymous prompts, consistency checks, and an LLM oracle for correctness failures.

    Source: §1 · p. 1

  • Results

    TOOLFUZZ uncovers more than 20× the number of erroneous prompts compared with prompt-engineering approaches.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The paper presents TOOLFUZZ as a method for systematically testing agent tools and identifies automated documentation refinement and multi-tool testing as future directions.

    Source: §6 · p. 8

Abstract

from arXiv · show

Large Language Model (LLM) Agents leverage the advanced reasoning capabilities of LLMs in real-world applications. To interface with an environment, these agents often rely on tools, such as web search or database APIs. As the agent provides the LLM with tool documentation along the user query, the completeness and correctness of this documentation is critical. However, tool documentation is often over-, under-, or ill-specified, impeding the agent's accuracy. Standard software testing approaches struggle to identify these errors as they are expressed in natural language. Thus, despite its importance, there currently exists no automated method to test the tool documentation for agents. To address this issue, we present ToolFuzz, the first method for automated testing of tool documentations. ToolFuzz is designed to discover two types of errors: (1) user queries leading to tool runtime errors and (2) user queries that lead to incorrect agent responses. ToolFuzz can generate a large and diverse set of natural inputs, effectively finding tool description errors at a low false positive rate. Further, we present two straightforward prompt-engineering approaches. We evaluate all three tool testing approaches on 32 common LangChain tools and 35 newly created custom tools and 2 novel benchmarks to further strengthen the assessment. We find that many publicly available tools suffer from underspecification. Specifically, we show that ToolFuzz identifies 20x more erroneous inputs compared to the prompt-engineering approaches, making it a key component for building reliable AI agents.

1. Introduction

Tool documentation can be underspecified, overspecified, or misaligned with tool functionality, while automated methods for finding such errors are lacking. TOOLFUZZ tests for runtime errors and incorrect agent responses, and the introduction presents new tool-use benchmarks.

1. Introduction

  • Effective tool interaction is critical because LLM agents use tools such as web search, code execution, and databases to perform real-world tasks.Agents perform sequences of actions with external tools to solve user queries.Source: §1 · p. 1
  • Tool documentation errors are often underspecified, overspecified, or misaligned with functionality, but automated detection methods are lacking.Traditional fuzzing with random strings failed to find realistic agent queries in a several-hour test.Source: §1 · p. 1
  • TOOLFUZZ combines LLM-generated natural-language queries with fuzzing and consistency checks to find runtime-error and incorrect-response cases.The paper describes this as an end-to-end method for automated tool testing.Source: §1 · pp. 1–2
  • The new file-management and GitHub benchmarks emphasize precise tool invocation over sophisticated reasoning and planning.The introduction also reports a broad evaluation of TOOLFUZZ across agent tools and tasks.Source: §1 · p. 2

2. Background and Related Work

Prior work studies LLMs and tool-using agents, but existing agent benchmarks emphasize reasoning and planning or function-call accuracy rather than tool documentation. TOOLFUZZ applies a hybrid fuzzing approach to probe agent-tool interactions.

2. Background and Related Work

  • LLMs remain susceptible to hallucinations despite strong benchmark performance, and prior work uses prompting, LLM judges, and cross-checking generations to improve verification (Manakul et al., 2023; Mündler et al., 2024).The paper also names GPT-4, GPT-4o, and GPT-4o-mini as the instruction-tuned models it uses (OpenAI et al., 2024; OpenAI, 2024).Source: §2 · p. 2
  • LLM agents use tools such as Web APIs, databases, and code execution, and this paper focuses mainly on failures from incomplete or erroneous documentation (Qu et al., 2024).The paper defines an agent system A as a language model L with access to tools F.Source: §2 · p. 2
  • Existing agent benchmarks cover diverse tasks and function-call accuracy, but primarily emphasize reasoning and planning rather than tool correctness and documentation (Liu et al., 2023; Debenedetti et al., 2024; Tang et al., 2023; Xu et al., 2023).TOOLFUZZ adds File Management and GitHub benchmarks focused on correct tool use rather than sophisticated planning.Source: §2 · p. 2
  • TOOLFUZZ combines fuzzing with machine learning to generate semantically meaningful tests using partial knowledge of tool semantics, rather than purely random strings (Jiang et al., 2024).The related-work discussion notes concerns about false positives in fuzzing and cost, scope, and reliability in fully LLM-driven fuzzing.Source: §2 · p. 2

3. Overview

Tool documentation errors can manifest as runtime or correctness failures in agent-tool use. TOOLFUZZ uses fuzzing to find runtime failures and invariance checks, supplemented by an LLM oracle, to identify correctness failures.

3. Overview

  • Documentation may be underspecified, overspecified, or misaligned with tool functionality, including missing constraints, restricted use cases, or inaccurate argument details.These issues can lead agents to misuse tools.Source: §3 · p. 3
  • Agent tool failures include runtime errors from invalid inputs and correctness errors in which the tool returns incorrect results for a query.The paper links these failures to mismatches between developer-written documentation and LLM interpretation.Source: §3 · p. 3

3.1. TOOLFUZZ

  • TOOLFUZZ uses fuzzing-inspired testing for runtime failures and invariance-based testing for correctness failures.The overview illustrates both techniques with an open-street-map-search tool that processes free-form location queries.Source: §3.1 · p. 3, Figure 1 · p. 3
  • For runtime detection, TaintFuzzer finds inputs that trigger tool errors, then an LLM uses the documentation and failing input to generate a natural-language query tested on the agent.The process monitors agent tool execution and collects queries that lead to runtime errors.Source: §3.1 · p. 3
  • For correctness detection, TOOLFUZZ generates synonymous prompts and checks whether they produce consistent tool calls and outputs.The example instantiates “Find [A] near [B]” with synonymous place and location terms.Source: §3.1 · pp. 3–4
  • An LLM oracle compares agent responses with a generated expected answer because consistency checks alone can miss outputs that are consistently wrong.The method combines the checks to reduce false positives and treats prompts as erroneous when all checks fail.Source: §3.1 · p. 4

4. Methodology

The methodology separates runtime-failure testing from correctness-failure testing. It uses taint-based input generation and documentation checks for runtime failures, and synonymous prompts with cascading consistency and plausibility checks for correctness failures.

4. Methodology

  • The methodology tests runtime failures by generating tool-breaking inputs and correctness failures by assessing agent outputs without ground-truth data.The paper introduces a two-stage runtime process and a prompt-set approach for correctness assessment.Source: §4 · p. 4, §4.1 · p. 5, §4.2 · p. 5

4.1. Runtime tool failures

  • TaintFuzzer generates syntax- and semantic-aware inputs, then documentation-based sanity checks and LLM prompt generation filter and convert failing inputs into agent queries.For the example tool, generated arguments exceeding 100 characters are collected as failures.Source: §4.1 · p. 5

4.2. Correctness failure detection

  • For correctness detection, synonymous prompt sets should produce equivalent tool calls and outputs, while an LLM plausibility check helps assess answers without ground truth.The paper generates templates and synonymous mask infills using the tool documentation.Source: §4.2 · p. 5

Correctness Detection

  • Input consistency compares argument values across calls, while output consistency compares tool responses by exact matching.Inconsistent inputs indicate underspecification of tool arguments; output consistency serves as a proxy for comparing natural-language agent responses.Source: Correctness Detection · p. 6
  • The LLM oracle addresses false negatives from consistently wrong agents and false positives from nondeterministic tools by comparing responses with a majority-vote answer.An oracle rates similarity from 1 to 10, with 5 as the threshold; prompts are considered faulty when all checks fail.Source: Correctness Detection · p. 6

5. Experimental Evaluation

The evaluation tests TOOLFUZZ on curated and custom tools, then assesses its error detection and documentation fixes on new tool-use benchmarks. TOOLFUZZ increases true positives while reducing false discovery, and documentation informed by its findings improves benchmark performance.

5.1. Experimental setup

  • The evaluation uses ReAct agents with GPT-4o or GPT-4o-mini, 32 LangChain Community tools, and 35 manually developed tools; positives are manually inspected for FDR.The new benchmarks use pass rate, defined as successfully completed tasks over total tasks; prompt-engineering baselines include gray-box and white-box approaches.Source: §5.1 · p. 6
  • The File Management Benchmark has 32 folder structures and 193 tasks, while the GitHub Benchmark has 54 tasks across six categories; both have training and unseen test sets.The benchmarks emphasize precise tool use based on documentation rather than complex planning.Source: §5.1 · p. 6

5.2. Evaluation on 32 LangChain Tools

  • On runtime failures, TOOLFUZZ finds more diverse errors, especially in external-API tools, using only 1/6 and 1/9 of the prompts used by other approaches.The authors report that its fuzzing process avoids degenerate prompts and uses the Faker vocabulary library.Source: §5.2 · p. 7, Table 1 · p. 7
  • 45% lower FDR and 20× more true positives than prompt-based methods show TOOLFUZZ’s stronger correctness-failure detection.The authors manually check whether agent responses match the queries; they attribute performance to diverse prompts, input-output consistency checks, and the LLM Oracle’s expected answers.Source: §5.2 · p. 7, Table 3 · p. 7

5.3. Evaluation on Two Custom Benchmarks

  • Documentation fixes based on TOOLFUZZ prompts improve File Management task performance by ≈10% for both the terminal tool and File Management Toolkit, and GitHub performance by 6%.For the GitHub toolkit, TOOLFUZZ-informed documentation improves results by 4% over uninformed improvements; gains are bounded where documentation is already well-crafted.Source: §5.3 · p. 8, Table 5 · p. 8

5.4. Case Study

  • A PubMed case shows that omitting field-specific date-query syntax led a 2020 search to return a 2024 paper; updated documentation enabled a correctly constrained query.The case study reports the correct output as a paper published in 2020.Source: §5.4 · p. 8

6. Conclusion

TOOLFUZZ is presented as the first method for testing tools in agent environments, identifying more than 20× the erroneous prompts. The conclusion points to multi-tool testing and automated documentation refinement as future directions.

6. Conclusion

  • More than 20× as many erroneous prompts were uncovered by TOOLFUZZ in the experiments.The authors describe this as evidence of its effectiveness.Source: §6 · p. 8
  • The authors identify testing multiple tools together and automating documentation refinement as future research directions.These directions target cross-coordination failures and tool descriptions.Source: §6 · p. 8

A. Source code

  • The source code is supplied in a zip file, with setup and run instructions in the README.The codebase uses Python 3.10 and a Conda environment, with dependencies listed in requirements.yml.Source: A. Source code · p. 14

B. Agent

  • The evaluated agent design uses an LLM for reasoning and planning plus tools, minimizing system overhead.The authors state that the work focuses on tool usage.Source: B. Agent · p. 14

C. Additional evaluation

Additional evaluation reports low false-positive rates across tool categories and finds unplanned tool use across many tools. The cross-tool experiment sampled prompts intended for different tool groups.

C.1. Cross tool calling

  • 492 prompts led to unplanned tool usage across 53 tools in the cross-tool calling experiment.The prompts were sampled for different tool groups to test unintended activations.Source: §C.1 · p. 14
  • The authors grouped tools by domain for cross-tool testing, including NASA media and HTTP-request tool sets.The listed NASA tools concern media metadata, search, and video captions; the HTTP group includes request methods and related tools.Source: §C.1 · p. 14, §1 · p. 14, §10 · p. 15

C. Additional evaluation

D. Case studies

The case studies illustrate how tool descriptions can leave important usage requirements implicit. In the OpenStreetMap example, requiring country names in location arguments makes the tested prompt set pass.

OpenStreetMap directions

  • OpenStreetMap prompts produced two output buckets: four prompts returned 939224.1 meters, while the “City of Paris”/“City of Lyon” prompt returned 465460.3 meters.The five prompts varied how Paris and Lyon were specified.Source: D. Case studies · p. 16
  • Requiring country names in location arguments fixes the OpenStreetMap prompt set; city-only names work only half the time.The revised description makes the example prompt set pass.Source: D. Case studies · p. 17

D. Case studies

  • The terminal description is short and generic, omitting how to format command parameters and which commands are usable.Figure 6 compares original and improved descriptions for open_street_map, pubmed, and terminal.Source: Figure 6 · p. 16, D. Case studies · p. 17

1. How big is the assignment_1_draft.docx file in the Contract Law assignments folder?

The assignment-file case study shows TOOLFUZZ finding path-related failures that were resolved by a more informative terminal description. The appendix also documents prompt-generation, checking, oracle, and baseline procedures.

1. How big is the assignment_1_draft.docx file in the Contract Law assignments folder?

  • The example task asks for the size of the first assignment draft, while the agent is unaware of its working directory.The path is masked in the appendix because it is part of the machine’s file structure.Source: §1 · p. 17
  • The initial prompts produced unique inputs and outputs, including invalid paths; after description revision, the set became valid using repeated ls commands to navigate.The terminal description did not tell the agent where it was running or how to navigate.Source: §1 · p. 17

E. TOOLFUZZ Prompts

E.2. TOOLFUZZ correctness detection

  • Correctness prompt generation creates a template from the tool description, fills it with synonymous phrases, and can humanize context-derived facts.The appendix points to separate prompts for template generation, synonymous infills, and humanizing.Source: §E.2 · p. 18, Figure 9 · p. 19

E.3. LLM Oracle

  • The LLM Oracle expectation is formed from answers to generated prompts, and its evaluation uses reasoning and a 1-to-10 score rather than yes/no.The authors report that yes/no evaluation produced more false positives.Source: §E.3 · p. 18

E.4. Baseline prompts

  • Baseline prompt examples include degenerate string generation, and the white-box baseline includes tool source code while the gray-box baseline does not.The appendix also describes a separate LLM judge for baseline evaluation.Source: §E.4 · p. 18

F.1. File Management toolkit benchmark

The file management benchmark spans 32 domain-specific environments and verifies agent task outcomes against changes to the initial file system. The evaluation also categorizes tested tools and describes three approaches to automatic description fixing.

F.1. File Management toolkit benchmark

  • The benchmark covers 32 domain-specific environments, with each task run in a Docker container initialized with a file system and a ReAct agent.Task success is verified by comparing initial and final states with git diff.Source: §F.1 · p. 19
  • Task success is verified using a git diff between the file system’s initial and final states.Source: §F.1 · p. 19
  • Automatic tool-description fixing uses GPT-4o with prompts based on failure examples, the description alone, or the description and source code.Source: §F.1 · p. 19

G. Tested Tools

  • The evaluated tools are divided into categories for runtime-failure detection and correctness detection.Source: G. Tested Tools · p. 19

G.1. Runtime failure tools

The runtime-failure evaluation includes synthetic and LangChain community tools. The collected LangChain tools most commonly use string arguments, and typically have two declared arguments but one practical argument.

G.1. Runtime failure tools

  • The synthetic runtime-failure tools include examples such as check-basketball-match-stats, package-items, and file_reader.Source: §G.1 · p. 19
  • LangChain community tools tested for runtime failures include arxiv, duckduckgo_search, file_search, and python_repl.Source: §G.1 · p. 20

G.2. Tools statistics

  • String is the most common argument type, while two is the most common declared argument count; the runtime manager means most tools practically accept one argument.Source: §G.2 · p. 20
Loading 2503.04479v3…