Source-linked AI summary
EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction
Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Yongliang Shen, Ren Kan, Dongsheng Li, Deqing Yang
TL;DR
LLM agents need to use diverse tools, but documentation is often inconsistent, redundant, incomplete, and difficult to fit within context limits. EASYTOOL converts documentation into concise, unified instructions with standardized descriptions, parameters, and demonstrations. Experiments show reduced incorrect tool usage and improved tool utilization across datasets and real-world scenarios.
Problem
Tool documentation’s inconsistent formats, redundancy, missing information, and context-length burden limit effective tool utilization by LLM-based agents.
Method
EASYTOOL reorganizes documentation into unified instructions, retaining core functions and adding parameter-focused guidelines with usage demonstrations.
Results
Experiments on three datasets from distinct domains show improved tool utilization and significantly fewer incorrect tool usages.
Takeaways & Limitations
Concise, standardized tool instructions provide a plug-and-play way to improve LLM tool utilization, including for open-source LLMs.
Takeaways & Limitations
EASYTOOL is limited to documentation within ChatGPT’s input limit, single-documentation settings, and models with instruction-following ability.
Abstract
from arXiv · showhide
To address intricate real-world tasks, there has been a rising interest in tool utilization in applications of large language models (LLMs). To develop LLM-based agents, it usually requires LLMs to understand many tool functions from different tool documentation. But these documentations could be diverse, redundant or incomplete, which immensely affects the capability of LLMs in using tools. To solve this, we introduce EASYTOOL, a framework transforming diverse and lengthy tool documentation into a unified and concise tool instruction for easier tool usage. EasyTool purifies essential information from extensive tool documentation of different sources, and elaborates a unified interface (i.e., tool instruction) to offer standardized tool descriptions and functionalities for LLM-based agents. Extensive experiments on multiple different tasks demonstrate that EasyTool can significantly reduce token consumption and improve the performance of tool utilization in real-world scenarios. Our code will be available at \url{https://github.com/microsoft/JARVIS/} in the future.
1 Introduction
LLM agents increasingly use external tools, but inconsistent, redundant, and incomplete documentation makes precise tool utilization difficult. EASYTOOL converts such documentation into concise, unified instructions, and experiments show improved tool use with fewer incorrect invocations.
- External tools help LLM agents address complex user requests, making precise tool utilization important for autonomous agents.
- Existing approaches either require additional tool-use training or struggle with context limits, unusual tools, and inflexible integration.
- Tool documentation suffers from inconsistent formats, redundant information, and missing functions, parameters, or demonstrations.
- EASYTOOL transforms diverse documentation into clear, structured, unified instructions that support tool selection and parameter prediction.
- Experiments on three datasets from distinct domains show that EASYTOOL improves tool-utilization capability and significantly reduces incorrect tool usage.
2 Related Work
Prior tool-use methods either fine-tune models with synthetic tool-use data or prompt LLMs with external documentation. Prompt-based approaches avoid extra training but remain constrained by context size and documentation quality.
- Fine-tuning methods teach open-source LLMs to generate tool calls but require additional training data and do not extend readily to powerful closed models.
- Prompt-based methods use LLMs as controllers and provide tool descriptions directly, enabling plug-and-play use without extra training.
- These prompt-based approaches remain limited by context size and the quality of tool documentation, which can produce failed or incorrect invocations.
3 Preliminary
LLM agents typically plan tasks, retrieve candidate tools, select tools and parameters, then execute them with bounded retries. Documentation problems affect this pipeline through inconsistent formats, redundancy, and incompleteness.
- The agent pipeline comprises task planning, tool retrieval, tool selection, and tool execution.
- Task Planning: Task planning decomposes a request into dependent subtasks with execution orders, each optimized for a single tool.
- Tool Retrieval: Tool retrieval matches subtasks with inventory tools and selects the top-K candidates by similarity.
- Tool Selection and Execution: Tool selection chooses among candidates and prepares execution parameters, while failed execution triggers alternative selection until success or R trials.
- Documentation Limitations: Inconsistent formats challenge understanding, redundant content increases documentation length, and missing usage context contributes to invalid parameters.
4 Method
EASYTOOL condenses tool documentation through reorganization and functional-guideline generation. It removes irrelevant information, preserves function descriptions, and adds parameters and usage examples for LLM instruction.
- EASYTOOL first reorganizes documentation by eliminating irrelevant information and retaining each tool’s function description.
- It then generates functional guidelines containing tool parameters and examples to instruct LLMs in tool usage.
I: Tool Description Generation /*
EASYTOOL uses ChatGPT prompts to streamline tool documentation into concise descriptions and functional guidelines, while supplementing parameter information for tool execution. Annotator evaluations found the generated descriptions accurate and the scenarios reasonable.
- Table 2 presents prompts for generating tool descriptions and constructing tool function guidelines, with ChatGPT-generated text highlighted in green.
- EASYTOOL streamlines documentation by removing irrelevant information and retaining core tool functionality for LLMs.
- ChatGPT follows designed instructions and extra demonstrations to summarize each tool’s general purpose from its documentation.
- The framework supplements tool instructions with extracted parameters because parameter errors can prevent successful tool execution.
- Three annotators found all 100 evaluated tool descriptions accurate and all 100 sampled functionality-guideline scenarios reasonable.
5 Experiment
EASYTOOL is evaluated across tool retrieval, selection, execution, real-world web-service paths, and numerical reasoning. The experiments examine token cost, tool-use performance, retrieval quality, selection accuracy, error rates, and correct path rates across multiple benchmarks and models.
- Experimental scope: The experiments cover three distinct tool-use applications: answering real-world user requests, finding correct tool solution paths, and solving complex math problems.These applications use ToolBench, RestBench, and FuncQA-related evaluations.
- Token efficiency: 70.43%: EASYTOOL reduces the average ToolBench token cost when replacing tool documentation with concise tool instructions.Each instruction contains a tool description and functionality guidelines.
- ToolBench results: EASYTOOL-generated instructions achieve state-of-the-art performance on ToolBench, with ChatGPT + DFSDT-EASYTOOL surpassing GPT-4 + DFSDT in success rate.ToolBench reports pass rate, win rate, and success rate, with win rate compared against ChatGPT-ReACT.
- Selection: On I1-Instruction, the selection experiment compares LLM accuracy across different numbers of candidate tools using original versus EASYTOOL-generated descriptions.The setup uses 100 single-tool-solvable ToolBench requests and retrieves candidate tools by cosine similarity.
- Retrieval: EASYTOOL-Retriever retrieves tools whose performance is comparable to, and sometimes better than, the ground-truth tool set on ToolBench subsets.The authors attribute this to retrieving similar tools with better functionalities that can replace some ground-truth tools.
- Additional evaluations: Across additional evaluations, EASYTOOL improves correct tool paths on RestBench and tool utilization on complex math problems while lowering tool errors.FuncQA results specifically associate the lower error rate with selecting correct tools and passing valid parameters.
6 Conclusion
EASYTOOL enhances LLM-based agents by simplifying and refining tool documentation into clear, structured, practical instructions. The paper reports improved performance in different real-world applications.
- EASYTOOL enhances tool utilization by simplifying and refining tool documentation into clear, structured, practical instructions.
- The method is presented as an easy and effective approach for improving LLM-based agents’ tool utilization capabilities.
- EASYTOOL improves performance in different real-world applications.
Limitations
The paper identifies constraints on documentation length, tool dependency modeling, and model compatibility.
- EASYTOOL processes only tool documentation whose token length does not exceed the ChatGPT input limit.Longer documentation requires additional preprocessing.
- The method is limited to single documentation and neglects dependencies among tools.The paper notes that representing such dependencies could enhance effectiveness in certain scenarios.
- EASYTOOL works only on models with instruction-following ability.The paper suggests training specialized models with EASYTOOL-generated tool instructions as future work.
Ethics Statement
The paper states adherence to the ACL Code of Ethics and Code of Conduct, describes protections for human annotators, and acknowledges risks in benchmark data and evaluation.
- The authors state that all authors are informed about and adhere to the ACL Code of Ethics and Code of Conduct.
- Use of Human Annotations: Human annotators were recruited to annotate tool descriptions and functionality guidelines, with privacy rights respected during annotation.
- Use of Human Annotations: Annotators received compensation exceeding the local minimum wage and consented to research use of EASYTOOL-generated instructions.
- Risks: The publicly sourced tool benchmarks may contain socially harmful or toxic language.
- Risks: Tool-instruction data quality evaluation is based on common sense, which can vary across individuals from diverse backgrounds.
- Use of Human Annotations: The annotation interface and instructions are documented through screenshots in Figures 7 and 8.
B.1 Success Rate Evaluation
This appendix describes the prompt template and model-handling procedures used to evaluate success rates on ToolBench.
- The success-rate evaluation uses a prompt that asks GPT-4 to judge whether a response reasonably and accurately answers the question.
- GPT-4 is instructed to provide reasons first and then output a parsable JSON decision with a Yes or No choice.
- Baselines on ToolBench: Vicuna-7B is described as a LLaMA variant fine-tuned on instructions and user-shared conversations.
- ChatGPT and GPT-4 directly use function calling, while other models receive synthesized inputs in function-call format.
C Robustness Evaluation
The robustness evaluation tests whether EASYTOOL’s task prompts remain effective when rewritten by ChatGPT. It compares outputs produced from the original prompts with outputs produced from three rewritten prompt variants, using annotator judgments.
- Evaluation Setup: 100 ToolBench documentations were processed with the original task prompts and three ChatGPT-generated prompt variants.Two annotators evaluated the resulting tool descriptions and functionality guidelines across four prompt conditions.
- Prompt Variants: The rewritten prompts target succinct tool descriptions focused on functions and scenarios that incorporate the provided parameters.They also specify an empty-parameter output format when both required and optional parameters are absent.
- Evaluation Criteria: The evaluation compares percentage win, tie, and loss outcomes between EASYTOOL’s task prompts and ChatGPT-generated task prompts.Figure 6 summarizes these comparison categories, while Figures 7 and 8 show the annotation interfaces for tool descriptions and functionality guidelines.
- Compression Comparison: LLMLingua is considered as a prompt-compression alternative but may remove parameter and function tokens needed for successful tool execution.The paper therefore states that this compression method cannot be applied directly to the task.