PaperlayerView exact revision on arXiv

Source-linked AI summary

Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agents

Zhengliang Shi, Shen Gao, Lingyong Yan, Yue Feng, Xiuyi Chen, Zhumin Chen, Dawei Yin, Suzan Verberne, Zhaochun Ren

arXiv:2405.16533v2cs.CL

TL;DR

Prior tool-use methods depend on expert-built documentation formats and workflows that do not flexibly accommodate diverse tools. AutoTools automates function creation and programmatic tool use, and experiments report better performance and efficiency than previous baselines, with further expertise gains from AutoTools-Learning.

  • Problem

    Previous methods require expertise to parse tool documentation and create demonstrations, while ad-hoc workflows generalize poorly across tool specifications and flexible multi-tool use.

    Source: §1 · p. 2

  • Method

    AutoTools converts tool documentation into verified callable functions and uses executable programs to integrate them, while AutoTools-Learning trains documentation understanding, relevance learning, and function programming.

    Source: §1 · p. 2, §8 · p. 8

  • Results

    Experiments report that AutoTools surpasses previous baselines with higher efficiency, and AutoTools-Learning further enhances LLM expertise within the framework.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors identify integrating AutoTools into vision foundation models for multimodal real-world task-solving as a direction for future research.

    Source: §8 · p. 9

    Most reported errors involve selecting incorrect functions or mismatching expected return types, and reducing candidate tools from 20 to 10 improved success rates by 2–3 points across three benchmarks.

    Source: §7 · p. 8

Abstract

from arXiv · show

Augmenting large language models (LLMs) with external tools has emerged as a promising approach to extend their utility, enabling them to solve practical tasks. Previous methods manually parse tool documentation and create in-context demonstrations, transforming tools into structured formats for LLMs to use in their step-by-step reasoning. However, this manual process requires domain expertise and struggles to scale to large toolsets. Additionally, these methods rely heavily on ad-hoc inference techniques or special tokens to integrate free-form LLM generation with tool-calling actions, limiting the LLM's flexibility in handling diverse tool specifications and integrating multiple tools. In this work, we propose AutoTools, a framework that enables LLMs to automate the tool-use workflow. Specifically, the LLM automatically transforms tool documentation into callable functions, verifying syntax and runtime correctness. Then, the LLM integrates these functions into executable programs to solve practical tasks, flexibly grounding tool-use actions into its reasoning processes. Extensive experiments on existing and newly collected, more challenging benchmarks illustrate the superiority of our framework. Inspired by these promising results, we further investigate how to improve the expertise of LLMs, especially open-source LLMs with fewer parameters, within AutoTools. Thus, we propose the AutoTools-learning approach, training the LLMs with three learning tasks on 34k instances of high-quality synthetic data, including documentation understanding, relevance learning, and function programming. Fine-grained results validate the effectiveness of our overall training approach and each individual task. Our methods are an important step towards the use of LLMs for solving real-world tasks with external tools.

1 Introduction

AutoTools automates tool-use workflow through tool encapsulation and programming, addressing the expertise and flexibility challenges of manually prepared tool-use methods. Experiments report stronger performance and efficiency than previous baselines, with further gains from AutoTools-Learning.

1 Introduction

  • Earlier tool-use methods require substantial expertise to parse documentation and create demonstrations, and their ad-hoc workflows generalize poorly across tool specifications.Incomplete in-context examples can diminish performance, and some workflows restrict flexible integration of multiple tools.Source: §1 · p. 2
  • AutoTools has two stages: the LLM converts tool documentation into callable functions, then flexibly integrates those functions through programming to solve tasks.The framework verifies function syntax and runtime correctness before use.Source: §1 · p. 2
  • AutoTools-Learning trains LLMs on documentation understanding, relevance learning, and function programming using synthesized data filtered from public resources.The paper reports releasing 34k high-quality training examples and says the approach further improves LLMs within AutoTools.Source: §1 · p. 2
  • Experiments across RestBench, ToolBench, and the more challenging AutoTools-Eval report that AutoTools surpasses previous baselines with higher efficiency, while AutoTools-Learning further enhances LLM expertise.AutoTools-Eval contains 224 tasks across 107 real-world tools and features longer-term planning, complex documentation, and strong input-output dependencies.Source: §1 · p. 2
  • The evaluation covers RestBench and ToolBench as well as AutoTools-Eval, a newly collected benchmark designed around more challenging real-world tool-use scenarios.AutoTools-Eval has 224 tasks across 107 tools and includes long-term planning, complex documentation, and strong input-output dependencies.Source: §1 · p. 2

2 Related work

Related work describes manual tool preparation, programming-based tool use, and synthetic training-data approaches. AutoTools differs by automating function creation and reformatting established datasets for three tool-learning tasks.

Tool learning with foundation models

  • Earlier tool-learning approaches manually convert tools into executable functions and require human expertise to create structured formats and in-context examples, limiting scalability.AutoTools instead explores having LLMs automatically encapsulate tools using their documentation.Source: §2 · p. 3

Programming-enhanced LLMs

  • Programming-enhanced LLM work uses code for planning, reasoning, or tool actions, but generally relies on functions that people have already processed.AutoTools extends this approach by generating callable functions grounded in tool documentation and creating demonstrations.Source: §2 · p. 3

Data synthesis

  • Prior tool-use data synthesis often generates queries and chain-of-thought solutions from scratch, a practice reported to suffer from low diversity and uncontrollable quality [4] [15].AutoTools-Learning reformats established datasets and supervises documentation understanding, query-tool relevance, and programmatic tool use.Source: §2 · p. 3

3 The proposed method: AutoTools

AutoTools first turns tool documentation into verified callable functions and then uses executable programs to combine them for task-solving. Its method includes dependency-aware integration verification and a programming stage that uses a unified language.

3 The proposed method: AutoTools

  • AutoTools unifies diverse tool-use specifications by encapsulating documentation as callable functions and having the LLM integrate them through executable programs.The framework verifies runtime correctness and provides test instances as demonstrations in a function library.Source: §3 · p. 3

3.1 Encapsulation: tools

  • For each tool, the LLM transforms natural-language documentation into a directly callable function, then checks generated code for syntax errors and retries when needed.The documentation can include arguments, functionality, access URLs, and state codes; signature parameters must match the documentation.Source: §3.1 · pp. 3–4

3.2 Integration verification: funcs

  • Integration verification tests a target function alongside relevant prerequisite functions, so private inputs can be obtained before the target call.Verified functions are cached and augmented with test instances as in-context demonstrations; the process repeats until tools are processed or the iteration limit is reached.Source: §3.2 · p. 4

3.3 Tool programming: LLM

  • In tool programming, the LLM directly manipulates multiple encapsulated functions using a unified programming language and generates executable programs as tool-use actions.The functions are loaded into an execution environment, where generated programs call them sequentially and use execution results in further steps.Source: §3.3 · p. 4

4 Learning with AutoTools

Motivated by improving LLM expertise within AutoTools, especially for smaller models, AutoTools-Learning trains three tool-use skills using synthetic data.

4.1 Learning tasks and objectives

  • AutoTools-Learning trains documentation understanding, relevance selection, and function programming, including generating executable programs that can use control flow and reuse intermediate results.The tasks cover converting documentation into functions, selecting relevant functions for a query, and solving tasks through interactions with an execution environment.Source: §4.1 · p. 5

Dataset

  • A test-set comparison table presents AutoTools-Eval alongside existing benchmarks and lists task, tool-path, and documentation statistics.The supplied table text does not support assigning values to those statistics.Source: Dataset · p. 5, Table 2

4.2 Training data synthesis

  • Training data combines ToolBench tools, tool-retrieval datasets, and task-solving trajectories reformatted for the three learning tasks.GPT-4o generates program solutions for the function-learning data by referencing the original ground truth.Source: Dataset · p. 5

Dataset

  • After filtering empty responses, unsolvable queries, and incorrect parameters, the training set contains 2.6k, 15k, and 17k examples for the three tasks, respectively.The examples are reformatted into a unified interactive format, and training combines all three tasks.Source: Dataset · p. 5

5 Dataset and evaluation setup

The evaluation uses RestBench, ToolBench, and the new AutoTools-Eval benchmark, with task-specific metrics and established tool-use baselines.

5.1 Dataset

  • AutoTools-Eval contains 224 complex tasks across 107 tools, with long-term reasoning, at least five tool calls, and dependencies between successive tools.The benchmark covers long tool documentation and uses tasks designed to be stable over time.Source: §5.1 · pp. 5–6

5.2 Evaluation metrics

  • RestBench is evaluated by success rate, correct path rate, and tool precision; ToolBench uses pass rate, while encapsulation uses correctly encapsulated tools.Success rate measures whether all required tools are called; correct path rate measures the share of ground-truth tools in generated calls.Source: §5.2 · p. 6

5.3 Baselines

  • Baselines include ReAct, CodeAct, ToolLLM-DFSDT, RestGPT, and ConAgents, plus up-to-three-run ReAct@3 and ToolLLM@3 variants.Evaluation inputs use candidate toolsets of about 20 tools, including target and randomly sampled tools.Source: §5.3 · p. 6

6 Experimental results

Experiments assess tool encapsulation, task-solving performance, learning gains, and inference efficiency. AutoTools outperforms baselines, improves further on a challenging benchmark, and uses fewer tokens while achieving better performance.

6.1 RQ1 – Performance on tool encapsulation.

  • GPT-4 correctly encapsulates almost 90%–95% of tools, while Mixtral-8x7B encapsulates 82.5%–88.2%.Removing syntax compilation reduces correct encapsulations by 3–7 points, and removing integration verification causes a substantial decrease.Source: §6.1 · p. 6, Table 3 · p. 6

6.2 RQ2 – Overall tool-use performance

  • 89.00% success rate on TMDB (RestBench); AutoTools surpasses all baselines across metrics on RestBench and ToolBench.Results with GPT-3.5-turbo are reported in Table 4; the framework also performs best with Mistral-8x7B and GPT-4.Source: §6.2 · p. 7, Table 4 · p. 7, Table 5 · p. 7
  • 60.21% success rate on AutoTools-Eval versus the best baseline’s 44.70%, a 15.51-point increase.Success rises as maximum interaction turns increase from one to four, then stabilizes from four to five; tasks average 7.31 tool calls.Source: §6.2 · p. 7, Figure 4 · p. 7

6.3 RQ3 – Improvement through learning

  • 16.52% success rate on AutoTools-Eval for trained Mistral-7B, up from 10.71% with the vanilla model.The learning approach trains open-source LLMs with synthetic examples and multiple learning tasks.Source: §6.3 · p. 8, Table 6 · p. 8
  • Removing function learning reduces success by 4.00 points on TMDB and 2.1 points on AutoTools-Eval; removing document understanding reduces it by 3.00 and 1.30 points, respectively.Removing relevance learning decreases correct path rate on both datasets; function learning has the most pronounced ablation impact.Source: §6.3 · p. 8, Table 6 · p. 8

6.4 RQ4 – Inference efficiency analysis

  • AutoTools consumes fewer tokens than all baselines while achieving better performance.GPT-3.5-turbo-16k uses 2703 tokens on average to encapsulate a tool with usage examples; encapsulated functions can be cached for reuse.Source: §6.4 · p. 8, Figure 5 · p. 8

7 Discussion

The discussion examines failure patterns and a candidate-tool reduction experiment, alongside case studies and human evaluation.

7 Discussion

  • Most errors involve selecting incorrect functions or mismatching expected return types among similar functions.Table 7 reports the error statistics.Source: §7 · p. 8, Table 7 · p. 8
  • Reducing candidate tools from 20 to 10 improves success rate by 2–3 points across RestBench, AutoTools-Eval, and ToolBench.Filtering irrelevant functions is proposed as future work.Source: §7 · p. 8
  • Case studies and human evaluation supplement automatic evaluation; examples and results are in Appendix A.4.Source: §7 · p. 8

8 Conclusions

The conclusion presents AutoTools as an automated tool-learning framework and reports its evaluation on existing and newly created datasets. It also introduces AutoTools-Learning and points to multimodal agents as a future direction.

8 Conclusions

  • AutoTools automates tool use by turning documentation into callable functions, checking syntax and runtime correctness, and integrating functions into executable programs.Source: §8 · p. 8
  • The framework addresses reliance on human expertise to structure documentation and examples, and limitations of handcrafted tool-use control flows.Source: §8 · p. 9
  • AutoTools-Learning further enhances LLM capabilities, particularly for smaller open-source models.Source: §8 · p. 9
  • The authors propose integrating the framework into vision foundation models to develop multimodal agents for real-world tasks.Source: §8 · p. 9

Ethical Use of Data and Informed Consent

The authors describe their data and methods as following ethical standards.

Ethical Use of Data and Informed Consent

Ethical Use of Data and Informed Consent

The authors state that they followed ethical standards and used publicly accessible platforms or widely used benchmarks. They report no intended harm or deception and plan to open-source research materials after anonymization.

Ethical Use of Data and Informed Consent

  • The study used tools from publicly accessible platforms or widely used benchmarks, which the authors say supports transparency and reproducibility.They also state these choices minimize potential bias and promote fairness.Source: Ethical Use of Data and Informed Consent · p. 12
  • The authors report that the research was not intended to harm individuals or groups, involve deception, or enable misuse.They state that the tools pose no harm and that no malicious behavior is associated with the LLMs or tools.Source: Ethical Use of Data and Informed Consent · p. 12
  • The authors state that they ensured the LLMs do not produce harmful or toxic outputs and will make code, prompts, and datasets available after the anonymization period.Source: Ethical Use of Data and Informed Consent · p. 12

A.1 Training Data Synthetic

This section describes AutoTools-Learning’s interactive synthetic training data for tool understanding, relevance learning, and function programming, alongside appendix materials on human evaluation, AutoTools-Eval’s design and collection, a case study, and experiment instructions. The instructions prompt models to produce Python programs and identify relevant APIs, including APIs that provide dependencies needed for a successful call.

A.1 Training Data Synthetic

  • AutoTools-Learning synthesizes interactive training data for tool understanding, relevance learning, and function learning from tool documentation, retrieval examples, and step-level task-solving trajectories.For function learning, GPT-4o reformats annotated solutions into programs; examples with syntax errors or mismatched tool parameters are discarded.Source: §A.1 · p. 12

A.2 More Experiment Details

  • 0.21 absolute improvement in Executability over ToolLLM@3; our method also achieves higher Utility, according to the human evaluation.Three volunteers scored 30 cases; annotator agreement was 0.70 for Executability and 0.69 for Tool utilization.Source: Table 8 · p. 12, §A.2 · pp. 12–13

A.3 A new benchmark – AutoTools-Eval

  • AutoTools-Eval targets long-term planning and connected reasoning with at least seven tool calls on average, interdependent tool arguments, and time-stable outputs.Seven NLP researchers manually constructed and solved tasks; the benchmark contains 227 examples across 107 tools in four domains, with ground-truth tool paths and arguments.Source: §A.3 · p. 13, §A.3.1 · p. 13

A.3.2 Strategy for quality improvement.

  • Benchmark quality controls include annotator training, repeated independent annotations with third-expert review for discrepancies, ambiguity filtering, and periodic audits.Source: §A.3.2 · p. 13

A.4 Case Study

  • Case studies illustrate tool encapsulation and program-based coordination, including a Christopher Nolan query that chains person search, movie credits, and poster retrieval.The example program returns poster paths as its output.Source: §A.4 · pp. 15–16

A.5 Experiment Instruction

  • Experiment instructions cover encapsulating APIs as callable Python functions, selecting relevant tools, and writing executable programs that integrate functions to print answers.Encapsulation guidance calls for self-contained functions with required parameters and HTTP error handling.Source: §A.5 · p. 17
  • The query instruction requests a Python program as the output, while a separate instruction asks the model to select relevant tools.Source: §A.5 · p. 17
  • The API-selection prompt supplies development documentation and a candidate API list as inputs.Source: §A.5 · p. 17
  • The API-selection instruction asks the model to find candidate APIs whose outputs provide the input parameters required for a successful call.The prompt frames the APIs as having strong input-output dependencies and requests relevant API names in a one-line Python list.Source: §A.5 · p. 17
Loading 2405.16533v2…