PaperlayerView exact revision on arXiv

Source-linked AI summary

Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmark

Mengsong Wu, Tong Zhu, Han Han, Chuanyuan Tan, Xiang Zhang, Wenliang Chen

arXiv:2405.08355v1cs.CL

TL;DR

LLM agents need high-quality tool-learning data, while direct LLM generation can produce duplicates, simple instances, and hallucinated tool calls. Seal-Tools uses a controlled self-instruct process to construct tool-use data, including nested calls, and evaluates systems that still show room for improvement.

  • Problem

    Tool-learning datasets are needed to teach LLM agents to use tools, but direct LLM generation can yield duplicates, simple instances, and incorrect calls from hallucination.

    Source: §1 · p. 1

  • Method

    Seal-Tools uses a self-instruct process to generate tools and single- or multiple-tool instances, with staged checks and strict JSON descriptions.

    Source: §1 · pp. 1–2

  • Results

    Current agent systems still have room for improvement on Seal-Tools, especially in nested calling.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Seal-Tools is presented as a benchmark for evaluating LLM tool-calling ability with metrics spanning three dimensions.

    Source: §1 · p. 2, §5 · p. 9

    Funding constraints kept the dataset below the upper scale limit of the construction method and model capacity.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

This paper presents a new tool learning dataset Seal-Tools, which contains self-instruct API-like tools. Seal-Tools not only offers a large number of tools, but also includes instances which demonstrate the practical application of tools. Seeking to generate data on a large scale while ensuring reliability, we propose a self-instruct method to generate tools and instances, allowing precise control over the process. Moreover, our Seal-Tools contains hard instances that call multiple tools to complete the job, among which some are nested tool callings. For precise and comprehensive evaluation, we use strict format control and design three metrics from different dimensions. Therefore, Seal-Tools can serve as a new benchmark to evaluate the tool-calling ability of LLMs. Finally, we evaluate several prevalent LLMs and our finetuned model on Seal-Tools. The results show that current systems are far from perfect. The code, data and experiment results are available at https://github.com/fairyshine/Seal-Tools .

1 Introduction

Tool-learning datasets are needed to teach LLM agents to use tools, but existing datasets and direct LLM generation have limitations in scale, evaluation, and reliability. Seal-Tools uses a controlled self-instruct process to build tool-use instances and evaluate systems, which still have room to improve, especially on nested calls.

1 Introduction

  • Existing tool-learning datasets have limited scale, coarse evaluation, or closed training data, while direct LLM generation risks duplicates, simple instances, and hallucinated calls.These shortcomings motivate a large-scale dataset with automatic, precise evaluation.Source: §1 · p. 1
  • Seal-Tools uses field-based tool generation and staged instance generation with checks to reduce duplication and errors from LLM hallucination.Tools and instances are described in strict JSON format.Source: §1 · p. 1
  • Seal-Tools includes difficult instances with nested tool calls, designed to support agent fine-tuning.The paper describes these instances as valuable for finetuning.Source: §1 · p. 2
  • Its evaluation covers Output Format, Tool Selection, and Tool-Parameter Filling-in, with JSON post-processing enabling more automatic and precise evaluation.The introduction also describes strict format control as supporting automatic evaluation.Source: §1 · p. 2
  • Current systems show room for improvement on Seal-Tools, especially in nested calling.The authors report evaluating mainstream agent systems and a foundation model finetuned with Seal-Tools.Source: §1 · p. 2

2 Related Work

The related work frames tool learning as part of LLM agents and distinguishes prompt-based from finetune-based approaches. It also situates in-context learning as a prompting paradigm that can be extended to scenarios without human supervision.

2.1 In-Context Learning

  • In-context learning uses task demonstrations in prompts, and Auto-ICL generates similar question-answer examples for scenarios without human supervision.Prior work also studies demonstration selection and the mechanisms behind in-context learning.Source: §2.1 · p. 2

2.2 Tool Learning

  • The paper describes tool-learning agents as comprising a foundation model, retriever, and tool pool, and groups tool learning into prompt-based and finetune-based approaches.The foundation model decides whether to call tools, while the retriever finds relevant tool information.Source: §2.2 · pp. 2–3
  • Prompt-based agents select tools from an external pool, whereas finetune-based agents learn tools through finetuning without needing an extra pool.The paper uses a prompt-based framework to explore accurate tool calling across many tools; it notes that finetune-based agents need advance training to remember tools.Source: §2.2 · p. 3

3 Method

Seal-Tools uses a staged self-instruct process to generate fields, tools, and practical tool-calling instances, with checks to improve reliability. Its dataset analysis compares the resulting tools, instances, and evaluation approach with existing tool-learning datasets.

3 Method

  • The construction pipeline generates fields, tools, and instances in stages, using LLMs with quality checks to manage duplication and instance reliability.The method is intended to be convenient to try, require limited human involvement, and allow data scale to be controlled within LLM capability.Source: §3.1 · p. 3, §3 · p. 3

3.1.1 Field Generation

  • Hierarchical fields and subfields anchor tool generation to improve diversity, producing 146 fields and 5,860 subfields.The authors report that tools repeated frequently when they skipped field generation.Source: §3.1.1 · pp. 3–4

3.1.2 Tool Generation

  • Tools are generated by subfield with example parameter values, then format-checked and deduplicated; sensitive personal parameters receive rule-generated values, yielding 4,076 tools.The authors found that including parameter-value examples helped avoid low-quality, repetitive values not mentioned in queries.Source: §3.1.2 · pp. 4–5

3.1.3 Instance Generation

  • Single-tool generation produces 4,076 instances; multi-tool generation selects compatible tools, fills a calling template, and produces 10k instances.Nested instances allow a previous tool’s response to become the next tool’s parameter, so calls can form a directed acyclic graph rather than a simple flow line.Source: §3.1.3 · p. 5

3.2 Dataset Analysis

  • Seal-Tools is described as competitive among ToolBench, API-Bank, and ToolAlpaca.The comparison is presented in Table 1.Source: §3.2 · p. 5, Table 1 · p. 6
  • Nearly 34% of ToolBench tools have no required parameters, compared with 6% of Seal-Tools tools.The authors also report that models must fill in more parameters per instance in Seal-Tools.Source: §3.2 · pp. 5–6, Figure 6 · p. 6
  • Seal-Tools is second only to ToolBench in instance scale and includes cross-field and nested multi-tool requests resolved in one turn.The authors describe these instances as more difficult and closer to real-world scenarios than step-by-step multi-tool tasks.Source: §3.2 · p. 6
  • Seal-Tools post-processes model outputs into JSON to evaluate tool selection and parameter filling, while the authors characterize prior evaluation approaches as less detailed.The paper contrasts its approach with ToolBench’s ChatGPT-provided pass and win rates and API-Bank’s Rouge score.Source: §3.2 · p. 6

4 Experiment

The experiments evaluate tool-format compliance, tool selection, and parameter filling, then compare models on overall, single/multiple-tool, and nested-instance tasks. Seal-Tools fine-tuning improves LLaMA2-7B’s tool-calling scores, while multiple-tool and nested instances remain challenging.

4.1 Evaluation Metric

  • Format ACC measures output-format accuracy, Tool P/R/F1 measures tool selection, and Parameter P/R/F1 measures parameter filling.Together, these metrics examine how well models understand and use tools.Source: §4.1 · pp. 6–7

4.2.1 Overall

  • The evaluation uses DPR’s top-5 relevant tools at test time and adds missing gold tools to the training prompts.The authors selected DPR after evaluating different retrievers; Figure 7 roughly depicts instance simplicity, with higher bars indicating simpler instances.Source: §4.2.1 · p. 7, Figure 7 · p. 7
  • Tool F1 rises 45.92% and Parameter F1 rises 47.59% after Seal-Tools fine-tuning; the model outperforms ChatGPT but remains slightly below GPT4.Open-source LLMs perform similarly, while the paper says ToolLLaMA’s single-dataset fine-tuning harms its understanding of other instructions.Source: §4.2.1 · p. 7, Table 2 · p. 7

4.2.2 Single/Multiple-Tool Instances

  • Most models do better on single-tool than multiple-tool instances, while Seal-Tools fine-tuning beats GPT4 on single-tool and trails slightly on multiple-tool instances.LLaMA2 and Vicuna are exceptions to the general pattern; the authors find they tend to call more tools on single-tool instances.Source: §4.2.2 · p. 7, Table 3 · p. 8

4.2.3 Nested Instances

  • 77.12% Parameter F1 is reported for the fine-tuned model on nested instances, 501 of which appeared in training; unseen nested instances score 5.11% below multiple-tool instances.The authors identify nested instances as the most difficult; LLaMA2-Chat and Mistral perform poorly on them.Source: §4.2.3 · p. 8, Table 4 · p. 8

4.3 Extended Result

With only gold tools in the prompt, the paper examines parameter filling on multiple-tool instances and analyzes errors from the DPR-based system. Tool-selection errors point to retrieval and hallucination, while parameter errors include omissions, overfilling, and query misunderstanding.

4.3.1 Evaluation of Parameter Filling-In

  • With only gold tools supplied, the fine-tuned model meets the expected Tool P of 100%, unlike LLaMA2 and ChatGPT, whose tool-selection results are affected by in-context hallucination.The subsection focuses on parameter filling for multiple-tool instances; its table reports results with only needed tools in the prompt.Source: §4.3.1 · p. 8, Table 5 · p. 8

4.3.2 Error Analysis

  • Tool-selection errors make retrieving all needed tools the urgent issue, while hallucination can produce tools absent from retrieval results.For parameter filling, errors include omitted parameters (7%), unmentioned overfilled parameters (9%), and query misunderstanding or failure to convert requirements (14%).Source: §4.3.2 · p. 8

5 Conclusion

The paper presents Seal-Tools and its controlled construction method as a tool-learning dataset and benchmark. It includes hard nested and cross-field calls, uses three evaluation dimensions, and concludes that current agent systems still have room to improve.

5 Conclusion

  • Seal-Tools is built from tools and practical instances through a construction method that controls data quality to improve reliability and diversity.The paper presents the dataset as a new benchmark for tool learning with LLMs.Source: §5 · pp. 8–9
  • Hard instances include nested tool calls and cross-field calls, areas the paper says have seldom been investigated in prior studies.The conclusion identifies these instances as part of Seal-Tools’ contribution.Source: §5 · p. 9
  • The paper evaluates tool learning across three dimensions and concludes that current agent systems still have room for improvement.It proposes Seal-Tools as a benchmark intended to support further research on tool learning with LLMs.Source: §5 · p. 9

A Prompts of Dataset Construction

The construction prompts organize tools by coarse fields and fine-grained subfields, then generate API descriptions and single- or multiple-tool tasks. For multi-tool tasks, prompts guide API selection, parameter completion, and coherent task descriptions.

A.1 Field Generation

  • Fields are generated first at a coarse level, then divided into fine-grained subfields for task classification.The prompts ask for broad coverage across areas and then a subfield list for each field.Source: §A.1 · p. 10

A.2 Tool Generation

  • For each field and subfield, the prompts request related APIs described as functions with input parameters and output responses.They also require JSON-formatted API definitions with typed parameters and responses.Source: §A.2 · p. 10

A.3 Instance Generation

  • Single-tool instance prompts ask for a natural task description that requires an API, while keeping API names out of the description.Examples pair a function call with a plain-language request, such as translating a phrase or booking a meeting.Source: §A.3 · pp. 10–11
  • For multiple-tool tasks, the prompts direct the model to select APIs with a strong association and arrange them in execution order.The tool-combination prompt allows the original API order to change when the task's chronology requires it.Source: §A.3 · p. 11
  • A single-tool example selects hotel-search, hotel-detail, and cancellation APIs for a task about finding and canceling a reservation.The example returns the chosen API list separately from the task description.Source: §A.3 · p. 11
  • Multi-tool generation fills API parameters with specific values and can reuse previous API responses as later parameters.Prompts also encourage shared parameter values where possible to make the task more consistent.Source: §A.3 · pp. 11–12
  • The generated task description is based on the completed API calls and should mention their parameter values without naming APIs.The prompt requests the completed calling list and task description as the output.Source: §A.3 · pp. 11–13

B Generated Tool Amount per Time

The figure reports generated tool amounts per 200 generation attempts, with duplicate tools filtered in real time. The text says pre-generated field information allows larger-scale generation than direct ChatGPT generation, which may yield around 100 tools.

B Generated Tool Amount per Time

  • Pre-generated field information supports larger-scale tool generation, whereas direct ChatGPT generation may yield only around 100 tools.Figure 9 tracks the generated tool amount per 200 attempts and filters duplicates in real time.Source: B Generated Tool Amount per Time · p. 13, Figure 9 · p. 13

C Detailed Formulae for Evaluation Metrics

The supplied formula fragments name Tool P, Tool R, Parameter P, and Parameter R in the evaluation metrics. They do not provide complete expressions in these passages.

C Detailed Formulae for Evaluation Metrics

D Retriever Comparison

The comparison tests BM25 and DPR retrieval, with DPR performing better and chosen for the agent system. The authors note that retriever optimization remains future work.

D Retriever Comparison

  • The DPR retriever is trained with ranking loss after contrastive loss performs poorly and one-by-one gold-tool training seems to retain only the last tool.The task requires retrieving all needed tools, unlike open-domain QA where one of multiple answers may suffice.Source: D Retriever Comparison · p. 13
  • DPR performs better than BM25, so the agent system uses DPR as its retriever.The comparison is presented as evidence that tool retrieval involves semantic understanding beyond simple keyword matching.Source: Figure 10 · p. 13, D Retriever Comparison · p. 13
  • Further retriever optimization remains an open research direction.The authors identify retriever optimization as deserving further research.Source: D Retriever Comparison · p. 13

E Generalization Verification

This section tests generalization on a small benchmark transformed from task-oriented dialogues, using four in-context prompt styles and reporting mixed behavior for the finetuned model.

E Generalization Verification

  • The benchmark transforms SGD, Multiwoz2.2, and Taskmaster task-oriented dialogue datasets into 20 tools and 519 instances.Table 6 reports experiments on this benchmark.Source: E Generalization Verification · p. 13, Table 6 · p. 13
  • Four in-context prompt styles are vanilla, role-play, task description, and imitation, respectively using a basic instruction, a role, detailed task guidance, or examples.The paper describes the four styles as prompts for in-context evaluation.Source: E Generalization Verification · p. 14
  • The finetuned model outperforms the chat model in Section 4.3.1 but performs badly in Section 4.2; the authors describe different learned abilities under different tool-list conditions.With all needed tools in the prompt, it learns to fill parameters; with a retrieved tool list, it learns tool selection and use.Source: E Generalization Verification · p. 14
Loading 2405.08355v1…