PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM With Tools: A Survey

Zhuocheng Shen

arXiv:2409.18807v1cs.AI

TL;DR

LLMs may hallucinate or lack the current information needed for precise, real-time, or complex tasks, motivating external-tool use. The paper presents a standardized framework and reviews related tool-learning approaches; its Chameleon reproduction reports QA results close to the original, with some diagram differences.

  • Problem

    LLMs may be unreliable in precision or real-time settings because training data may not cover current or specific information, and tools may be needed for calculations or other specialized tasks.

    Source: §I · p. 1, A. Time to invoke the tool · p. 1

  • Method

    The paper frames external-tool use as a standardized process and examines research on LLMs creating tools.

    Source: §III · p. 2, H. LLM creates its own tools · p. 7

  • Results

    The reproduced Chameleon QA results are basically similar to the reported results, differing by only a decimal point; its state-transition diagrams differ slightly, while the call-scale diagram matches.

    Source: B. Experiement results · p. 8

  • Takeaways & Limitations

    The standardized framework offers a macro perspective for examining stages involved in using or developing external tools.

    Source: §III · p. 2

    As tool caches grow, selecting tools through context alone may make context length a significant challenge.

    Source: H. LLM creates its own tools · p. 7

Abstract

from arXiv · show

The integration of tools in augmenting large language models presents a novel approach toward enhancing the efficiency and accuracy of these models in handling specific, complex tasks. This paper delves into the methodology,challenges, and developments in the realm of teaching LLMs to use external tools, thereby pushing the boundaries of their capabilities beyond pre-existing knowledge bases. We introduce a standardized paradigm for tool integration guided by a series of functions that map user instructions to actionable plans and their execution, emphasizing the significance of understanding user intent, tool selection, and dynamic plan adjustment. Our exploration reveals the various challenges encountered, such as tool invocation timing, selection accuracy, and the need for robust reasoning processes. In addressing these challenges, we investigate techniques within the context of fine-tuning and incontext learning paradigms, highlighting innovative approaches to ensure diversity, augment datasets, and improve generalization.Furthermore, we investigate a perspective on enabling LLMs to not only utilize but also autonomously create tools, which may redefine their role from mere tool users to tool creators. Finally,we reproduced Chameleon's results on ScienceQA and analyzed the code structure.

I. INTRODUCTION

LLMs can be unreliable in precision, real-time, and complex tasks, motivating tool use to extend their capabilities. Effective use depends on recognizing when tools are needed and selecting them accurately.

I. INTRODUCTION

  • Tool use can improve accuracy and efficiency on specific tasks where models lack current information or may hallucinate.The paper identifies complex mathematical operations and recent knowledge as examples.Source: §I · p. 1

A. Time to invoke the tool

  • Call external tools for real-time information, complex domain calculations, or special-format documents, but not automatically when model knowledge and reasoning suffice.Unnecessary calls may reduce efficiency, increase costs, or produce errors [6], [7].Source: A. Time to invoke the tool · pp. 1–2

B. Tool selection and accuracy

  • Tool selection becomes harder as available tools grow, and mismatched capabilities or errors in multi-tool reasoning can derail task completion [8].The introduction frames a broader process around intent understanding, tool knowledge, task decomposition, and dynamic plan adjustment.Source: §I · p. 1, B. Tool selection and accuracy · p. 2

C. Method of tool call

Tool-use systems must manage accurate calls, error recovery, time costs, and generalization as tools and tasks vary. The paper frames tool use as an iterative process that connects user intent, planning, execution, feedback, and plan adjustment.

C. Method of tool call

  • Understanding tool interfaces and parameter types helps avoid failed or incorrect calls and supports efficient operation [9]– [11].The required parameters include their number, type, and value.Source: C. Method of tool call · p. 2

D. Robustness of reasoning process

  • Errors can accumulate and amplify through reasoning chains, making detection, correction, and backtracking difficult in multi-step operations [12], [13].Correction should avoid crashing or making the process inefficient.Source: D. Robustness of reasoning process · p. 2

E. Time efficiency

  • More complex reasoning and tool calls increase pipeline time, while network requests or large-data processing can add delays.The paper identifies optimizing efficiency and parallelism without reducing task quality as a challenge.Source: E. Time efficiency · p. 2

F. Generalization ability

  • Whether models can select different tools across scenarios and innovate for unfamiliar tools remains an open generalization question.The paper links this challenge to the prospect of achieving complete AGI.Source: F. Generalization ability · p. 2

III. RESEARCH STATUS

  • The paper introduces a standardized method for examining the stages of external-tool use and development from a macro perspective [14].It positions this framework amid a shift from teaching tool use toward exploring model-created tools.Source: §III · p. 2

A. Paradigm of tool use

  • Tool use iterates from interpreting an instruction and planning with available tools through execution, feedback, summary, and plan adjustment until task completion.The process is represented as a sequence of functions over user instructions, tools, and environment state.Source: A. Paradigm of tool use · p. 2, Figure 1 · p. 3

B. Fine tuning

The fine-tuning discussion describes training samples that combine model-generated text with marked tool invocations. During inference, the model can call a tool at a designated marker and incorporate its response into continued generation.

B. Fine tuning

  • Fine-tuning uses samples that place a tool-invocation construct within model-generated text, which also surrounds the invocation with ordinary generated words.The construct identifies the tool invocation and its result.Source: B. Fine tuning · p. 3
  • Fine-tuning on C* incrementally equips the model to decide when, where, and how to use tools based on context and internal feedback.The paper presents C* as a simplified dataset for explaining the fine-tuning procedure.Source: B. Fine tuning · p. 3
  • At inference, the model recognizes a tool marker, forms a request from the tool name and input, then integrates the returned result into its continuing output.The procedure is described as text generation with tool integration.Source: B. Fine tuning · p. 3

C. Fine tuned dataset construction

Fine-tuning depends on high-quality annotated datasets, and the paper contrasts human-interaction data with datasets generated using advanced models. The approaches trade off realism and relevance against collection cost, scope, and potential data-quality risks.

C. Fine tuned dataset construction

  • A central fine-tuning challenge is obtaining high-quality annotated datasets for the target tasks or tool applications.The paper surveys ways to acquire such data and their advantages and disadvantages.Source: C. Fine tuned dataset construction · p. 3
  • Human tool-interaction data, as in WebGPT [16] and WebCPM [17], offers integrity and relevance but entails labor costs and restricted tool coverage.Reliance on particular tools or combinations can limit coverage of diverse operational settings.Source: C. Fine tuned dataset construction · pp. 3–4
  • LLM-generated datasets such as Toolformer [15] and ToolCoder [18] reduce labor and cover more scenarios, but may contain contamination or distortional bias.The paper emphasizes data quality and robust quality-enhancement measures.Source: C. Fine tuned dataset construction · p. 4

1) Diversity:

The section links diverse tool-use data to stronger generalization and describes methods for generating varied examples, especially for complex multi-tool tasks.

1) Diversity:

  • Datasets spanning a broad range of case types substantially improve generalization to tool applications, as reported by GPT4Tools [6], ToolAlpaca [19], and ApiBank [9].Source: 1) Diversity · p. 4
  • Image-linked tool instructions broaden instructional diversity by drawing on variation in image content, according to GPT4Tools [6].Source: 1) Diversity · p. 4
  • Short-term memory records recent trial-and-error outcomes, while long-term memory preserves prior experience to help diversify generated tool-use cases [20].Source: 1) Diversity · p. 4

2) Data augmentation:

  • Multi-agent simulation reduces the labor of drafting multi-stage tool-use examples and better matches real-world dialogue, supporting practicality and generalization.The user, assistant, and tool-executor agents simulate the interaction workflow.Source: 2) Data augmentation · p. 5
  • TaskBench [23] builds tool graphs from dependencies, while ToolLLM [8] samples APIs within categories to generate varied multi-tool scenarios.Source: 2) Data augmentation · p. 5

D. Other issues during the fine-tune phase

The section discusses fine-tuning challenges involving retention, tool growth, and evaluation, then describes few-shot prompting with sequential tool use and its scope limits.

D. Other issues during the fine-tune phase

E. In context learning without retrieval

F. In context learning with retrieval

This section presents retrieval as a way to select tools or task examples for in-context learning, and describes documentation-based benefits and retrieval approaches.

F. In context learning with retrieval

G. Other issues during the in context learning phase

The section surveys ways to make tool use more adaptable, including online planning, self-correction, and tree search, then considers tool creation and tool selection. It also identifies unresolved challenges in reusing created tools and scaling similarity-based selection.

G. Other issues during the in context learning phase

  • Online planning [12], a learner module [32], and DFSDT tree search [8] are three approaches to adapting decisions or correcting errors during in-context tool use.Chatcot adjusts future actions using prior decisions and feedback with minimal added token consumption; the learner attempts to rectify execution errors, while DFSDT explores multiple decision paths.Source: G. Other issues during the in context… · p. 7

H. LLM creates its own tools

  • Creator [14] uses creation, decision, execution, and recognition stages to generate tools, guide their use, execute them, and revise tools or decisions.Creator and Large language models as tool makers [33] are presented as evidence that LLMs can generate tools and solve unknown problems.Source: H. LLM creates its own tools · p. 7
  • Created tools can transfer to similar scenarios, but reuse at scale remains unresolved; a dispatcher [33] searches cached tools, while context length may constrain selection.The passage proposes retrieval to locate suitable tools and reduce context-length pressure, while noting that systematic research is lacking.Source: H. LLM creates its own tools · p. 7

I. Other methods

  • Gear [34] combines semantic similarity between a question and tool descriptions with pattern similarity between preliminary answers and tool outputs to score tools.The scores support automatic tool selection and answer generation.Source: §I · p. 8
  • Gear’s selection scheme calls all tools to obtain outputs, which may be costly at scale, and is mainly limited to single problems and available tool coverage.The passage suggests optimizing computational cost and broadening support for complex scenarios.Source: §I · p. 8

IV. REFLECTION AND OUTLOOK

The outlook identifies open research directions for tool-using LLMs, spanning scheduling, pipeline speed, continuous adaptation, error recovery, and tool selection.

A. Research on tool scheduling topology optimization and graph decision making

  • Tool scheduling research could optimize tool-call topology and connect graph data structures or graph neural networks with LLM-based decomposition and decisions.The proposed objective is to generate accurate answers from predefined tools and user queries.Source: A. Research on tool scheduling… · p. 8

B. Time optimization of model pipelines

C. Model optimization and continuous learning

D. Compensate for API errors and prevent error cascading

E. Learn to use tools

  • Finding the most suitable tools remains unresolved, and the accuracy of existing tool-selection research is described as far from satisfactory.Source: E. Learn to use tools · p. 8

F. Pretrain tool augmented LLMs

The paper reports a reproduction of Chameleon evaluated on ScienceQA and describes tool pretraining as a possible research direction. The reproduced QA accuracy is nearly the same as the paper’s, although the state-transition diagrams differ slightly.

F. Pretrain tool augmented LLMs

  • The authors suggest pretraining tools to augment LLMs, noting the diversity of tool-call cases and the prevalence of fine-tuning or in-context learning in existing research.Source: F. Pretrain tool augmented LLMs · p. 8

A. Experiment settings

  • The experiment used GPT-3.5 (gpt-35-turbo-16k-0613) through Azure OpenAI Service and tested CoT and Chameleon on 4,241 ScienceQA test examples.Source: A. Experiment settings · p. 8

B. Experiement results

C. Code structure and overall process of Chameleon

The code description presents Chameleon as a solver with specialized modules and an iterative execution process that runs predicted modules and records evaluation statistics.

Loading 2409.18807v1…