PaperlayerView exact revision on arXiv

Source-linked AI summary

Tool Learning with Large Language Models: A Survey

Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, Ji-Rong Wen

arXiv:2405.17935v3cs.CLcs.AI

TL;DR

Tool learning with LLMs addresses fragmented research and limitations arising from fixed, parametric knowledge. This survey systematically organizes the field by why tool learning is beneficial and how it is implemented across four workflow stages, while covering benchmarks, evaluation methods, challenges, and future directions. It concludes that tool learning offers a structured basis for understanding and advancing this emerging area, subject to limitations in tool accessibility, comprehensiveness, model compatibility, and computational cost.

  • Problem

    LLMs struggle with complex computations and accurate, timely information because their fixed parametric knowledge can produce plausible but incorrect or outdated responses, while tool-learning research remains fragmented.

  • Method

    The survey synthesizes tool-learning research around its benefits and a four-stage workflow: task planning, tool selection, tool calling, and response generation.

  • Results

    The survey provides a systematic account of tool-learning benefits, implementation methods, benchmarks, evaluation methods, challenges, and future directions.

  • Takeaways & Limitations

    The survey offers researchers and developers a structured foundation for understanding and further exploring tool learning with LLMs.

  • Takeaways & Limitations

    Current tool learning is constrained by limited tool accessibility and comprehensiveness; tuning-based methods require open-source LLMs and substantial computational resources.

Abstract

from arXiv · show

Recently, tool learning with large language models (LLMs) has emerged as a promising paradigm for augmenting the capabilities of LLMs to tackle highly complex problems. Despite growing attention and rapid advancements in this field, the existing literature remains fragmented and lacks systematic organization, posing barriers to entry for newcomers. This gap motivates us to conduct a comprehensive survey of existing works on tool learning with LLMs. In this survey, we focus on reviewing existing literature from the two primary aspects (1) why tool learning is beneficial and (2) how tool learning is implemented, enabling a comprehensive understanding of tool learning with LLMs. We first explore the "why" by reviewing both the benefits of tool integration and the inherent benefits of the tool learning paradigm from six specific aspects. In terms of "how", we systematically review the literature according to a taxonomy of four key stages in the tool learning workflow: task planning, tool selection, tool calling, and response generation. Additionally, we provide a detailed summary of existing benchmarks and evaluation methods, categorizing them according to their relevance to different stages. Finally, we discuss current challenges and outline potential future directions, aiming to inspire both researchers and industrial developers to further explore this emerging and promising area. We also maintain a GitHub repository to continually keep track of the relevant papers and resources in this rising area at https://github.com/quchangle1/LLM-Tool-Survey.

1 Introduction

Tool learning with LLMs addresses limitations in fixed, parametric knowledge by enabling dynamic interaction with external tools. This survey organizes the field around why tool learning is beneficial and how it is implemented, while reviewing benchmarks, evaluation methods, challenges, and future directions.

  • LLMs can produce plausible but factually incorrect or outdated responses because their knowledge is fixed and parametric.
  • Tool learning enables LLMs to interact dynamically with external tools, improving problem-solving capabilities, functional scope, and response accuracy.Examples include calculators, weather APIs, and programming interpreters.
  • The survey examines tool learning through two dimensions: why it is beneficial and how it is implemented.
  • The paper also summarizes benchmarks and evaluation methods, discusses challenges and future directions, and maintains a repository of relevant resources.
  • The survey compares tool learning with related surveys by providing focused analysis of its benefits and systematic implementation for LLMs.

2 Background

The background section defines tools broadly as external means that enhance LLMs and characterizes tool learning as effective interaction with tools to accomplish complex tasks. It also describes reassessment and alternative tool use when an initial solution fails.

  • A tool may be an external object, a computer-program function interface, or any external method that enhances LLM capabilities.
  • Tool learning is the process of enabling LLMs to interact effectively with various tools to accomplish complex tasks.
  • When a tool is required, ChatGPT can explain its tool-based problem-solving process and the rationale for its response.
  • If an initial solution fails, ChatGPT can reassess its tool selection and use an alternative tool to generate a new response.

3 Why Tool Learning?

Tool learning is important because it extends LLMs beyond pre-training and language-only limitations while supporting knowledge access, specialized expertise, automation, multimodal interaction, transparency, and robustness. These benefits arise from both integrating external tools and adopting a tool-learning paradigm.

  • Knowledge acquisition: External tools let LLMs dynamically acquire contemporary and structured information beyond their finite pre-training knowledge.Search engines provide contemporary information, while databases support structured retrieval and complex queries.
  • Expertise enhancement: Calculators and programming resources help LLMs perform complex calculations, solve equations, analyze data, and refine generated code through execution feedback.
  • Automation and efficiency: Task automation tools enable LLMs to execute external actions and automate repetitive activities such as scheduling, reminders, and email filtering.
  • Interaction enhancement: Specialized multimodal tools can improve LLMs’ perceptual capabilities when user queries vary across languages and modalities.Examples include speech recognition and image analysis.
  • Transparency and interpretability: Tool learning can make LLM operations more transparent by exposing decision-making steps, addressing the opacity of black-box LLMs.This transparency is especially relevant where response accuracy is imperative, including healthcare, aviation, and finance.
  • Robustness and adaptability: Specialized tools are proposed to reduce LLM reliance on statistical patterns in training data, supporting robustness across diverse prompts.

4 How Tool Learning?

The survey explains tool learning as a workflow with four stages—task planning, tool selection, tool calling, and response generation—and reviews recent advances at each stage. It also distinguishes one-step from iterative task-solving paradigms.

  • Four stages: Tool learning is organized into task planning, tool selection, tool calling, and response generation.
  • Two paradigms: Tool learning uses either one-step task solving or iterative task solving as its two typical paradigms.
  • Four stages: The survey reviews each workflow stage and pairs the discussion with practical GPT-4 examples to clarify how the stages address specific problems.

4.1 Overall Paradigm of Tool Learning

Tool learning is organized as a workflow that transforms a user question into tool-supported responses. The survey distinguishes one-step solving from iterative solving, where tool feedback progressively revises the plan.

  • The four workflow stages are task planning, tool selection, tool calling, and response generation.
  • Task planning analyzes user intent and decomposes a request into solvable sub-questions before tools are selected.
  • Tool selection may use a retriever to filter candidate tools or allow the LLM to choose directly from a provided list.
  • One-step task solving: One-step solving plans all subtasks upfront and generates a response from tool results without revising the plan using feedback.
  • Iterative task solving: Iterative solving adjusts subtasks progressively through repeated tool interactions and feedback, refining the plan step by step.

4.2 Task Planning

Task planning decomposes complex user intent into a minimal set of sub-questions while representing dependencies and execution order. Existing methods use prompting, retrieval, structured planning, or fine-tuning, but perfect upfront plans and feedback use remain challenging.

  • Task planning decomposes complex user questions into sub-questions and identifies dependencies and execution order.
  • Illustrative example: GPT-4 decomposes a currency-valued asset query into three sub-questions covering gold price, Amazon stock price, and USD–CNY exchange rate.
  • Tuning-free methods: Few-shot and zero-shot prompting can support planning by decomposing complex tasks into simpler sub-tasks.
  • Existing methods: Planning methods include iterative decomposition, decision-tree action spaces, task graphs, progress summaries, budget-aware planning, and branch-based tool-use examples.
  • Tuning-based methods: Fine-tuning methods enhance tool-use planning through API-call training, human feedback, planner replication, abstract reasoning chains, and generated usage examples.
  • Open challenges: Generating a perfect plan for complex issues remains difficult, and effectively using tool feedback to improve planning remains an open question.

4.3 Tool Selection

Tool selection matches decomposed sub-questions with suitable tools, using retrieval when tool libraries are large and direct LLM choice when candidate lists are manageable. Methods range from sparse and dense retrieval to prompting and fine-tuning, with unresolved concerns about hierarchy and retrieval completeness.

  • Tool selection chooses appropriate tools for decomposed sub-questions through retrieval or direct LLM selection from a provided list.
  • Illustrative example: GPT-4 selects the Metals Prices Rates API because it provides real-time updated gold-price information.
  • Retriever-based selection: Large tool libraries motivate retrievers that narrow candidates to the top-K suitable tools before LLM-based selection.
  • Retriever-based selection: Sparse retrieval uses term matching with methods such as TF-IDF and BM25, whereas dense retrieval learns query–tool semantic relationships and uses similarity measures.
  • Open challenges: Traditional retrieval may focus on semantic similarity while ignoring tool hierarchy, motivating retrievers tailored to tool-specific needs.
  • LLM-based selection: Tuning-free selection relies on prompting and mechanisms such as chain-of-thought, ReAct, depth-first search, and multi-turn reasoning.
  • LLM-based selection: Tuning-based selection can encode extensive tool knowledge but applies only to open-source LLMs and consumes substantial computational resources.

4.4 Tool Calling

Tool calling converts a selected tool and user sub-question into correctly formatted parameters, after which the tool returns information for response generation. Methods use prompting, rules, fine-tuning, or special tokens, while error handling and generalization remain important constraints.

  • Tool calling extracts parameter content and format from the user query and selected tool documentation.
  • Illustrative example: A GPT-4 example outputs the Metals Prices Rates API parameters as symbols: XAU and base: USD.
  • Tuning-free methods: Tuning-free methods use few-shot demonstrations or rules to improve parameter extraction.
  • Tuning-based methods: Tuning-based methods enhance tool calling through fine-tuning, special tool tokens, and autoregressive integration of tool use with reasoning.
  • Error handling: Tool calling requires error handling for malformed parameters, out-of-range inputs, and tool-server failures.
  • Method comparison: Tuning-based methods can perform poorly on unseen tools and face catastrophic forgetting and limited robust generalization.

4.5 Response Generation

Response generation combines tool outputs with an LLM’s internal knowledge to produce answers, with information integration methods offering more flexibility than direct insertion. The main challenges are unpredictable or lengthy outputs and the need to validate external results.

  • Response-generation role: LLMs synthesize tool outputs with internal knowledge because tool results can include text, numbers, code, videos, and images.The goal is to construct a response relevant to the user’s query rather than present raw tool results directly.
  • Response-generation role: GPT-4 combines real-time gold prices, Amazon stock prices, and exchange rates to calculate a final CNY value.The example reports a total of 941,269,495.87 CNY for 5 ounces of gold and 1 million Amazon stocks.
  • Direct insertion methods: Direct insertion replaces placeholders in generated text with tool results, making it straightforward but suitable mainly for simple outputs.Unpredictable tool outputs can affect the user experience.
  • Information integration methods: Information integration feeds tool outputs into the LLM context, while schemas, truncation, compression, and adaptive extraction address lengthy results.Truncation may lose information, whereas other methods retain only content considered useful for answering the query.
  • Risks and safeguards: External tool results require rigorous validation because manipulated outputs can cause LLMs to generate harmful or malicious responses.The survey also notes that tool results can influence responses in ways that may mitigate some model-originated bias or harmful content.

5 Benchmarks, Toolkits, and Evaluation

The survey organizes benchmarks, toolkits, and evaluation methods around the four stages of tool learning. It catalogs 33 popular benchmarks, describes task-specific resources and open-source frameworks, and links stage-specific evaluations to planning, selection, and calling abilities.

  • Overview: The survey categorizes benchmarks, toolkits, and evaluation methods by tool-learning stage to structure how methods are validated and implemented.The four stages are task planning, tool selection, tool calling, and response generation.
  • Benchmarks: 33 popular benchmarks are compiled into general and other benchmark classes, although the list may not include every available benchmark.The survey states that each benchmark evaluates distinct facets of tool learning.
  • Benchmarks: General benchmarks evaluate different stages, including tool-use awareness, planning, selection, calling, and broader tool usage.MetaTool and WTU-Eval assess whether models recognize the need for tools and appropriate tools, while ToolBench2 spans a large tool-learning dataset.
  • Benchmarks: Task-specific benchmarks cover external-tool question answering, multi-turn tool use, scientific reasoning, remote-sensing workflows, and ambiguous tool retrieval.Examples include ToolQA, ToolTalk, ToolSandbox, SciToolBench, GeoLLM-QA, and ToolLens.
  • Toolkits: Open-source resources include LangChain, Auto-GPT, BabyAGI, BMTools, and WebCPM for building workflows that connect LLMs with APIs, databases, tools, or web search.Their described uses range from autonomous task execution and tool integration to interactive web-based question answering.
  • Evaluation: Task-planning evaluation measures tool-use awareness, plan effectiveness, and plan precision through accuracy, pass rates, human evaluation, or comparison with references.These measures target whether a model identifies when an external tool is needed and forms an effective plan.
  • Evaluation: Tool-selection evaluation uses Recall, NDCG, and COMP to assess retrieved-tool relevance, ranking, and completeness.COMP checks whether the top-K selected tools contain the complete ground-truth tool set, while NDCG incorporates ranking positions.
  • Evaluation: Tool-calling evaluation checks whether generated parameters conform to tool documentation, including whether required parameters are supplied.This directly assesses the validity of requests generated for tool invocation.

6 Challenges and Future Directions

The survey identifies fragmented evaluation, tool resources, workflow coverage, real-world robustness, user data, and multimodal interaction as central challenges for tool learning with LLMs.

  • Evaluation: ToolEval improves evaluation efficiency and reproducibility but may not reflect genuine user preferences.The survey calls for evaluation covering efficiency, precision, cost, practicality, and stage-specific contributions.
  • Tool resources: Current tools are limited in accessibility, comprehensiveness, domain diversity, and description consistency.Many tools come from existing datasets or public APIs, while heterogeneous source formats hinder unified tool-learning frameworks.
  • Robustness and safety: Tool-learning research must address unavoidable noise and safety considerations in practical deployments.Existing work predominantly evaluates tool use in well-structured environments, whereas real-world scenarios introduce noise and potential attacks.
  • Unified framework: Research often focuses on one workflow stage, leaving task planning, tool selection, invocation, and response generation insufficiently integrated.The survey links this fragmentation to challenges in scalability and generality in practical scenarios.
  • Real-world data: Benchmarks mostly use LLM-generated queries rather than authentic user interactions, limiting their representation of genuine human interests and search behavior.The survey notes that no published dataset then covered authentic interactions between users and tool-augmented LLMs.
  • Multimodality: Multimodal tool learning remains underexplored despite opportunities to use images, audio, 3D, and video for better user-intent understanding.MLLM-Tool is cited as an early system that combines open-source LLMs with multimodal encoders for tool selection.

7 Conclusion

The survey reviews more than 150 papers to organize tool learning with LLMs, explaining its benefits and implementation across four workflow phases. It also categorizes stage-specific benchmarks and evaluation methods and identifies challenges and future research directions.

  • Scope and background: The survey reviews more than 150 papers and introduces foundational concepts and terminology for tool learning with LLMs.Its introductory background is intended to support newcomers to the field.
  • Why tool learning: It examines why tool learning matters by detailing benefits of tool integration and the paradigm across six aspects.The survey presents these benefits as a basis for understanding tool learning with LLMs.
  • How tool learning works: It explains how tool learning is conducted through task planning, tool selection, tool calling, and response generation.Each phase is discussed with recent research advancements.
  • Evaluation and outlook: The survey categorizes benchmarks and evaluation methods according to their relevance to the workflow stages.It provides a structured overview of evaluation protocols and highlights challenges and future directions.
Loading 2405.17935v3…