PaperlayerView exact revision on arXiv

Source-linked AI summary

Tool Learning with Foundation Models

Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang, Jing Yi, Yuzhang Zhu, Zhenning Dai, Lan Yan, Xin Cong, Yaxi Lu, Weilin Zhao, Yuxiang Huang, Junxi Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang, Cheng Yang, Tongshuang Wu, Heng Ji, Zhiyuan Liu, Maosong Sun

arXiv:2304.08354v3cs.CLcs.AIcs.LG

TL;DR

Tool learning with foundation models lacks a comprehensive account of its challenges, opportunities, and evaluation. This paper systematically reviews the field, formulates a general framework, and finds that current foundation models can effectively use tools with simple prompting.

  • Problem

    Tool learning lacks a comprehensive understanding and systematic evaluation of its key challenges, opportunities, and future directions.

  • Method

    The paper reviews tool learning, formulates a framework for instruction understanding, decomposition, adaptive reasoning, and tool selection, and evaluates 18 representative tools.

  • Results

    Current foundation models, including ChatGPT, can effectively use tools to solve tasks with simple prompting.

  • Takeaways & Limitations

    Foundation models show potential as general agents for tool learning across tasks involving tool manipulation.

  • Takeaways & Limitations

    Research remains limited in scenarios requiring multiple steps and different tools for different subtasks.

Abstract

from arXiv · show

Humans possess an extraordinary ability to create and utilize tools, allowing them to overcome physical limitations and explore new frontiers. With the advent of foundation models, AI systems have the potential to be equally adept in tool use as humans. This paradigm, i.e., tool learning with foundation models, combines the strengths of specialized tools and foundation models to achieve enhanced accuracy, efficiency, and automation in problem-solving. Despite its immense potential, there is still a lack of a comprehensive understanding of key challenges, opportunities, and future endeavors in this field. To this end, we present a systematic investigation of tool learning in this paper. We first introduce the background of tool learning, including its cognitive origins, the paradigm shift of foundation models, and the complementary roles of tools and models. Then we recapitulate existing tool learning research into tool-augmented and tool-oriented learning. We formulate a general tool learning framework: starting from understanding the user instruction, models should learn to decompose a complex task into several subtasks, dynamically adjust their plan through reasoning, and effectively conquer each sub-task by selecting appropriate tools. We also discuss how to train models for improved tool-use capabilities and facilitate the generalization in tool learning. Considering the lack of a systematic tool learning evaluation in prior works, we experiment with 18 representative tools and show the potential of current foundation models in skillfully utilizing tools. Finally, we discuss several open problems that require further investigation for tool learning. In general, we hope this paper could inspire future research in integrating tools with foundation models.

1 Introduction

Tool learning with foundation models combines specialized tools and foundation models to improve problem-solving, while requiring models to understand intents, plan, reason, and select tools. This paper develops a unified framework, reviews the field, evaluates foundation models with 18 tools, and identifies open research problems.

  • Motivation: Tools extend human capabilities by helping overcome physical limitations, enhance productivity, and solve increasingly complex tasks efficiently.Tool creation and use are presented as longstanding components of human activity.
  • Motivation: Effective tool manipulation requires understanding tool functionalities and user intents, alongside planning and reasoning for tool use.Before powerful foundation models, tool-oriented AI research was exceedingly challenging, although some basic tools could be fitted with statistical or neural models.
  • Paradigm: Tool learning combines specialized tools with foundation models to enable more accurate, efficient, autonomous, and streamlined solutions for real-world problem-solving.Foundation models contribute complex-data interpretation, human-like planning, and broad output generation, while specialized tools target specific goals.
  • Contributions: The paper formulates a general tool learning framework comprising a foundation-model controller, tool set, environment, perceiver, and human, then systematically reviews existing research.The framework is used to organize core research problems and existing solutions across tool learning.
  • Evaluation: 18 representative tools were used to evaluate foundation-model tool manipulation, showing that state-of-the-art models such as ChatGPT can solve tasks with simple prompting.The evaluation investigates both the efficacy and limitations of foundation models and supports their potential as general agents for tool learning.
  • Open Problems: The paper highlights safety, trustworthiness, governance, adversarial risks, and challenges in applying tool learning to large complex systems and high-stakes scenarios.It calls for careful consideration before deploying tool-learning models in high-stakes settings and aims to inspire further integration of tools with foundation models.

2 Background

This section frames tool learning through human tool use, a user-interface-based taxonomy of tools, the foundation-model paradigm shift, and the complementary integration of specialized tools with foundation models. It emphasizes both the persistent gap between human and model tool use and the potential for integrated systems to improve capability and robustness.

  • Human Tool Use: Human tool use is rooted in cognitive and perceptual systems and has been essential throughout evolution.Humans possess neural systems involved in grasping objects, using tools, and observing tool use.
  • Tool Categorization: The paper classifies instruction-manipulable tools by interaction mode as physical interaction-based, GUI-based, and program-based tools.These modes range from direct physical-world interaction to graphical interfaces and source-code-level access.
  • Tool Categorization: The three interaction modes vary in connectivity with the tool kernel, can intermingle, and may all be executed through foundation-model intermediary interfaces.The paper also notes that humans flexibly use different tool types for complex tasks.
  • Foundation-Model Paradigm: Pre-trained language models shifted NLP toward learning general linguistic ability and world knowledge from massive corpora, enabling a pre-train-then-fine-tune paradigm.This paradigm unified NLP tasks and achieved state-of-the-art performance on GLUE and SuperGLUE.
  • Tools and Foundation Models: Integrating foundation models with specialized tools combines model understanding and reasoning with tools’ task-specific design, enabling more complex tasks and greater robustness.The passage specifically associates tools with resistance to input perturbations and reduced risk of malicious attacks.

3 Tool Learning

This section presents a four-component framework for tool learning and explains how models plan, interact with environments, select tools, and improve their capabilities. It also identifies challenges in multi-step, multi-tool use and discusses supervised, feedback-based, interface, and meta-tool learning approaches.

  • General Framework: Tool learning is framed around four interacting components: tool set, environment, controller, and perceiver.The section introduces each component’s characteristics, functions, and interactions within the general framework.
  • Reasoning and Planning: Extrospective reasoning incrementally revises plans using execution results and real-time feedback from the environment and user.Unlike introspective reasoning, this approach conditions subsequent actions on previous outcomes and supports long-term goals requiring extensive planning.
  • Multi-Step Multi-Tool Scenarios: Multi-step multi-tool tasks remain challenging because models must decompose tasks, select suitable tools, understand tool interactions, and coordinate sequential or parallel execution.The section also highlights movement from single-agent problem-solving toward multi-agent collaboration.
  • Training Methods: Supervised behavior cloning trains perceiver-controller models and foundation models for tool-oriented tasks, with WebShop agents showing non-trivial product-purchasing performance.WebShop provides a web-based interactive environment in which agents browse and purchase products from human instructions.
  • Training Methods: Feedback-based learning enables models to assess action success or failure and iteratively update their planning strategies, but RLHF remains task-specific and vulnerable to reward-model biases.WebShop uses a hand-coded reward comparing human-bought and model-bought products to assess final-task success.
  • Interfaces and Generalization: Semantic interfaces unify tool invocation through action-name text spans, while modality composition and meta-tool learning support multimodal interaction and adaptation to unfamiliar tool-use situations.Semantic triggers include forms such as “Action:Search,” and metacognition enables models to reflect on learning and adapt new strategies.

4 Application and Experiment

The experiments evaluate foundation-model tool use across 18 representative tools and find that simple prompting usually improves performance, while some tasks remain difficult even with few-shot examples. The authors also release BMTools as an open-source platform for integrating and sharing tools.

  • Experimental setup: 18 representative tools are evaluated under No Tool, Zero-shot, and Few-shot settings, with results reported for text-davinci-003 and ChatGPT.Table 1 defines # APIs, test sets, and the three prompting settings.
  • Resource: BMTools integrates the evaluated tools into an open-source repository where users can build plugins with Python functions and integrate external tools.The repository is intended as a community platform for building and sharing tools.
  • Results: In most cases, simple prompting enables models to use tools effectively and improve task performance.The reported conclusion is based on the overall results in Table 1.
  • Results: Zero-shot tool use can reduce performance on tasks solvable with internal knowledge, including calculator and search-engine cases.The paper attributes this outcome to sub-optimal utilization of tools.
  • Results: Few-shot prompting yields satisfying API-call completion for Map, Weather, Slides, Tables, Cooking Assistant, and AI Painting among manually curated test sets.Direct comparison across tools is difficult because their evaluation setups differ; the comparison is limited to successful API-call rates.
  • Results: Few-shot prompting remains unsatisfactory for knowledge graphs, Wikipedia, online shopping, and 3D model construction because their tool usage is difficult to learn from few examples.Generating executable API parameters, such as for the knowledge-graph search_by_query API, is especially arduous.

5 Discussion

The discussion identifies governance, security, efficiency, privacy, and complexity as central challenges for tool learning, while highlighting model-specific tools, AI-created tools, personalized interaction, and digital embodiment as promising directions. It also emphasizes the need for safe, adaptive systems and evaluation of models’ capacity to create genuinely novel tools.

  • Governance and Safety: Adversaries could modify tool documentation or tools themselves, such as news APIs, to mislead foundation models into erroneous outcomes.Safe and robust systems must address the interplay between models and tools, not tool use alone.
  • Governance and Safety: Governance must determine which tools should be involved and regulate how foundation models use tools to achieve objectives.The paper presents tool selection and usage regulation as increasingly urgent because foundation models can perform broader tasks than task-specific models.
  • System Challenges: Large complex systems contain numerous components and functions, making their interaction and management substantially more flexible and complicated than limited-functionality tools.Relational databases may include over 500 knobs controlling different functions.
  • System Challenges: Foundation models struggle to memorize and master the skills, functions, syntax, and maintenance knowledge required by large complex systems.Relational databases combine built-in functions, ORM or SQL query syntax, and maintenance problems.
  • System Challenges: In real-time scenarios such as fraud detection, existing foundation models’ lengthy reasoning and planning can be intolerable when responses are expected within milliseconds.Efficiency is therefore a critical implementation metric for real systems.
  • Privacy and Personalization: Tool learning raises privacy risks because training requires sensitive user-behavior data, while personalized systems face data-extraction attacks and costly deployment constraints.The passages identify federated learning and model distillation as possible responses to privacy concerns.
  • Future Directions: Tools designed specifically for models may improve tool learning through properties such as modularity, while large code models already generate executable programs as task-specific tools.This motivates both designing tools for AI and studying tools created by AI.
  • Future Directions: Whether models can create genuinely novel tools beyond memorizing, composing, and interpolating human tools remains an open research problem requiring new evaluations.The paper links this question to assessing models’ creativity and potential contributions to tool development.

6 Conclusion

The paper studies tool learning with foundation models by tracing tool use from its cognitive origins, reviewing the foundation-model paradigm shift, and formulating a general framework grounded in existing research.

  • 6 Conclusion: The paper examines tool learning with foundation models and reviews its cognitive origins, tool categorization by user interface, and the complementary roles of tools and models.It also surveys existing tool-learning research and formulates a general tool-learning framework.

Contributions

The paper’s contributions section records author responsibilities across research organization, writing, experiments, advising, discussion, and proofreading. It also identifies section-specific drafting and experimental roles, while noting that Yujia Qin participated in all sections.

  • Research and writing: The authors organized the research and distributed drafting responsibilities across the paper’s sections, abstract, introduction, and conclusion.Yujia Qin, Shengding Hu, Yankai Lin, Zhiyuan Liu, and Maosong Sun initiated and organized the research; the passages then assign section-specific drafting roles.
  • Advising and review: Senior contributors advised the project and joined discussions, other researchers provided detailed suggestions, and designated authors proofread the whole paper.Yujia Qin participated in all sections, while Yujia Qin, Yankai Lin, and Weize Chen proofread the paper.

A Case Study

This section presents the prompts and ChatGPT model responses used for each studied tool, with implementations to be released in BMTools.

  • A Case Study: The case study provides specific prompts and model responses for each tool examined.The responses come from ChatGPT’s Mar 23, 2023 version.
  • A Case Study: The evaluated model is the ChatGPT version dated Mar 23, 2023.
  • A Case Study: Implementations for the different APIs and datasets will be made available in BMTools.

A.1 3D Models

Shape Editor constructs 3D models by embedding 2D primitives in 3D shapes, combining them, and applying transformations. Its workflow supports rendering modeled objects and executing queries involving object creation and spatial adjustment.

  • Modules: Shape Editor has three modules: shape_2d creates 2D shapes, shape_3d turns them into 3D shapes, and shape combines and transforms 3D shapes.The documented workflow normally places a 2D shape into shape_3d before using shape operations.
  • 2D primitives: The shape_2d module provides triangle, circle, rectangle, and oval primitives with coordinate- or dimension-based initialization parameters.Examples include triangle vertices, a circle center and radius, rectangle corner points, and oval parameters.
  • 3D shapes and transforms: The shape_3d module provides cylinder, donut, and sphere shapes, plus translation and rotation transformations.Cylinders use a 2D shape, a height dimension and range, and a color vector; spheres use position, radius, and color parameters.
  • Operations and queries: Shape Editor supports transform, merge, and render operations, with render saving a shape’s modeling data to data.json.The examples create and transform a red ball and blue cube, and separately adjust the distance between two objects.

A.2 Stock

The stock task requires solving market queries by composing a PRICE API with permitted arithmetic and aggregation APIs. PRICE supports daily or monthly price queries for one company, while date-specific and date-range requests use different parameters.

  • PRICE API: PRICE returns daily or monthly open, close, high, or low prices for a specified company and date or date range.Its parameters include type1 for frequency, type2 for price type, and symbol for the company.
  • PRICE API: A PRICE call uses either an exact date or the optional date_start and date_end range, leaving the fifth parameter blank for an exact-date query.Each call accepts only one company symbol.
  • Compositional APIs: MIN, MAX, AVG, and SUM aggregate lists, while MINUS, ADD, and DIVIDE operate on two data inputs.These APIs can take PRICE calls as parameters, enabling nested computations; no other APIs are allowed.
  • Query-solving procedure: Queries are solved by retrieving the relevant prices first and then applying the needed aggregation or arithmetic API.For example, finding the highest open price of Alibaba in 2020 uses a full-year date range, type2 set to open, and MAX.

A.3 Making Slides

The slide-making workflow uses API actions to create, populate, illustrate, and submit a themed PowerPoint. The demonstrated presentation traces English from its origins through Old, Middle, and Modern English using historical developments and literary examples.

  • Slide-generation workflow: The workflow creates a PowerPoint with a selected theme, adds title and content pages, optionally includes local images, and submits the completed file.Available themes include wooden, tech, flat, green, and orange; the process requires submit_file() when all steps are done.
  • Presentation content: The demonstrated 20-slide presentation covers English-language origins, major historical influences, linguistic changes, and the development of Old, Middle, and Modern English.Its outline includes Germanic tribes, Latin and French influence, the Great Vowel Shift, the printing press, and standardization.
  • Literary examples: Literary examples represent each period through Beowulf, The Canterbury Tales, and Shakespeare’s Hamlet.The examples connect Old English with Anglo-Saxon values, Middle English with pilgrims and contemporary social norms, and Modern English with revenge, madness, and mortality.
  • Influences and rationale: The presentation also highlights English’s borrowing from Latin, French, Norse, Celtic, Spanish, and other languages through colonization and globalization.It identifies historical figures and explains that studying language history supports understanding cultural evolution, literary diversity, communication, and identity.

A.4 Movie Hunter

The Movie Hunter section provides tools for filtering upcoming films and retrieving detailed movie information. Demonstrations show filtering by region, category, count, and popularity, alongside detail lookup for a named film.

  • Film filtering: coming_out_filter filters currently playing films by region, category, output count, and desired popularity, with defaults for each parameter.The demonstrated call requests Chinese drama films and the top two most wanted movies.
  • Film filtering: The demonstrated comedy query returns five upcoming Chinese films: Journey to the West, One and Only, Ride On, Godspeed, and Look At Me.These films are also identified as the five most wanted movies in the result.
  • Movie details: print_detail retrieves a named movie’s information by parsing movie pages, locating the target link, and extracting details from that link.The procedure applies parse_coming_page and parse_nowplaying_page before retrieving the selected film’s details.
  • Movie details: The Deep Sea lookup returns its genre, region, director, cast, and synopsis involving Can Xiu’s journey through a dreamy underwater world.The film is described as a Chinese mainland animated fantasy film directed by Tian Xiaopeng and starring Su Xin, Wang Tingwen, Teng Kuixing, and others.

A.5 Search Engine

The search engine tool supports multiple-choice question answering by searching Bing, loading a selected result page, and using the retrieved content to identify an answer. Demonstrations show this workflow answering questions about songs and a proposed app-ban bill.

  • Tool workflow: The assistant can call Search(query:str) to retrieve Bing’s most relevant webpage and LoadPage(idx:int) to inspect a returned page using index 1, 2, or 3.The instruction directs the model to determine needed information before calling the APIs.
  • Demonstration: song identification: For a Taylor Swift song question, the workflow searches for songs, reviews result summaries, and selects “Best Days of Your Life” as the answer.The cited observation states that Swift co-wrote Kellie Pickler’s song and gives choice 1 as the final answer.
  • Demonstration: app identification: For a question about a bipartisan bill empowering President Joe Biden to ban an app, the search workflow identifies TikTok and returns choice 3.The retrieved information describes a bill targeting technology linked to foreign adversaries such as TikTok.

A.6 Chemicals Mining

The Chemicals Mining section demonstrates compound identification and property lookup through complementary tools, including name, structure, ID, and property queries. Worked examples show retrieval of an EGFR IUPAC name and calculation of the molecular-weight change from methane to chloromethane.

  • Tool Functions: Four tools support chemical lookup: get_name returns three synonyms, get_struct maps a SMILES formula to an ID, get_id returns five possible IDs, and get_prop returns compound properties.The get_struct input should be a SMILES formula rather than a quoted string.
  • Demonstrations: The EGFR demonstration retrieves compound ID 9549299 before using its properties to obtain the IUPAC name.The reported IUPAC name is N-[3-[[6-[3-(trifluoromethyl)anilino] pyrimidin-4yl]amino]phenyl]cyclopropanecarboxamide.
  • Demonstrations: The methane substitution query compares methane and chloromethane by first identifying methane as CID 297 and chloromethane as CID 6327.The retrieved molecular formulas are CH4 and CH3Cl, respectively.
  • Demonstrations: 34.447 is the molecular-weight increase when one hydrogen in methane is replaced by chlorine, changing 16.043 for methane to 50.49 for chloromethane.These values come from the retrieved MolecularWeight properties for methane and chloromethane.

A.7 Cooking Assistant

The cooking assistant uses search, load, and navigation APIs to identify relevant recipes and answer questions from retrieved recipe information. Demonstrations show it selecting an exact or general recipe before extracting the requested cooking details or typical ingredients.

  • Tool API and procedure: The assistant accesses search, load, and back APIs, first determining needed information, then retrieving it before answering.search returns result lists, load retrieves recipe details by index, and back returns to the previous page.
  • Banana pie example: For banana pie, it searches for the recipe, loads the exact matching item, and reports a total time of 1 hrs 40 mins.The retrieved recipe lists 15 mins preparation, 25 mins cooking, and 1 hrs additional time.
  • Vegetable soup example: For vegetable soup, it loads the most general recipe and identifies baby carrots, potatoes, onion, celery, beans, cabbage, tomatoes, and green beans as typical vegetables.The recipe combines these vegetables with broth, water, herbs, and salt, then simmers them until tender for about 90 minutes.

A.8 AI Painting

The AI Painting section describes two image-manipulation tools: generating images from text prompts and replacing specified objects with new ones. It illustrates both operations through bouquet and puppy examples.

  • All images are handled as URLs, and generate(prompt) creates an image from a textual description.The prompt specifies the image to generate.
  • replace(image, origin_obj, new_obj) replaces an identified object in an image with a newly described object.The input image is a URL; origin_obj names the object to remove, while new_obj describes its replacement.
  • A bouquet prompt generates an image, after which the flowers are replaced with trees.The example calls generate("Bouquet of colorful flowers in a vase.") and then replace("out1.png", "bouquet of colorful flowers", "trees").
  • A puppy-running prompt generates an image, followed by replacing the playful puppy with a playful cat.The illustrated calls are generate('A playful puppy running through a field.') and replace(lastImage, 'a playful puppy', 'a playful cat').

A.9 Navigating Knowledge Graphs

The knowledge-graph navigation procedure combines entity and relation lookup with table-based inspection and query execution. Demonstrations show it resolving both a factual question and a multi-step relation query, including recovery when direct entity lookup fails.

  • Navigation tools: The toolkit supports finding entities by head or tail, retrieving minor information, and inspecting graph relations through structured tables.Entity inputs must be exactly one ID, such as Q42.
  • Demonstration: relation recovery: After finding Stockholm University, the procedure inspects its outgoing relations and searches the resulting table for “Concordia Salus.”The relation table includes part-of and located-in entries, but the motto lookup produces an empty DataFrame, yielding a negative answer.
  • Demonstration: relation recovery: The second demonstration concludes that “Concordia Salus” does not belong to the university associated with IPv4 routing prefix 130.237.88.0/21.The final answer follows the failed motto match after identifying Stockholm University as Q221645.

A.10 ALFWorld … A.18 Database

The examples span embodied interaction, arithmetic, weather, shopping, maps, table processing, translation, Wikipedia question answering, and database querying. Across these tasks, models follow constrained tool interfaces, compose multi-step calls, and revise actions when intermediate results are insufficient.

  • A.10 ALFWorld: ALFWorld constrains actions to a fixed command grammar while requiring agents to locate, take, transform, and place objects through environment feedback.Demonstrations show agents checking likely locations sequentially and adapting when an object is not found.
  • A.11 Calculator: Calculator tasks require comprehending each word problem, selecting the appropriate arithmetic operation, calling the calculator API, and returning the result, including 66,018 people for 65,899+119.Another demonstration computes 88/8 and returns 11 pink flower stones per bracelet.
  • A.12 Weather: Weather tasks call GetWeatherToday or ForecastWeather as needed and compare returned information with the question, such as sunset at 06:08 PM versus a 6:00pm arrival.The London example identifies partly cloudy conditions from the returned weather dictionary.
  • A.14 Map: Map tasks compose specialized APIs: DISTANCE and DIVIDE estimate travel time, SEARCH and NUM count nearby targets, and DISTANCE, DIVIDE, and ADD combine sequential route times.ROUTE returns a list of routes between two positions, while arithmetic APIs support list and scalar calculations.
  • A.15 Processing Tables: Processing-table tasks use a single python_repl_ast action to manipulate pandas dataframes through statistical, visualization, filtering, sorting, and editing APIs.Examples convert elevations to numeric before filtering above 8,500 meters and revise an Auckland query to use LIKE when exact matching fails.
  • A.16 Translation / A.17 Wikipedia: Translation uses an MT(text) API to translate input into English, while Wikipedia question answering interleaves search, lookup, disambiguation, and reasoning to reach answers such as 1,800 to 7,000 ft.The Wikipedia trace searches High Plains, disambiguates an initially unrelated page, and then finishes with the elevation range.
  • A.18 Database: Database demonstrations execute select_database_data queries, returning 100000 rows for ordering supplier comments and 394 rows for a rewritten query.The examples report the returned row counts as final command results.
Loading 2304.08354v3…