Source-linked AI summary
Tool Learning with Foundation Models
Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang, Jing Yi, Yuzhang Zhu, Zhenning Dai, Lan Yan, Xin Cong, Yaxi Lu, Weilin Zhao, Yuxiang Huang, Junxi Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang, Cheng Yang, Tongshuang Wu, Heng Ji, Zhiyuan Liu, Maosong Sun
TL;DR
Tool learning with foundation models lacks a comprehensive account of its challenges, opportunities, and directions. This paper systematically reviews the field, proposes a general framework, and evaluates 18 representative tools, finding that state-of-the-art models can use tools effectively with simple prompting.
Problem
Tool learning with foundation models lacks a comprehensive understanding of its key challenges, opportunities, and future directions.
Source: §1 · p. 5Method
The paper systematically reviews tool learning, formulates a framework spanning instruction understanding, task decomposition, adaptive reasoning, tool selection, training, and generalization, and evaluates 18 representative tools.
Source: §1 · p. 5, §6 · p. 34Results
State-of-the-art foundation models can effectively use tools to solve tasks with simple prompting across experiments involving 18 representative tools.
Source: §1 · p. 5Takeaways & Limitations
The findings highlight the potential of foundation models to serve as general agents for tool learning.
Source: §1 · p. 5Research on using different tools for different subtasks in multi-step, multi-tool scenarios remains limited.
Source: §3.2.2 · p. 16
Abstract
from arXiv · showhide
Humans possess an extraordinary ability to create and utilize tools, allowing them to overcome physical limitations and explore new frontiers. With the advent of foundation models, AI systems have the potential to be equally adept in tool use as humans. This paradigm, i.e., tool learning with foundation models, combines the strengths of specialized tools and foundation models to achieve enhanced accuracy, efficiency, and automation in problem-solving. Despite its immense potential, there is still a lack of a comprehensive understanding of key challenges, opportunities, and future endeavors in this field. To this end, we present a systematic investigation of tool learning in this paper. We first introduce the background of tool learning, including its cognitive origins, the paradigm shift of foundation models, and the complementary roles of tools and models. Then we recapitulate existing tool learning research into tool-augmented and tool-oriented learning. We formulate a general tool learning framework: starting from understanding the user instruction, models should learn to decompose a complex task into several subtasks, dynamically adjust their plan through reasoning, and effectively conquer each sub-task by selecting appropriate tools. We also discuss how to train models for improved tool-use capabilities and facilitate the generalization in tool learning. Considering the lack of a systematic tool learning evaluation in prior works, we experiment with 18 representative tools and show the potential of current foundation models in skillfully utilizing tools. Finally, we discuss several open problems that require further investigation for tool learning. In general, we hope this paper could inspire future research in integrating tools with foundation models.
1 Introduction
The paper presents tool learning with foundation models as a paradigm combining specialized tools with foundation models to improve accuracy, efficiency, and autonomy in problem-solving. It systematically reviews the field, formulates a general framework, evaluates tool use, and identifies open research challenges.
Motivation and foundations
- Tools extend human capabilities, while foundation models make AI tool learning practicable through improved semantic understanding, reasoning, decision-making, and grounded interaction.Earlier AI systems struggled with complex tool operations, large decision spaces, stability, and generalization across tools.Source: §1 · p. 4
Tool learning paradigm
- Tool learning combines foundation models’ broad understanding and planning with specialized tools to automate sophisticated procedures with greater accuracy, efficiency, and autonomy.The paper frames this combination as tool learning with foundation models (Figure 1).Source: §1 · pp. 4–5, Figure 1 · p. 4
Scope and contributions
- The paper provides a systematic investigation of tool learning to characterize its key challenges, opportunities, current progress, and future directions.It also reviews the cognitive origins of tool use, tool categories, the foundation-model paradigm shift, and the complementary roles of tools and models.Source: §1 · p. 5
General tool learning framework
- The proposed framework requires models to understand user intent and tool functionality, decompose tasks, plan and reason dynamically, and select appropriate tools for each subtask.The framework includes a controller, tool set, environment, perceiver, and human; training is organized around demonstrations or feedback, while unified interfaces support generalization to new tools and situations.Source: §1 · p. 5
Evaluation
- Experiments with 18 representative tools show that state-of-the-art foundation models such as ChatGPT can effectively solve tool-use tasks with simple prompting.The results support using foundation models as general agents for tool learning while exposing their efficacy and limitations in tool manipulation.Source: §1 · p. 5
Open problems and future directions
- The paper identifies open problems spanning safety, large complex systems, tool creation, personalization, embodiment, knowledge conflicts, machine intelligence, and scientific discovery.It emphasizes careful consideration before deploying tool-learning models in high-stakes scenarios and aims to inspire further integration of tools with foundation models.Source: §1 · pp. 5–6
2 Background
The background traces tool use from human cognition and physical interaction to conceptual tools, then frames foundation models as natural-language interfaces for integrating diverse tools. It highlights complementary benefits: tools provide specialized, current, interpretable, and robust execution, while models provide understanding, planning, reasoning, and accessibility.
2.1 Cognitive Origins of Tool Use
- Human tool use relies on causal and technical reasoning, interaction with the external world, and feedback from observation, communication, and hands-on practice.Humans create complex tools, harness natural forces, and transform actions into mechanical effects, distinguishing their tool behavior from other animals.Source: §2.1 · p. 6
2.2 Tool Categorization: A User-Interface Perspective
- Tools are categorized by user-interface proximity into physical interaction-based, GUI-based, and program-based tools, whose interfaces can be unified through intermediary mechanisms.Physical tools affect the environment directly, GUI tools simplify visual interaction but limit flexibility, and program-based tools maximize customizability while requiring greater technical expertise.Source: §2.2 · p. 7, §2.3 · p. 8
2.3 Paradigm Shift of Foundation Models
- Foundation models unify tasks through pretraining, prompting, and natural-language interfaces, extending language-based capabilities toward tool-mediated tasks beyond purely linguistic work.PLMs learn general linguistic ability and world knowledge from massive corpora, while prompts can elicit task execution, reasoning, and generalization without parameter updates.Source: §2.3 · p. 8, §2.4 · p. 9
2.4 Complementary Roles of Tools and Foundation Models
- Tool learning decomposes complex tasks into natural-language sub-actions and executable instructions, enabling models to translate user intent into operations on specialized tools.This approach targets tasks such as slide generation, CAD modeling, and calendar-based scheduling, while supporting applications from personal assistants to space exploration.Source: §2.4 · p. 9
- Integrating tools with foundation models combines specialized, current, interpretable, and robust execution with models’ broad knowledge, reasoning, planning, and user-intent understanding.Tools mitigate memorization limits and hallucination, add domain expertise, improve transparency and resistance to perturbations, while models improve decision-making, interaction, and access for novice users.Source: §2.4 · pp. 9–10
3 Tool Learning
This section presents tool learning as a structured interaction among models, tools, environments, and users, covering intent and tool understanding, reasoning-based planning, and multi-step tool use. It also examines training through demonstrations and feedback, along with generalization via abstracted tool knowledge, unified interfaces, and adaptive learning strategies.
3.1 Components of Tool Learning
- Tool learning is organized around four interacting components—tool set, environment, controller, and perceiver—that jointly connect user instructions, tool execution, and feedback.The controller plans and executes tool actions, while the perceiver summarizes user and environmental feedback for subsequent decisions.Source: §3.1 · p. 10, §3 · p. 10
3.1.1 Understanding the Components
- The tool set provides heterogeneous functionalities, environments execute tools and return results, controllers plan tool use, and perceivers process multimodal user and environmental feedback.Virtual environments are cheaper and more replicable but may cause overfitting and poor generalization, whereas real environments are more realistic but costlier and harder to access.Source: §3.1.1 · p. 10, §3.1 · p. 11
3.1.2 Connecting the Components
- At each round, the controller selects a tool and generates actions from the query, summarized feedback, and history, then iterates execution and feedback until the task is fulfilled.Plans may include reasoning traces, and tool selection can be separated from deciding the actions executed with the selected tool.Source: §3.1.2 · pp. 11–12
3.2.1 Understanding Intent and Tools
- Successful tool learning requires understanding both the user’s underlying intent and each tool’s functionality, linking natural-language requests to appropriate tools and operations.Intent understanding faces vague, ambiguous, diverse, and personalized instructions, while tool understanding concerns what tools do and how to use their inputs and outputs.Source: §3.2.1 · pp. 12–13
- Zero-shot prompts describe tool functionality and interfaces, whereas few-shot prompts provide usage demonstrations; prompting is flexible but constrained by model capability and context length.Tool retrieval can reduce the number of tools placed in context, while fine-tuning reduces the need for tool definitions but requires annotations and retraining after updates.Source: §3.2.1 · pp. 13–14, Figure 4 · p. 13
3.2.2 Planning with Reasoning
- Reasoning enables controllers to decompose complex tasks into sequenced sub-tasks and select tools, with introspective plans contrasted against extrospective plans that incorporate execution feedback.Extrospective reasoning is better suited to complex tasks because iterative user and environment feedback supports adaptation to intermediate results and long-term goals.Source: §3.2.2 · pp. 14–16
- Multi-step multi-tool learning remains underexplored: models must understand tool interactions, choose sequential or parallel execution, and support coordination among multiple agents.Existing efforts include ReAct (Yao et al., 2022b), MM-ReAct (Yang et al., 2023), and Auto-GPT4, while substantial practical challenges remain.Source: §3.2.2 · p. 16
3.3 Training Models for Improved Tool Learning
- Training-based tool learning optimizes foundation models with supervision, using guidance from humans or environments through demonstration or exploration rather than relying only on frozen-model prompting.The section categorizes training strategies into streams motivated by how infants learn tools from adult demonstrations or their own exploration.Source: §3.3 · p. 16
- The training framework combines learning from concrete demonstrations with reinforcement learning from environmental or human feedback, while emphasizing generalization because the tool set is potentially massive.The paper identifies human- or model-annotated demonstrations and feedback-based learning as complementary training routes, and treats generalization as necessary for practical tool learning.Source: §3.3 · p. 17, Figure 6 · p. 17
3.3.1 Learning from Demonstrations
- Learning from demonstrations progresses from supervised human annotations to semi-supervised pseudo-labels and self-supervised bootstrapping, reducing annotation requirements while improving tool-use capabilities.Behavior cloning trains models to imitate expert actions; WebGPT surpassed human experts in search manipulation, WebShop achieved non-trivial purchasing performance, and Toolformer improved GPT-J’s tool use.Source: §3.3.1 · pp. 17–18
3.3.2 Learning from Feedback
- Learning from feedback lets agents use environmental consequences and human preferences to revise tool selection and actions through reinforcement learning, but RLHF remains task-specific and bias-prone.Environment feedback may be result-based or intermediate; human feedback may be explicit or implicit, while RLHF reduces annotation demands by learning a reward model.Source: §3.3.2 · pp. 18–19
3.3.3 Generalizable Tool Learning
- Generalizable tool learning depends on abstraction and unified interfaces that transfer knowledge across tools, complemented by meta-tool and curriculum learning for task-specific adaptation.Candidate interfaces include semantic, GUI, and programming protocols; programming interfaces support complex control flow and explicit API calls, while interface choice should match model capabilities.Source: §3.3.3 · pp. 19–21
4 Application and Experiment
The section evaluates tool learning across 18 representative tools using text-davinci-003 and ChatGPT, finding that simple prompting often improves performance but that effectiveness depends on tool complexity and prompting quality. The authors also release BMTools to support tool integration and future research.
4.1 Evaluated Tools
- The evaluation covers 18 representative tools spanning translation, calculation, search, web interaction, data processing, embodied tasks, creative generation, scientific queries, and database access, using existing or curated test sets.The experiments compare no-tool, zero-shot, and few-shot settings when feasible, and evaluate manually curated tasks primarily through API-call traces.Source: §4 · p. 21, §4.2 · pp. 24–25
4.2 Experiments
- Simple prompting usually enables models to use tools effectively and improve task performance, while few-shot prompting consistently outperforms no-tool use; zero-shot tool use can sometimes hurt performance.These conclusions are reported across the evaluated tools in Table 1, with tool use especially beneficial when models employ APIs effectively.Source: Table 1 · p. 23, §4.2 · p. 25
- BMTools integrates the evaluated tools into an open-source repository where users can build plugins with Python functions and incorporate external tools.The repository is intended to facilitate future research and community tool sharing.Source: §4.2 · p. 24
- Few-shot prompting yields satisfying API-call completion rates for Map, Weather, Slides, Tables, Cooking Assistant, and AI Painting, which the authors characterize as less challenging tools.Both ChatGPT and text-davinci-003 can use these tools proficiently despite not being directly fine-tuned on them.Source: §4.2 · p. 25
- Few-shot prompting remains insufficient for Knowledge Graphs, Wikipedia, online shopping, and 3D model construction, especially when tools require executable code as API parameters.The authors argue that these difficulties demonstrate the need to train foundation models specifically for tool use.Source: §4.1 · p. 22, §4.2 · pp. 24–25
5 Discussion
The discussion examines central challenges in tool learning, including safety, governance, scalability, privacy, personalization, embodiment, and conflicting knowledge. It also outlines opportunities and open questions involving AI-designed tools, proactive systems, source verification, model architecture, evaluation, ethics, physical safety, and scientific discovery.
5.1 Safe and Trustworthy Tool Learning
- Tool learning must secure both foundation models and incorporated tools against adversarial manipulation, govern access to potentially harmful tools, and enforce usage constraints.Attackers may target model prompts, training data, documentation, APIs, or tools themselves; governance must consider which tools models may access and how they use them.Source: §5.1 · p. 26
5.2 Tool Learning for Large Complex Systems
- Large complex systems require system-specific learning with documentation or code, carefully designed prompts, checking layers, efficient reasoning, and privacy-preserving training methods.Proposed measures include SMT-based validation, input filtering, zero-prompting hints, high-speed knowledge caches, federated learning, and model distillation.Source: §5.2 · p. 27
5.3 From Tool User to Tool Maker: AI’s Evolutionary Role
- Foundation models could become tool makers by generating programs and composing existing tools, while model-oriented tools should emphasize modularity and AI-specific input and output formats.Whether models can create genuinely novel tools beyond memorizing, composing, or interpolating human-made tools remains an open question requiring new benchmarks and creativity-focused techniques.Source: §5.3 · p. 28, §5.4 · p. 29
5.4 From General Intelligence to Personalized Intelligence
- Personalized tool learning must integrate heterogeneous user information and adapt tool planning and calls to individual preferences, while proactive systems require fail-safe risk mitigation.Privacy-preserving approaches such as edge computing and federated learning are proposed because personalization can expose sensitive user data and cloud deployment requires uploading private information.Source: §5.4 · p. 29, §5.5 · p. 30
5.5 Tool Learning and Embodied Learning
- Tool learning can provide digital embodiment through language-mediated interaction with APIs and other agents, offering accessible, cognitively demanding, and scalable environments for embodied learning.Future work should incorporate embodied tools into simulated environments and investigate automatically generated assets or tools to improve agent capabilities.Source: §5.5 · p. 30, §5.6 · p. 31
5.6 Knowledge Conflicts in Tool Augmentation
- Knowledge augmentation must address conflicts between memorized model knowledge and information obtained from tools because such conflicts threaten generation and planning reliability in domains such as medical assistance and legal advice.Prior approaches use retrieval, search engines, calculators, simulators, Python interpreters, and scratchpads to augment foundation models, but the supplied discussion identifies conflict types and solutions as subsequent topics.Source: §5.6 · pp. 31–32
- Models must detect and resolve conflicts between internal knowledge and tool responses, as well as among tools differing in credibility, bias, or implementation.Conflicts can arise from outdated model parameters, false pre-training knowledge, misleading tool outputs, and inconsistent tool results.Source: §5.6 · p. 32
- ChatGPT can correct beliefs using retrieved information and identify conflicts across sources, but existing approaches often assume one reliable augmented source.The paper calls for models to verify source reliability, explain which knowledge they used, and clarify how it shaped the final response.Source: §5.6 · p. 32, Figure 9 · p. 33, §5.7 · p. 33
5.7 Open Problems
- Future systems must balance modular architectures that rely on external tools against unified foundation models that internalize capabilities such as translation.The open question is where models should fall on the spectrum between flexible tool integration and self-contained functionality.Source: §5.7 · p. 33
- Tool-use performance could serve as a practical gauge of machine intelligence because it tests task accomplishment beyond memorization and aligns with real-world assistance and collaboration.Such evaluation better reflects practical intelligence and human judgments of intelligence.Source: §5.7 · p. 33
- Ethical human-model collaboration requires safeguards that protect human dignity, agency, compensation, working conditions, transparency, and human rights.The paper recommends boundaries on delegated tasks and collaboration among researchers, ethicists, policymakers, and other stakeholders.Source: §5.7 · pp. 33–34
- Physical-tool access creates risks from erroneous or malicious actions, requiring trustworthy systems with uncertainty estimation, fail-safe strategies, and continuous monitoring.Applications include drones, robots, sensors, autonomous driving, agriculture, and smart homes, where failures could cause property damage, financial losses, or threats.Source: §5.7 · p. 34
- Tool learning could advance scientific discovery by enabling models to use software and experimental platforms, but heterogeneous scientific data and physical-experiment safety remain major challenges.Examples include MATLAB-based analysis and simulation, synthetic robots, and systems that design, plan, and execute catalyzed cross-coupling experiments.Source: §5.7 · p. 34
6 Conclusion
This paper systematically investigates tool learning with foundation models, covering its origins, framework, research challenges, and future directions. It aims to facilitate future research integrating tools with foundation models.
Contributions
- The paper presents a systematic investigation of tool learning, reviewing its origins, foundation-model paradigm shift, complementary tool–model roles, existing research, and a general learning framework.The framework emphasizes bridging user intents with appropriate tools, planning through foundation-model reasoning, training strategies, and generalization.Source: §6 · p. 34
Future directions
- It identifies future research topics including safe and trustworthy tool learning, large complex systems, AI tool creation, personalization, embodied learning, and knowledge conflict in tool augmentation.These topics define directions for advancing tool learning and integrating tools with foundation models.Source: §6 · p. 34
Contributions
The section attributes the paper’s organization, drafting, experiments, appendix preparation, software development, advising, and proofreading across the author team and contributors.
Authorship and drafting
- The authors distributed responsibility for organizing the research and drafting the paper’s sections, abstract, introduction, and conclusion.Yujia Qin, Shengding Hu, Yankai Lin, Zhiyuan Liu, and Maosong Sun initiated and organized the research; Yujia Qin participated in all sections.Source: Contributions · p. 35
Experiments and appendix
- The named authors conducted the § 4 experiments across 17 tools and tasks, drafted appendix A, organized the experiments, proofread appendix A, and participated in human evaluation for some experiments.The experiments covered 3D models, translation, maps and stocks, making slides, movie search, knowledge graphs, AI painting, search, calculation, chemicals, ALFWorld, weather, shopping, tables, cooking, Wikipedia, and databases.Source: Contributions · p. 35
Project support
- The project also credits contributors for building the first BMTools version, advising and discussing the work, providing detailed suggestions, and proofreading the full paper.Several senior and external contributors advised the project or supplied feedback, while designated authors proofread the whole paper.Source: Contributions · p. 35
A Case Study
This section provides ChatGPT’s prompts and model responses for each tool studied in § 4, with implementations for the APIs and datasets to be released in BMTools.
- The case study documents prompts and ChatGPT responses for each § 4 tool, using the Mar 23, 2023 model version, while implementations will be available in BMTools.The implementations cover different APIs and datasets.Source: A Case Study · p. 50
A.1 3D Models
Shape Editor constructs 3D models by embedding 2D primitives in 3D shapes, combining and transforming them, then rendering the result as modeling data. Its documented examples create colored objects and adjust their spatial relationships through traced actions.
A.1 3D Models
- Shape Editor uses shape_2d for planar primitives, shape_3d for 3D shapes and transformations, and shape for combining or transforming completed models.The 2D module provides triangles, circles, rectangles, and ovals; the 3D module provides cylinders, donuts, spheres, translations, and rotations.Source: §A.1 · p. 50
- The documented 3D primitives accept geometric parameters such as height ranges, rotation directions, centers, radii, offsets, datum vectors, angles, and color vectors.Cylinders and donuts are built from 2D shapes, spheres use a center and radius, and translations and rotations specify spatial transformations.Source: §A.1 · p. 50
- The shape module exposes transform, merge, and render operations, with rendering saving a model’s modeling data to data.json.Transform applies a transformation to a shape, while merge combines two shapes into one.Source: §A.1 · p. 50
- Example queries create a red ball and blue cube, then increase the distance between two objects by creating the objects and applying transformations.Both traces describe separate actions for creating the ball and cube and applying transformations.Source: §A.1 · p. 51
A.2 Stock
The Stock task requires answering market queries by composing a PRICE API with permitted arithmetic and aggregation APIs. PRICE supports daily or monthly values for one company over an exact date or range, while other APIs compute list-based or pairwise results.
PRICE API
- PRICE returns a company’s daily or monthly open, close, high, low, or volume for a specified date or date range.The call uses type1, type2, and symbol, with either date or the optional date_start and date_end parameters.Source: §A.2 · p. 52
Call format
- Each API call places parameters inside parentheses after the API name, and a single PRICE call can query only one company.For a date range, date_start and date_end are supplied; for a single date, date is supplied and the fifth parameter is blank.Source: §A.2 · p. 52
Computation APIs
- MIN, MAX, AVG, and SUM operate on lists, whereas MINUS, ADD, and DIVIDE operate on two data values.PRICE calls may be nested as inputs to these APIs, and no other APIs are allowed.Source: §A.2 · p. 52
Query execution
- A highest-price query is solved by retrieving each company’s open price with PRICE and then applying MAX to the returned values.The demonstrated Alibaba query similarly uses a full-2020 date range with type2 set to open, followed by MAX.Source: §A.2 · p. 52
A.3 Making Slides
The slide-making task uses a structured API workflow to create themed PowerPoint files, add text and image slides, and submit the completed presentation. The example demonstrates this workflow for a 20-slide history of English presentation spanning major periods, influences, linguistic changes, and literature.
A.3 Making Slides
- The workflow creates a PowerPoint with a selected theme, adds title, text, and text-image pages, and submits the completed file.Available functions include create_file, add_first_page, add_text_page, add_text_image_page, and submit_file; get_image supplies local images for image slides.Source: §A.3 · p. 53
- The example uses get_image before adding literature-focused slides that combine bullet points with locally retrieved images.Image examples cover Beowulf, The Canterbury Tales, William Shakespeare, and literature from multiple English-language periods.Source: §A.3 · pp. 53–56
- The demonstration constructs a presentation on English-language history using a wooden theme and slides on origins, Old English, Middle English, and Modern English.The presentation includes Beowulf, The Canterbury Tales, Hamlet, and historical influences such as Latin, French, the Norman Conquest, and the printing press.Source: §A.3 · pp. 54–55
- The presentation also covers other-language influence, major linguistic changes, important historical figures, cross-period literature examples, and a concluding rationale for studying language history.The stated rationale includes understanding language’s cultural impact, appreciating English literature, improving communication, and deepening understanding of language and identity.Source: §A.3 · pp. 55, 57
A.4 Movie Hunter
Movie Hunter supports filtering upcoming films by region, category, count, and popularity, and retrieving detailed movie information through linked page parsing. Demonstrations show it returns ranked movie lists and concise descriptions with cast, genre, director, and plot.
Tool operation
- Movie filtering accepts region, category, output count, and popularity-sorting parameters, while detail retrieval parses movie pages, locates the target link, and extracts information.Defaults for filtering are region “all,” category “all,” outNum 100, and SortByWant false.Source: §A.4 · p. 58
Movie details
- The detail function describes “Deep Sea” as a Chinese mainland animated fantasy film directed by Tian Xiaopeng and starring Su Xin, Wang Tingwen, and Teng Kuixing.Its synopsis follows Can Xiu, who enters a dreamy underwater world and begins a unique journey of life.Source: §A.4 · p. 58
A.5 Search Engine
The search-engine task trains assistants to answer multiple-choice questions by planning information needs, selecting between search and page-loading APIs, and responding with concise, format-constrained thoughts, actions, or answers. Demonstrations show this workflow recovering answers from search results and loaded page content.
Task instruction
- The assistant must identify needed information, call either Search or LoadPage one API at a time, and produce only one succinct thought or action/answer per response.Responses must begin with “Thought:”, “Action:”, or “Answer:” and remain under one line.Source: §A.5 · p. 59
Demonstration example
- A Taylor Swift demonstration searches for relevant songs, loads the most promising page, and answers that “Best Days of Your life” was written by Taylor Swift.The example illustrates selecting the first relevant result before extracting supporting information from the loaded page.Source: §A.5 · p. 59
- A current-events demonstration searches for a bipartisan bill, identifies the most relevant result, loads its content, and concludes that the app was TikTok.The loaded page states that the bill could regulate or ban technology linked to foreign adversaries such as TikTok.Source: §A.5 · p. 59
A.6 Chemicals Mining
The chemicals-mining toolset maps compound names, IDs, and SMILES to identifiers, synonyms, structures, and properties, enabling multi-step chemical queries. Demonstrations show it retrieving an EGFR IUPAC name and calculating the molecular-weight change from methane to chloromethane.
- The toolset supports compound lookup by name, ID, or SMILES and returns synonyms, identifiers, structures, or compound properties.Available functions include get_name, get_struct, get_id, and get_prop.Source: §A.6 · p. 60
- The EGFR demonstration chains ID retrieval with property lookup to produce the compound’s IUPAC name.The retrieved ID is 9549299, and the returned properties include the IUPAC name N-[3-[[6-[3-(trifluoromethyl)anilino]pyrimidin-4yl]amino]phenyl]cyclopropanecarboxamide.Source: §A.6 · p. 60
- 34.447 is the molecular-weight increase when methane is converted to chloromethane by replacing one hydrogen atom with chlorine.Methane has molecular weight 16.043, whereas chloromethane has molecular weight 50.49.Source: §A.6 · p. 60
A.7 Cooking Assistant
The cooking assistant answers recipe questions by reasoning about needed information, using search and load APIs one call at a time, then responding succinctly. Demonstrations show it retrieves preparation times and identifies typical soup ingredients from recipe details.
- The assistant must identify needed information, call search, load, or back one API at a time, and return only a concise Thought, Action, or Answer.Its response must contain one thought or action/answer and remain no more than one line.Source: §A.7 · p. 61
- For a banana-pie query, it searches recipes, loads the exact match, and reports a total preparation time of 1 hrs 40 mins.The demonstration uses search(banana pie) and then loads item 1, Banana Pie, before answering.Source: §A.7 · p. 61
- For vegetable soup, it loads a general recipe and identifies baby carrots, potatoes, onion, celery, beans, cabbage, tomatoes, and green beans among the typical ingredients.The returned recipe information also lists chicken broth, vegetable stock, water, basil, sage, thyme, and salt.Source: §A.7 · p. 61
A.8 AI Painting
The AI Painting tool supports text-to-image generation and object replacement in images. Inputs and outputs are represented as image URLs, enabling sequential creation and editing workflows.
AI Painting
- The generate(prompt) function creates an image from a textual description, with the prompt specifying the desired image.The image input is a text prompt, and outputs are represented as image URLs.Source: §A.8 · p. 62
- The replace(image, origin_obj, new_obj) function replaces a described object in an image with a newly described object.The image argument identifies the source image, while origin_obj and new_obj specify the object to remove and its replacement.Source: §A.8 · p. 62
- A demonstrated workflow first generates a bouquet image, then replaces its flowers with trees using the generated output.The example calls generate("Bouquet of colorful flowers in a vase.") followed by replace("out1.png", "bouquet of colorful flowers", "trees").Source: §A.8 · p. 62
A.9 Navigating Knowledge Graphs
The knowledge-graph navigation toolkit supports entity and relation identification, graph traversal, SPARQL querying, and filtering of result tables. Demonstrations show it can answer multi-step factual questions, including both successful retrieval and correction when an input is misclassified.
Tool operations
- The toolkit combines seven operations for identifying entities or relations, traversing graph neighborhoods, querying with SPARQL, and filtering the latest result table.Inputs have strict formats, including exact entity IDs, relation IDs, valid tuples, or values.Source: §A.9 · p. 63
Demonstration examples
- In the cited records, identifier Q42 is labeled Douglas Adams, and the date-of-birth observation associated with that record reports 1952-03-11, corresponding to 11 March 1952.Source: §A.9 · p. 63
- When the IP address was not found as an entity, the procedure reinterpreted “IPv4 routing prefix” as relation P3761 and used a SPARQL query to identify Stockholm University.The failed entity lookup triggered relation lookup and query construction with the recovered relation ID.Source: §A.9 · p. 64
- Filtering Stockholm University’s retrieved properties for “Concordia Salus” returned an empty table, supporting the final negative answer.The system checked the university’s outgoing-property table before concluding that the motto did not belong to it.Source: §A.9 · p. 64
A.10 ALFWorld … A.18 Database
This section presents tool-use workflows across embodied interaction, information lookup, arithmetic, shopping, map reasoning, data manipulation, translation, question answering, and database querying, emphasizing constrained actions and iterative selection and application of specialized operations.
A.10 ALFWorld
- ALFWorld requires grammatical environment actions, locating and taking objects before applying operations, and recovering from failed commands through stepwise search and verification.Demonstrations show the agent searching candidate locations, taking a tomato, cooling it, opening the microwave, and successfully placing it inside.Source: §A.10 · pp. 64–65
A.11 Calculator
- Calculator tasks require interpreting each word problem, selecting the appropriate arithmetic operation, calling the calculator API, and returning the computed answer.Examples produce 11 from 88/8 and 66,018 from 65,899+119.Source: §A.11 · p. 66
A.12 Weather
- Weather tasks constrain the assistant to one succinct thought and one API action or answer at a time, using current-weather or forecast calls before responding.The demonstrations report partly cloudy conditions in London and infer that arrival at 6:00pm precedes the 06:08 PM sunset.Source: §A.12 · p. 66
A.13 Online Shopping
- Online shopping requires searching for matching products, checking attributes and prices, selecting required variants, and clicking Buy Now.Examples identify a $10.99 three-ounce bright-citrus deodorant and a $10.95 blue toothbrush before selecting their matching options.Source: §A.13 · pp. 67–68
A.14 Map
- Map tasks combine distance, search, route, and arithmetic APIs through explicit multi-step plans, such as dividing distances by speed and adding segment times.The examples also use SEARCH followed by NUM to count nearby coffee shops.Source: §A.14 · p. 68
A.15 Processing Tables
- Processing Tables exposes Python-based tools for dataframe statistics, visualization, filtering, sorting, aggregation, editing, and structural manipulation through a single python_repl_ast action.The demonstration converts elevations to numeric values and filters rows above 8,500 meters, while another task sorts by Term Expires ascending.Source: §A.15 · pp. 69–70
- Processing tables uses a filter_df API with SQL-like predicates, revising an exact-location filter to LIKE to retrieve facilities whose locations contain Auckland.The resulting rows include Auckland Prison, Mount Eden Corrections Facility, and Auckland Region Women's Corrections Facility.Source: §A.15 · p. 71
A.16 Translation
- Translation tasks instruct the model to insert an MT API call for text requiring English translation, producing the answer “Russian Southern Federal District.”The example demonstrates the MT call within a broader question-answering prompt that asks for the answer in English.Source: §A.16 · p. 71
A.17 Wikipedia
- Wikipedia question answering chains search, lookup, disambiguation, and follow-up search, yielding the High Plains elevation range of 1,800 to 7,000 ft.The trace resolves an ambiguous High Plains page by disambiguating related entities and selecting High Plains (United States).Source: §A.17 · p. 72
A.18 Database
- Database workflows translate natural-language descriptions into SQL, may rewrite the SQL query before execution, and return 100000 rows for supplier comments and 394 rows for a filtered join.The filtered join uses partsupp.availqty = 6331, part.type > ’LARGE POLISHED NICKEL’, and part.retailprice < 1758.76, with rewriting shown before retrieval.Source: §A.18 · pp. 74–75
Related papers
This paper cites
- Attention Is All You Need2017
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2018
- Language Models are Few-Shot Learners2020
- Training language models to follow instructions with human feedback2022
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer2019
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks2020
Cited by
- $\Delta\tau$-Bench: Pushing Agents Toward a Moving Capability2026
- Data-Driven Vision-Language Models for Remote Sensing: A Survey2026
- Reasoning-Search-Augmented LLMs: An Operational Taxonomy, Benchmark Analysis and Controlled Evidence Synthesis2026
- Tool-Augmented Large Language Models for Reliable Medical Calculation from Patient Notes2026
- REVEAL: Reverse Engineering Agentic Latents from Frontier-Model Traces2026
- It is Not Yet Another Tool: Creating and Deploying an Agentic AI Companion in a Security Operations Center2026