Source-linked AI summary
ToolNet: Connecting Large Language Models with Massive Tools via Tool Graph
Xukun Liu, Zhiyuan Peng, Xiaoyuan Yi, Xing Xie, Lirong Xiang, Yuchen Liu, Dongkuan Xu
TL;DR
LLMs remain limited in using massive external tool libraries, while conventional in-context methods provide tools as a list and leave selection to the model. ToolNet represents tool transitions in a directed graph and lets the LLM choose among successor tools. Across its evaluations, it consistently improves performance, resists noisy tools, and achieves up to 2.6x greater token efficiency.
Problem
Method
ToolNet organizes tools as nodes in a weighted directed graph and offers the LLM successor tools based on its previous action.
Source: §3 · p. 3Results
Takeaways & Limitations
Adaptive transition weights support rapid integration of new tools and alignment with changing tool repositories.
Source: §6 · p. 8Graph construction requires costly tool-use trajectories that include multi-hop cases, whereas existing benchmarks are primarily task-specific and mostly single-tool.
Source: §7 · p. 8
Abstract
from arXiv · showhide
While achieving remarkable progress in a broad range of tasks, large language models (LLMs) remain significantly limited in properly using massive external tools. Existing in-context learning approaches simply format tools into a list of plain text descriptions and input them to LLMs, from which, LLMs generate a sequence of tool calls to solve problems step by step. Such a paradigm ignores the intrinsic dependency between tools and offloads all reasoning loads to LLMs, making them restricted to a limited number of specifically designed tools. It thus remains challenging for LLMs to operate on a library of massive tools, casting a great limitation when confronted with real-world scenarios. This paper proposes ToolNet, a plug-and-play framework that scales up the number of tools to thousands with a moderate increase in token consumption. ToolNet organizes tools into a directed graph. Each node represents a tool, and weighted edges denote tool transition. Starting from an initial tool node, an LLM navigates in the graph by iteratively choosing the next one from its successors until the task is resolved. Extensive experiments show that ToolNet can achieve impressive results in challenging multi-hop tool learning datasets and is resilient to tool failures.
1 Introduction
ToolNet addresses the difficulty of using massive tool libraries by organizing tools into a graph that guides LLM selection. The paper reports consistent performance gains, resilience to noisy tools, and lower token use.
1 Introduction
- LLM agents commonly rely on few-shot examples for narrow purposes and connect to only a small number of specially designed tools.The introduction gives Toolformer and Chameleon as examples.Source: §1 · p. 1
- Scaling to thousands of tools strains token limits and tool selection; training-based approaches can be costly and hard to adapt to updated tools.The introduction also notes that more tools can bring hallucinations and tool-calling errors.Source: §1 · p. 1
- ToolBench trajectories show sparse tool transitions, so the next tool can often be selected from a limited set of successors.The paper uses this pattern to retain token efficiency.Source: §1 · p. 2
- ToolNet scores individual tool-use steps and adjusts transition weights to restrict low-quality tools and reduce failure calls.The weights encode tool-use experience in the graph.Source: §1 · p. 2
- Across five datasets, ToolNet consistently outperforms counterparts, resists noisy-tool interference, and uses significantly fewer tokens.The datasets are SciQA, TabMWP, MATH, APIBank, and ToolBench.Source: §1 · p. 2
- ToolNet organizes tools in a weighted directed graph that guides LLM tool calls and can be updated for new tools or tasks.The graph is built from LLM-produced tool-use trajectories.Source: §1 · p. 2
2 Problem Formulation
The section formalizes tool-augmented LLM interaction as a loop of thoughts, tool actions, and environment observations, with past interactions and available tools in context. It identifies long contexts and unreliable tool search as obstacles to scaling tool use.
2 Problem Formulation
- At each step, the LLM uses the current observation and interaction history to produce a thought and tool action, then receives a new environment observation.The loop can continue until the task is resolved.Source: §2 · pp. 2–3
- The interaction history is a bounded queue of past observation, thought, and action tuples, while the available tool set is also included in the LLM input.When the queue reaches capacity, its earliest tuple is removed.Source: §2 · p. 3
- Long history and tool inputs drive token use, while semantic search can be inaccurate for massive libraries with similar tools that differ in functionality.The section says existing approaches consequently rely on a limited set of designed tools or computationally intensive instruction fine-tuning.Source: §2 · p. 3
3 ToolNet
ToolNet represents tools as a weighted directed graph, letting an LLM select from graph-constrained successors rather than repeatedly receiving the full tool set. The paper describes static and dynamic ways to construct and update this graph.
3 ToolNet
- ToolNet supplies the LLM with a policy-selected tool subset; after the first step, the available tools are the previous tool's graph successors, reducing context when the graph is sparse.The first-step candidates are selected by semantic similarity to the initial task description.Source: §3 · pp. 3–4
- Transition weights encode prior tool preferences, letting the LLM rank candidate tools instead of treating them as equally preferred.The paper contrasts this with ReAct and Reflexion, which provide tools with equal preference.Source: §3 · p. 4
3.1 Graph Construction
- A useful graph should be sparse and can downweight out-of-date tools; a fully connected graph with equal edge weights degrades ToolNet to naive ReAct.The paper considers static and dynamic graph construction, while composition, pruning, and partition are left for future exploration.Source: §3.1 · p. 4
3.2 Static Construction
- Static construction estimates tool-transition weights from completed tool-use trajectories using a 2-gram model.Possible trajectory sources include code snippets and released tool-use datasets; the paper notes concerns about the quality of LLM-generated trajectories.Source: §3.2 · p. 4
- The start node connects to tools with equal weights, while semantic retrieval can select the initial tool when the tool library is massive.This avoids relying on the start node's links to expose every tool as the initial candidate set.Source: §3.2 · p. 4
3.3 Dynamic Construction
- Dynamic construction alternates trajectory generation with LLM evaluation of each used tool, whose integer scores from −3 to 3 inform transition-weight updates.The update blends learned weights with prior weights; α controls update speed and β controls the interpolation.Source: §3.3 · p. 5
4 Experiments
Across task-specific and multi-task tool-learning datasets, ToolNet achieves strong answer quality while using fewer tokens than Reflexion in reported comparisons. Experiments also examine noisy tools, dynamic tool selection and failures, and how transition-weight formatting affects performance.
4.1 Setup
- Experiments use gpt-3.5-turbo on five datasets, with task-specific QA experiments connecting the model to 16 tools, most irrelevant to the target dataset.For those task-specific datasets, the graph starts fully connected with equal edge scores; evaluation measures answer quality and token consumption.Source: §4.1 · p. 5, §4.2.1 · p. 6
4.2.1 Evaluation on Task-Specific Datasets
- ToolNet surpasses competing approaches on SciQA and TabMWP by at least 10 percentage points in EM, while using 16.4% and 22.8% of Reflexion’s tokens on TabMWP and MATH, respectively.Reflexion is slightly better on MATH but uses substantially more tokens and reasoning steps.Source: §4.2.1 · p. 6, Table 3 · p. 6
- During dynamic graph construction, scores rise for effective task-specific tools while irrelevant tools remain approximately zero; GoogleSearch, ExecuteCode, and RunPython are preferred for SciQA, TabMWP, and MATH, respectively.The Figure 3 analysis covers these three datasets.Source: §4.2.1 · p. 6, Figure 3 · p. 7
4.2.2 Evaluation on Multi-Task Datasets with Massive Tools
- ToolNet matches Reflexion’s win rate on APIBank and ToolBench using 38.5% and 49.7% of its tokens, respectively; combining ToolNet with Reflexion yields the highest win rates on both datasets.Tool graphs were built from dataset trajectories and updated dynamically at every iteration.Source: §4.2.2 · pp. 6–7, Table 4 · p. 7
- After the primary APIBank tool crashes at iteration 50, ToolNet downweights it and activates a fallback tool with identical functionality.The authors identify tool redundancy as important for system reliability.Source: Figure 4 · p. 7, §4.2.2 · p. 7
4.2.3 Effect of Transition Weights
- ToolNet’s transition weights affect performance: removing or scaling them down to decimals yields the worst results, while scaling them up generally increases EM and reduces token consumption until performance saturates at ×10.The APIBank conditions were each repeated three times.Source: §4.2.3 · pp. 7–8, Table 5 · p. 8
5 Related Work
Related work approaches tool use through fine-tuning or prompting, but in-context methods remain less effective in complex settings with unfamiliar or numerous tools.
Fine-tuning LLMs to use tools
- Fine-tuned tool-learning models range from domain-specific systems such as WebGPT to broader tools, but data collection can be costly and models may struggle with emergent or updated tools.Examples include REALM, RETRO (Borgeaud et al., 2022), WebGPT (Nakano et al.), Toolformer (Schick et al., 2023), and ToolLLM (Qin et al., 2023).Source: §5 · p. 8
Prompting LLMs to use tools
- Prompting methods combine tool descriptions and demonstrations with planning, reasoning chains, reflection, or self-evaluation to support tool use.Examples include Plan-and-Solve (Wang et al., 2023) and Self-reflection (Shinn et al., 2023; Madaan et al., 2023; Paul et al., 2023).Source: §5 · p. 8
- In-context methods are less effective in complex scenarios where tools are unfamiliar or numerous.The surveyed prompting approaches have also informed products including ChatGPT Store and LangChain (Chase, 2023), and embodied-agent and multimodal applications.Source: §5 · p. 8
6 Conclusion
ToolNet organizes massive tool libraries as directed graphs with adaptive transition weights. Evaluations report consistent performance improvements and up to 2.6x greater token efficiency.
6 Conclusion
- ToolNet is a plug-and-play method that organizes massive tools into a directed graph for LLM use via in-context learning.Its transition weights can be initially set and continually updated, supporting rapid integration of new tools and alignment with changing tool repositories.Source: §6 · p. 8
- Up to 2.6x greater token efficiency accompanies consistent performance enhancements across task-specific and complex multi-task tool-learning datasets.The conclusion reports evaluations on both dataset types.Source: §6 · p. 8
7 Limitations and Future Work
The paper identifies constraints on graph construction and evaluation, including the need for tool-use trajectories and the use of one LLM in experiments.
7 Limitations and Future Work
- Graph construction requires tool-use trajectories, which can be costly to collect.The method also requires multi-hop tool-use cases, whereas existing benchmarks are primarily task-specific and mainly contain single-tool cases.Source: §7 · p. 8
- Experiments use only gpt-3.5-turbo because of cost considerations.The authors identify evaluation with GPT-4 and the open-source Mixtrial model as potential future exploration.Source: §7 · p. 8
A Implementation Details
The appendix gives the ToolNet prompt and describes dynamic tool evaluation, including the evaluator prompt and score-submission function.
A.1 Prompt of ToolNet
- The ToolNet prompt uses iterative thought-and-function-call steps, requires a Finish call, and instructs the model to use subfunction names rather than origin tool names.The prompt says state changes are irreversible and allows multiple tries; tool descriptions and exemplars depend on runtime setup, and the prompt may be tailored to user needs.Source: §A.1 · p. 11
A.2 Dynamic Construction
- Dynamic construction uses a Tool Evaluator to assess tool effectiveness and support adaptive tool selection.Source: §A.2 · p. 11
A.2.1 Prompt of Tool Evaluator
- The evaluator scores tools using reasoning history, assigning higher scores to helpful tools and lower scores to noisy, erroneous, or unhelpful tools.Source: §A.2.1 · p. 11
A.2.2 Function of Submiting Score
- The Evaluate function accepts tool scores as integers from -3 to 3, with higher scores indicating better performance.Its submission includes each tool's name, score, and reason.Source: §A.2.2 · pp. 11–12
A.3 Overall Example
The ToolBench example follows a playlist query through album-track lookup, album-metadata lookup, and a final response. The trace shows the selected tools and returned album and track information.
A.3 Overall Example
- The playlist query asks for tracks, names, durations, artists, and cover art for album ID 3IBcauSj5M2A6lTeffJzdv.The visualization is identified as a ToolBench example of ToolNet’s reasoning process.Source: §A.3 · p. 12, Figure 5 · p. 13
- The trace starts with album_tracks_for_spotify_data, then proceeds to album_metadata_for_spotify_data with score 100, before selecting Finish with score 100.Source: §A.3 · p. 12
- The final answer names Makina by UZI at 3 minutes and 31 seconds, while showing placeholders for tracks 2 through 10.The trace reports a successful final answer.Source: §A.3 · p. 12
B Visualization of ToolNet’s Reasoning Process
A ToolBench visualization presents ToolNet’s reasoning graph and node scores for a real-world example. The authors describe the selected tools as relevant and the model as using both problem context and graph information when invoking tools.
B Visualization of ToolNet’s Reasoning Process
- The ToolBench example visualizes the graph structure generated during reasoning and the corresponding scores associated with each node.The authors say the suggested tools are highly relevant to the task.Source: B Visualization of ToolNet’s Reasoning… · p. 13
- The authors report that the model takes account of both problem context and graph information when invoking a tool.They describe this integration as illustrating the approach’s robustness.Source: B Visualization of ToolNet’s Reasoning… · p. 13
- The language model supplies semantic understanding, while the graph provides a systematic way to explore potential solutions.The authors describe the two as working synergistically and connect their interplay to observed model progress.Source: B Visualization of ToolNet’s Reasoning… · p. 13