PaperlayerView exact revision on arXiv

Source-linked AI summary

AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls

Yu Du, Fangyun Wei, Hongyang Zhang

arXiv:2402.04253v1cs.CL

TL;DR

AnyTool addresses the challenge of using more than 16,000 APIs and the artificially high pass rates produced by an existing evaluation protocol. It combines hierarchical retrieval, query solving, self-reflection, and a revised evaluation protocol, achieving state-of-the-art performance across various datasets.

  • Problem

    Using large API pools for user queries requires effective tool selection, while the existing evaluation protocol can artificially inflate pass rates by treating non-solvable queries as solved.

  • Method

    AnyTool uses a GPT-4 function-calling agent with a hierarchical API retriever, solver, and self-reflection mechanism over more than 16,000 APIs.

  • Results

    AnyTool achieves state-of-the-art performance and surpasses strong baselines such as ToolLLM and a tool-utilization GPT-4 variant across various datasets.

  • Takeaways & Limitations

    The work provides an agent and revised evaluation protocol, including the AnyToolBench benchmark, for more realistic large-scale API utilization experiments.

  • Takeaways & Limitations

    The prior evaluation protocol can label queries with unrelated API candidates as non-solvable, inflating the pass rate despite the queries remaining unsolved.

Abstract

from arXiv · show

We introduce AnyTool, a large language model agent designed to revolutionize the utilization of a vast array of tools in addressing user queries. We utilize over 16,000 APIs from Rapid API, operating under the assumption that a subset of these APIs could potentially resolve the queries. AnyTool primarily incorporates three elements: an API retriever with a hierarchical structure, a solver aimed at resolving user queries using a selected set of API candidates, and a self-reflection mechanism, which re-activates AnyTool if the initial solution proves impracticable. AnyTool is powered by the function calling feature of GPT-4, eliminating the need for training external modules. We also revisit the evaluation protocol introduced by previous works and identify a limitation in this protocol that leads to an artificially high pass rate. By revising the evaluation protocol to better reflect practical application scenarios, we introduce an additional benchmark, termed AnyToolBench. Experiments across various datasets demonstrate the superiority of our AnyTool over strong baselines such as ToolLLM and a GPT-4 variant tailored for tool utilization. For instance, AnyTool outperforms ToolLLM by +35.4% in terms of average pass rate on ToolBench. Code will be available at https://github.com/dyabel/AnyTool.

1. Introduction

Humanity’s tools have evolved from simple implements to complex technologies, and recent advances in large language models mark a new stage in that progression.

  • Human tools have progressed from simple stone implements to complex machines and other advanced technologies.
  • Recent advances in large language models are presented as a pivotal development in this ongoing technological evolution.
  • The broader trajectory is characterized by the pursuit of discovery, innovation, and increased production efficiency through improved tools.

Solver

AnyTool combines GPT-4-based hierarchical API retrieval, query solving, and self-reflection to use more than 16,000 APIs while addressing evaluation weaknesses in realistic scenarios.

  • Solver: More than 16,000 APIs are targeted for tool-assisted query answering, using retrieval followed by solution generation.The design first selects relevant API candidates and then uses them to formulate answers.
  • Solver: AnyTool uses a hierarchical API retriever, solver, and self-reflection mechanism to resolve queries with large API pools.The retriever organizes agents across meta-agent, category-agent, and tool-agent levels.
  • Solver: AnyTool requires no trained external modules beyond GPT-4’s function-calling capability, unlike ToolLLM’s trained API retriever.
  • Solver: The hierarchical retriever reduces each agent’s search scope and addresses LLM context-length constraints through Rapid API’s organization.The structure applies a divide-and-conquer arrangement across categories, tools, and individual APIs.
  • Solver: Self-reflection evaluates an initial solution and reactivates the system with failure reasons and historical context when the solution is impractical.This closed-loop process is intended to avoid oversearching simple queries while enabling deeper search for complex ones.
  • Solver: Up to 20% pass-rate improvement is reported across datasets with only 4–6 self-reflection iterations.
  • Solver: The revised evaluation protocol bypasses candidate-solvability screening and directly evaluates whether the proposed solution resolves the query.The revision addresses inflated pass rates caused by classifying unrelated API candidates as non-solvable.
  • Solver: AnyToolBench complements ToolBench, and AnyTool surpasses ToolLLM and a tool-use-tailored GPT-4 variant across various datasets.

2. Related Works

Prior work uses tools to reduce factual errors in LLM responses, while API retrieval and self-reflection remain important but unresolved challenges for reliable tool use.

  • Tool use can help mitigate factual errors that LLMs often make, especially for precise numbers and specialized knowledge.
  • Existing API-retrieval approaches use pretrained or fine-tuned embedding models but may retrieve inaccurate candidates and lack feedback mechanisms.
  • Self-reflection research spans alignment, inference-time evaluation, self-play, and self-rewarding models, but reported negative results leave API-calling use unresolved.

3. Preliminaries

This section describes GPT-4 function calling and formulates tool-use evaluation over more than 16K APIs, while identifying and revising a pass-rate protocol that can be artificially inflated.

  • 3.1. Function Calling: GPT-4 function calling iteratively selects functions, receives their outputs in historical context, and stops when it activates a designated Finish Function.Functions expose linguistic descriptions of purposes, parameters, outputs, and interpretations rather than implementation details.
  • 3.2. Problem Formulation and Evaluation: AnyTool targets user-query resolution with an API pool containing over 16K RapidAPI APIs, and GPT-4 evaluates proposed solutions with 96.5% correlation to human judgments.The evaluation processes both the query and proposed solution to assess whether the solution addresses the query.
  • 3.2. Problem Formulation and Evaluation: ToolLLM’s two-phase protocol first labels retrieved candidates solvable or non-solvable, then evaluates solutions only for queries labeled solvable.Its pass rate is defined as (Non-solvable + Solved) divided by (Non-solvable + Solved + Unsolved).
  • 3.2. Problem Formulation and Evaluation: Randomly selected, irrelevant API candidates can produce a 99.0% pass rate because GPT-4 labels queries non-solvable even when they remain unsolved.This exposes a significant flaw in the original evaluation protocol.
  • 3.2. Problem Formulation and Evaluation: The revised protocol evaluates the agent’s proposed solution directly and manually retains only ToolBench queries resolvable with specific APIs from the API pool.This bypasses the candidate-solvability phase to better align evaluation with real-world scenarios.

4. AnyTool

AnyTool combines a hierarchical API retriever, a function-calling solver, and self-reflection to search a large API pool and revise failed solutions. Its reflection process expands candidates selectively, removes irrelevant APIs, and reactivates unfinished agents and the solver.

  • 4. AnyTool: AnyTool uses a hierarchical retriever, solver, and self-reflection loop to create API candidates, attempt query resolution, and revisit unresolved queries.The system relies on GPT-4 function calling rather than training external modules.
  • 4.1. API Retriever: RapidAPI’s category, tool, and API tiers structure the retriever’s divide-and-conquer search, reducing each agent’s search scope and context-length demands.A meta-agent creates category agents, which create tool agents that add potentially relevant APIs to a shared candidate pool.
  • 4.1. API Retriever: Agents interactively call managed functions in parallel, maintain independent historical contexts, and stop searching when a tool agent verifies that the request is solvable.Agents marked Finished are excluded from later self-reflection.
  • 4.2. Solver: The solver uses candidate APIs with either DFSDT or CoT, iterating through function calls until it gives a solution or gives up with reasons and irrelevant API names.Self-reflection is triggered when GPT-4 rejects a proposed solution or when the solver gives up.
  • 4.3. Self-Reflection Mechanism: Self-reflection reactivates unfinished retriever agents from tool to category to meta-agent order, expands the candidate pool, removes APIs identified as irrelevant, and reruns the solver with cleaned context.The mechanism can repeat until a termination condition is met.

5. Experiments

Experiments evaluate AnyTool on filtered ToolBench and AnyToolBench using GPT-4-based pass-rate judgments, with GPT-4 showing 96.5% alignment with human evaluators. AnyTool outperforms comparison systems, while ablations support the contributions of hierarchical retrieval, self-reflection, and search configuration.

  • 5.1. Setup: Experiments use filtered ToolBench’s six subsets and a 400-instance AnyToolBench, with pass rate assessed by GPT-4-32K.Non-solvable ToolBench queries are manually excluded before evaluation.
  • 5.1. Setup: GPT-4 judgments align with human evaluations at 96.5%, compared with 73.9% for GPT-3.5, motivating GPT-4 as the evaluation judge.The alignment analysis uses samples from ToolBench’s G1-I subset.
  • 5.2. Main Results: AnyTool exceeds original ToolLLM and GPT-4 with reference APIs by 32.6 and 19.3 average-pass-rate points on filtered ToolBench.The comparison spans six filtered ToolBench subsets.
  • 5.2. Main Results: AnyToolBench evaluates agents using the entire API pool without reference APIs, making API retrieval essential and comparisons with retriever-free systems impractical.AnyTool shows consistent improvements over a strong ToolLLM variant and GPT-4 with a plain-agent retriever.
  • 5.3. Ablation Studies: Removing hierarchical retrieval or self-reflection harms performance, while replacing DFSDT with CoT lowers pass rates by 8.4 and 2.9 points.The ablations compare flat retrieval, no self-reflection, and CoT without DFSDT backtracking.
  • 5.3. Ablation Studies: Larger API pools improve success, candidate pools of 64 nearly saturate performance, and tool-agent capacity trades off search space against recall.Managing too many tools can overlook relevant APIs, whereas managing too few can reduce recall.

6. Conclusion

AnyTool is an agent for handling realistic queries with more than 16,000 APIs, combining hierarchical retrieval, solving, and self-reflection. The paper also points toward improved API organization and locally deployable API-focused LLMs.

  • AnyTool is designed to harness more than 16,000 APIs for realistic user inquiries.
  • The system combines a hierarchical API retriever, a solver, and a self-reflection mechanism.
  • The authors identify optimizing API organization and developing an open-source LLM for API utilization as future directions.

Impact Statements

The paper’s evaluation does not establish AnyTool’s performance in extremely complex scenarios and remains dependent on GPT-4’s function-calling capabilities.

  • AnyTool’s performance in extremely complex scenarios remains unverified because appropriate datasets are unavailable.
  • Because AnyTool relies on GPT-4 function calling, GPT-4’s capabilities affect the feasibility of generated solutions.
  • The reported scope therefore excludes demonstrated performance on extremely complex scenarios.

A.1. More Implementation Details of AnyTool

The implementation constrains search and computation with explicit API-call and token budgets, while the appendix documents the functions assigned to each agent type.

  • A.1. More Implementation Details of AnyTool: A.1. More Implementation Details of AnyTool limits DFSDT to 10 API calls and AnyTool to 200,000 tokens.The token budget covers the meta-agent, tool agents, category agents, solver, and self-reflection mechanism.
  • A.1. More Implementation Details of AnyTool: The 200,000-token limit covers token consumption across all major AnyTool components, including retrieval, solving, and self-reflection.
  • A.2. Detailed Function List: A.2. Detailed Function List records the function list for each type of agent in Table 8.

A.3. Results on the Original ToolBench

On the original ToolBench, AnyTool outperforms ToolLLM variants, while comparisons with plain-agent and AutoGen-RAG GPT-4 retrievers examine whether added retrieval information closes the gap. The reported plain-agent results attribute AnyTool’s advantage to its hierarchical structure.

  • A.3. Results on the Original ToolBench: AnyTool outperforms all ToolLLM variants on the original, unfiltered ToolBench evaluation.The evaluation tests all queries, including non-solvable ones, using pass rate.
  • A.4. GPT-4 with Various Plain Agents: Adding tool and API descriptions or detailed API information to plain GPT-4 agents yields only marginal performance improvements, while AnyTool remains superior.
  • A.4. GPT-4 with Various Plain Agents: The plain-agent comparison attributes AnyTool’s superior performance to its hierarchical structure.
  • A.5. GPT-4 with Various AutoGen-RAG Agents: The AutoGen-RAG GPT-4 variant retrieves relevant text segments by embedding similarity and then uses GPT-4 to extract API candidates.The described setup uses 1,000-token segments and can use either all-mpnet-base-v2 or text-embedding-ada-002 embeddings.

A.6. Consumption Analysis

The paper reports AnyTool’s resource consumption and details how it filters ToolBench queries, constructs AnyToolBench, and implements retrieval, solving, and self-reflection. A case study illustrates repeated agent reactivation and solver retries until resolution or a reflection limit.

  • A.6. Consumption Analysis: 13.5 × 104 tokens, 14.1 API candidates, 43.3 OpenAI API calls, and 4.6 self-reflections are consumed per query on average, with processing time infeasible because GPT-4 responses are unstable.These statistics summarize resource usage across datasets; Table 12 presents dataset-level values.
  • A.7. Filtering Process for ToolBench: ToolBench filtering removes queries lacking essential information, containing fake parameters, specifying a particular API, or being too broad to evaluate.Queries that specify a specific API are also problematic because alternative APIs may make resolution difficult to determine.
  • A.8. Construction of AnyToolBench: AnyToolBench is constructed by letting GPT-4 explore the API pool, add selected APIs to a candidate pool, generate parameters and queries from actual responses, and verify reference solutions.The verification process is intended to ensure that every constructed query is solvable.
  • B.1. Bootstrap Prompt for the API Retriever: The API retriever uses a meta-agent, category agents, and tool agents, reflecting a three-tier hierarchy that narrows API search through specialized roles.Bootstrap prompts for these agent types are provided separately.
  • B.2. Bootstrap Prompt for the Solver: The solver can give up or restart with a reason, while self-reflection reactivates tool, category, and meta-agents sequentially before retrying with an updated API candidate pool.The solver’s task description must be updated as the candidate pool changes.
  • C. Case Study: Self-reflection repeats the retrieval-and-solving cycle until the query is judged solved or the maximum number of reflections is reached, and not all agents are reactivated each time.Figure 7 presents a case study of this process.
Loading 2402.04253v1…