PaperlayerView exact revision on arXiv

Source-linked AI summary

EcoAct: Economic Agent Determines When to Register What Action

Shaokun Zhang, Jieyu Zhang, Dujian Ding, Mirian Hipolito Garcia, Ankur Mallick, Daniel Madrigal, Menglin Xia, Victor Rühle, Qingyun Wu, Chi Wang

arXiv:2411.01643v1cs.AIcs.CL

TL;DR

LLM agents commonly receive all candidate-tool information in context, even when many tools are irrelevant to a reasoning step. EcoAct integrates selective tool registration into agent reasoning, and experiments report cost reductions exceeding 50% on large-tool queries while maintaining performance.

  • Problem

    Current tool registration keeps all candidate-tool information in context across reasoning steps, although only one tool can be used per step, making registration opaque to agents.

    Source: §1 · p. 1

  • Method

    EcoAct equips agents with a tool-register meta-tool so they can select tools by name and register them as needed during reasoning.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    Cost reductions exceeding 50% on queries involving large ToolBench tools were reported across multiple models, without compromising performance.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    EcoAct reduces computational costs while maintaining comparable performance when augmenting reasoning methods on ToolBench.

    Source: §5 · p. 9

Abstract

from arXiv · show

Recent advancements have enabled Large Language Models (LLMs) to function as agents that can perform actions using external tools. This requires registering, i.e., integrating tool information into the LLM context prior to taking actions. Current methods indiscriminately incorporate all candidate tools into the agent's context and retain them across multiple reasoning steps. This process remains opaque to LLM agents and is not integrated into their reasoning procedures, leading to inefficiencies due to increased context length from irrelevant tools. To address this, we introduce EcoAct, a tool using algorithm that allows LLMs to selectively register tools as needed, optimizing context use. By integrating the tool registration process into the reasoning procedure, EcoAct reduces computational costs by over 50% in multiple steps reasoning tasks while maintaining performance, as demonstrated through extensive experiments. Moreover, it can be plugged into any reasoning pipeline with only minor modifications to the prompt, making it applicable to LLM agents now and future.

1 Introduction

EcoAct integrates tool registration into agents’ reasoning, addressing the inefficiency of processing all pre-registered tools at every step. Experiments report large cost savings while maintaining performance.

1 Introduction

  • External tools expand LLM agents’ action space for tasks including scientific research and robotic manipulation.The introduction describes tools as extending agents beyond conventional natural-language processing.Source: §1 · p. 1
  • Existing registration keeps all candidate-tool information in context across reasoning steps, although only one tool can be used per step.The paper describes registration as opaque to agents and notes that the burden grows as the number of pre-registered tools increases.Source: §1 · p. 1
  • EcoAct gives agents a tool-register meta-tool to select tools by name and register them when needed during reasoning.The agent receives full information about a selected tool for use in its skill library.Source: §1 · p. 2
  • Cost reductions exceeding 50% on queries involving large ToolBench tools were reported across multiple models, without compromising performance.The experiments also examine EcoAct with ReAct and DFSDT.Source: §1 · pp. 2–3

2 Method

The method formulates agent decisions as actions from an accessible tool set, with total cost as the quantity to minimize while preserving task performance. EcoAct makes registration a reasoning-time action, using tool names to select which tools to expose in detail.

2.1 Problem setup

  • At each decision step, the agent uses historical context and accessible tools to choose an action, defined as a call to one available tool.The action-space size equals the accessible tool-set size.Source: §2.1 · p. 3
  • The objective is to minimize total query cost while maintaining good response performance.The cost formulation counts input and output tokens, with per-token costs set by the inference provider.Source: §2.1 · p. 3

2.2 EcoAct

  • Once-for-all retrieval makes every decision process all retrieved tools, despite only one being used per step, increasing cost and latency.The paper also notes that choosing from a large, detailed tool set poses a needle-in-a-haystack challenge.Source: §2.2 · pp. 3–4
  • EcoAct initializes the agent with a tool-register meta-tool and tool names; the agent can register a named tool or invoke one already registered at each step.Registration makes the complete tool information available and adds it to the agent’s skill library.Source: §2.2 · pp. 4–5
  • Tool names account for 6.6% of tool-token usage, and EcoAct uses them as concise identifiers for selecting tools to learn in detail.The method is presented as usable across agent reasoning algorithms and especially beneficial with large toolsets.Source: §2.2 · pp. 4–5

3 Experiments

Experiments on filtered ToolBench evaluate EcoAct with ReAct and DFSDT; results show lower costs while maintaining comparable performance, alongside investigations of tool-scale effects and registration over time.

3.1 Experimental setup

  • ToolBench experiments use a filtered benchmark, AnyTool retrieval, and pass rate and monetary cost as evaluation metrics.ToolBench contains 16,464 tools; the filtered benchmark excludes non-solvable queries.Source: §3.1 · p. 5

3.2 Main results

  • 54.35% and 53.82% token savings occur in two models on Level 3 queries with more than 20 tools; EcoAct also surpasses the baseline in pass rate there.In some Level 1 cases, the extra registration call can outweigh the cost of including all tool information.Source: Figure 3 · p. 6, §3.2 · p. 6
  • EcoAct significantly reduces ReAct costs on ToolBench while maintaining comparable performance across aggregated subsets.The comparison reports pass rate and cost per query across different models.Source: Table 1 · p. 5, §3.2 · p. 6

3.3.1 Extension to complex reasoning strategy

  • With DFSDT, EcoAct reduces costs while maintaining comparable performance on GPT-4o; multi-trace reasoning has higher per-query costs than ReAct.The paper attributes DFSDT's higher costs to increased token use and reasoning steps.Source: Table 2 · p. 6, §3.3.1 · p. 7

3.3.2 Skill-library evolution

  • Across the studied large-tool subsets, registered tools make up approximately 30% of those available, with registrations occurring throughout problem solving.The authors interpret the remaining tools as mostly redundant and describe possible self-correction through later registrations.Source: §3.3.2 · p. 7, Figure 4 · p. 7

3.4 Ablations

The ablations compare registering one tool versus several at a time, and using tool names alone versus names and descriptions. Multiple-tool registration yielded only minor cost savings alongside lower performance, while adding descriptions raised costs without noticeable performance gains.

3.4.1 Single-Tool vs. multiple-tool registration

  • Performance fell 5.9% on G2-instruction and 7.9% on G3-instruction with multi-tool registration, despite minor cost savings.The experiments used EcoAct with ReAct on GPT-4o; multiple-tool registration also performed worse than standard ReAct.Source: §3.4.1 · p. 8, Table 3 · p. 8
  • Multi-tool savings came from fewer repeated LLM calls, but larger registered pools increased action complexity and risked error propagation.Repeated calls otherwise re-input the entire conversation history; the authors attribute the performance decline potentially to greedy registration of multiple tools.Source: §3.4.1 · pp. 7–8

3.4.2 Tool names vs. descriptions information in tool registration

  • 31.3% in G2-instruction and 65.5% in G3-instruction: adding tool descriptions raised costs without noticeable performance gains.The tested variant supplied both tool names and descriptions during EcoAct registration.Source: §3.4.2 · p. 8, Table 4 · p. 8

4 Related Works

Prior work studies tool retrieval and reasoning algorithms for LLM agents, but these lines of work do not integrate tool registration with reasoning. EcoAct is presented as a way to register necessary tools during that reasoning process.

Background

  • LLM agents use external tools to interact with environments and solve subtasks, including scientific research and robotic manipulation (Bran et al., 2023; Ghafarollahi & Buehler, 2024; Ahn et al., 2022; Huang et al., 2023).The passage also situates this work in broader research on LLM-based agents (Xi et al., 2023; Wu et al., 2023b; Peng et al., 2023; Shridhar et al., 2020; Song et al., 2024; Wu et al., 2024; Zhang et al., 2023a; Ma et al., 2024).Source: §4 · p. 8

Tool retrieval

Reasoning algorithms

  • ReAct interleaves reasoning traces and tool actions (Yao et al., 2022), while DFSDT expands a decision-tree search space (Qin et al., 2023); neither integrates tool registration with reasoning.The passage contrasts these methods with EcoAct, which integrates registration into reasoning so agents can autonomously register necessary tools.Source: §4 · p. 9

5 Conclusion

EcoAct integrates tool registration into LLM agents’ intrinsic reasoning processes. On ToolBench, it reduces computational costs while maintaining comparable performance.

Conclusion

  • EcoAct initializes agents with a meta-tool that selectively registers tools by name during reasoning, reducing computational costs while maintaining comparable performance on ToolBench.The approach was evaluated by augmenting various reasoning methods.Source: §5 · p. 9

A More Details about ToolBench

ToolBench is a large-scale tool-use benchmark built around real-world APIs. Its generated tasks cover single- and multi-tool use and were judged diverse through human evaluation.

Dataset overview

Task construction

  • ChatGPT-generated queries cover diverse tasks involving the APIs, including both single-tool and multi-tool use.The benchmark authors used human evaluation to assess the generated instructions.Source: A More Details about ToolBench · p. 13

Evaluation

  • Human evaluation found that the generated instructions exhibit high diversity, reflecting a wide range of practical applications.The passage notes that several studies have adopted ToolBench as a standard evaluation tool (Du et al., 2024; Ye et al.).Source: A More Details about ToolBench · p. 13

A.1 Subsets information

ToolBench instructions are grouped into three tool-use complexity levels and subdivided by what is unseen. Because EcoAct does not rely on training data, the distinctions among the latter subsets are minimal.

  • G1, G2, and G3 represent single-tool instructions, intracategory multi-tool instructions, and intra-collection multi-tool instructions, respectively.Source: §A.1 · p. 13
  • Each level has Instruction, Tool, and Category subsets, covering unseen instructions, unseen tools from a seen category, and tools from an unseen category.Source: §A.1 · p. 13
  • The distinctions among these subsets are minimal for EcoAct because the algorithm does not rely on training data.Source: §A.1 · p. 13

A.2 Evaluation protocol

The evaluation uses a two-stage pass-rate protocol, with manual query review to address inflated scores from non-solvable queries; monetary cost is calculated using Microsoft Azure pricing.

  • ToolBench first classifies API candidates as solvable or non-solvable, then judges whether solutions to solvable queries are solved or unsolved.The first-stage solvability assessment uses GPT-4 in this paper.Source: §A.2 · p. 13
  • Manual review retains only resolvable queries to mitigate artificially high pass rates when many queries are judged non-solvable.Source: §A.2 · p. 13
  • Monetary cost is computed using the corresponding Microsoft Azure pricing.Source: §A.2 · p. 13

B More Implementation Details

Implementation settings include a 64-tool AnyTool candidate pool, a 24-step reasoning limit, and ToolBench hyperparameters when unspecified.

C Prompts

The appendix prompts specify EcoAct’s tool-selection interaction and provide criteria for evaluating answer resolution and task solvability.

C.1 Prompt design for EcoAct

  • C.1's EcoAct prompt requires function selection before tool use and a final Finish call.Source: Table 5

C.2 tool register

  • C.2's function_register lets the agent choose a useful function by name, then provides its detailed information for possible use.Source: §C.2 · p. 14
  • C.2 requires the selected function's name as a string parameter named function_name.Source: §C.2 · p. 14

C.3 Prompt for ReAct/DSFDT

  • Table 6 is titled “Prompt for Creating ReAct/DSFDT.”Source: Table 6 · p. 15

C.4.1 Prompt template for verifying whether the query has been resolved

  • C.4.1's check_answer_status marks non-positive responses unsolved, insufficient positive responses unsure, and sufficiently assessable ones solved or unsolved.Source: §C.4.1 · p. 15
  • C.4.1's parse_answer_status marks answers unsolved for tool errors or invalid final answers, unsure when validity cannot be verified, and solved when successful calls resolve the query.Source: §C.4.1 · p. 16

C.4.2 Prompt template for verifying whether the query is solvable

  • C.4.2 treats invalid or underspecified queries as unsolvable, uncertain cases as unsure, and tasks supported by available tools as solvable.Source: §C.4.2 · p. 16
Loading 2411.01643v1…