PaperlayerView exact revision on arXiv

Source-linked AI summary

GTA: A Benchmark for General Tool Agents

Jize Wang, Zerun Ma, Yining Li, Songyang Zhang, Cailian Chen, Kai Chen, Xinyi Le

arXiv:2407.08713v2cs.CLcs.AI

TL;DR

Existing tool-use evaluations often use AI-generated queries, text-only interactions, and virtual tools, leaving gaps in their coverage of real-world tasks. GTA combines human-designed implicit-step queries, executable tools, and multimodal inputs in a benchmark of 229 tasks. GPT-4 completes fewer than 50% of the tasks, while most evaluated LLMs complete less than 25%.

  • Problem

    Existing tool-use evaluations often use AI-generated queries, text-only interactions, or virtual tools, leaving real-world multimodal and end-to-end tool use underrepresented.

    Source: §1 · pp. 1–2

  • Method

    GTA evaluates 229 human-designed, implicit-step tasks with authentic multimodal inputs, executable tools, and reference tool chains.

    Source: §1 · p. 3, §3.1 · p. 4

  • Results

    GPT-4 completes fewer than 50% of GTA tasks, while most evaluated LLMs complete less than 25%.

    Source: §4.3 · p. 8, §5 · p. 10

  • Takeaways & Limitations

    GTA's evaluation identifies real-world tool-use as a challenge for current LLMs and provides findings to guide development of general tool agents.

    Source: §1 · p. 3, §5 · p. 10

    All queries are in English, and human-written queries and tool chains cost more per data piece than AI-generated counterparts.

    Source: §6 · p. 10

Abstract

from arXiv · show

Significant focus has been placed on integrating large language models (LLMs) with various tools in developing general-purpose agents. This poses a challenge to LLMs' tool-use capabilities. However, there are evident gaps between existing tool-use evaluations and real-world scenarios. Current evaluations often use AI-generated queries, single-step tasks, dummy tools, and text-only interactions, failing to reveal the agents' real-world problem-solving abilities effectively. To address this, we propose GTA, a benchmark for General Tool Agents, featuring three main aspects: (i) Real user queries: human-written queries with simple real-world objectives but implicit tool-use, requiring the LLM to reason the suitable tools and plan the solution steps. (ii) Real deployed tools: an evaluation platform equipped with tools across perception, operation, logic, and creativity categories to evaluate the agents' actual task execution performance. (iii) Real multimodal inputs: authentic image files, such as spatial scenes, web page screenshots, tables, code snippets, and printed/handwritten materials, used as the query contexts to align with real-world scenarios closely. We design 229 real-world tasks and executable tool chains to evaluate mainstream LLMs. Our findings show that real-world user queries are challenging for existing LLMs, with GPT-4 completing less than 50% of the tasks and most LLMs achieving below 25%. This evaluation reveals the bottlenecks in the tool-use capabilities of current LLMs in real-world scenarios, which provides future direction for advancing general-purpose tool agents. The code and dataset are available at https://github.com/open-compass/GTA.

1 Introduction

Existing tool-use benchmarks often rely on AI-generated prompts, text-only interactions, and virtual tools, limiting their alignment with real-world tasks. GTA combines human-designed, implicit-step queries, executable tools, and authentic multimodal inputs for evaluating tool agents.

1 Introduction

  • Prior benchmarks often use AI-generated queries, text-only interactions, or virtual tools, limiting evaluation of real-world reasoning, multimodal capability, and end-to-end task completion.AI-generated prompts may be overly brief or monotonous; virtual tools test isolated invocation steps.Source: §1 · pp. 1–2
  • GTA evaluates human-designed queries with implicit tool-use steps, requiring agents to infer suitable tools and plan multiple invocation steps.The queries have clear objectives but do not explicitly specify the tools or steps.Source: §1 · p. 2
  • The benchmark provides executable tools across perception, operation, logic, and creativity, with a detailed reference tool chain for each task.The reference chain records each tool-use step, including tool names, argument values, and return values.Source: §1 · p. 2
  • GTA pairs queries with authentic images, including spatial scenes, webpage screenshots, tables, code, and printed or handwritten materials.Agents must solve tasks using both the multimodal context and the user query.Source: §1 · p. 3
  • The benchmark evaluates tool agents on 229 real-world tasks using executable tool chains and fine-grained tool-use metrics.The evaluation platform covers perception, operation, logic, and creativity; the paper evaluates 16 LLMs.Source: §1 · p. 3

2 Related Work

Prior work integrates LLMs with tools for specialized tasks and proposes benchmarks for tool-use evaluation. GTA focuses on general tool agents, using real-deployed tools and executable chains for fine-grained evaluation in real-world scenarios.

LLM-based agents

  • Many LLM-based systems integrate specialized tools for web search, visual tasks, data analysis, machine-learning models, or smartphone interaction.Examples include WebGPT [18], Visual ChatGPT [29], DataCopilot, and AppAgent.Source: §2 · p. 3
  • Real-world environments require planning and coordination across varied tool types, challenging LLM tool-use capabilities.Earlier systems commonly focus on invoking a certain class of tools for specialized tasks.Source: §2 · pp. 3–4

Tool-use evaluations

  • Many tool-use benchmarks rely on AI-generated queries with explicit steps, while GAIA [16] targets general assistants and Osworld [30] computer environments.GTA instead targets tool agents with real-deployed tools and executable tool chains for fine-grained real-world evaluation.Source: §2 · p. 4

3 GTA Benchmark

GTA pairs multimodal, real-world queries requiring tool-based reasoning and planning with reference tool chains. This section describes the sample formulation, human query construction, executable-chain annotation, and dataset statistics.

3.1 Dataset Formulation

  • Each sample combines image files, a query, involved tools, a multi-step reference tool chain, and a final answer; the query leaves tool choices and steps implicit.Each reference-chain step records the tool, its arguments, and its return value.Source: §3.1 · p. 4

3.2 Query Construction

  • Experts create diverse initial exemplars, then annotators expand them into real-world queries while keeping the tool set fixed or slightly modified and filtering out unsuitable samples.Exemplars specify image files, queries, tools, solution steps, answers, and evidence; solution steps must exceed two.Source: §3.2 · p. 5
  • Search questions must require web search rather than internal knowledge and constrain answers to remain stable over time.Questions can specify a time frame, web page, or organization.Source: §3.2 · pp. 5–6

3.3 Tool Chain Construction

  • Three computer-science annotators manually run deployed tools for each query, record the tool chain and final answer, and discard queries when a tool fails to provide accurate recognition.This process is intended to ensure query feasibility, executable tool chains, and precise final answers.Source: §3.3 · p. 6

3.4 Dataset Statistics

  • GTA contains 229 questions, 252 images, and 14 tools; most questions use 2–3 tools and require 2–4 steps, with Perception+Logic and Perception+Operation the most frequent combinations.Questions span 156 objective, 16 subjective, and 57 image-generation queries; solution lengths range from 2 to 8 steps.Source: §3.4 · p. 6, Figure 3 · p. 6

4 Evaluation and Analysis

GTA evaluates 16 LLMs in step-by-step and end-to-end modes using fine-grained tool-use metrics. Models struggle on the benchmark, with argument prediction emerging as a key bottleneck; further analysis distinguishes model error patterns and finds that instruction fine-tuning improves some metrics but leaves argument accuracy low.

4.1 Experiment Settings

  • Step-by-step mode predicts the next reference-chain action without tool execution, while end-to-end mode calls tools dynamically and evaluates the resulting task performance.The two modes compare model outputs with reference chains versus actual tool choices and execution results.Source: §4.1 · pp. 6–7

4.2 Evaluation Metrics

  • Step-by-step metrics assess instruction following, tool choice, argument names, and answer summarization; AnsAcc measures execution-result accuracy in end-to-end mode.The evaluation also calculates tool-selection F1 scores across perception, operation, logic, and creativity categories.Source: §4.2 · p. 7
  • AnsAcc excludes image-generation queries; AnsAcc w/ ImgGen evaluates predicted image-generation parameters, while subjective-answer scoring uses the highest cosine similarity to three annotated responses.For objective answers, correctness requires all whitelist phrases and no blacklist phrases.Source: §4.2 · p. 7

4.3 Main Results

  • GPT-4 and GPT-4o each solve fewer than 50% of GTA problems, while the rest of the models solve less than 25%.The benchmark’s main results are reported in Table 4; AnsAcc reflects overall performance.Source: Table 4 · p. 7, §4.3 · p. 8
  • ArgAcc has the highest correlation with AnsAcc, and argument prediction is weak for most models.The authors describe overall performance as affected by the lowest metric and identify argument value and format correctness as relevant capabilities.Source: §4.3 · p. 8
  • Yi and Deepseek tend to call tools aggressively, Qwen conservatively, and GPT moderately; GPT has the highest final answer accuracy.GPT-4o has the most successful calls, while GPT-4 has the highest successful-call rate.Source: §4.3 · pp. 8–9
  • Most models favor either format errors or argument-format errors; Claude-3’s errors are 82.86% argument-format-related and 4.29% format errors.Table 5 reports the percentage of different error types.Source: §4.3 · p. 9, Table 5 · p. 9

4.4 Further Analysis and Exploration

  • GPT-4’s errors often involve taking no action, while Llama-3’s errors commonly involve invalid action or answer formatting; Agent-Flan-7B improves InstAcc and ToolAcc but retains low ArgAcc.The reported Llama-3 error categories include invalid JSON arguments, simultaneous tool calls, redundant text, and failure to perform the correct action.Source: §4.4 · pp. 9–10

5 Conclusion

GTA is a real-world tool-use benchmark with human-designed, step-implicit queries, multimodal context, executable tools, and fine-grained evaluation. Evaluation of 16 LLMs found the tasks challenging, including for GPT-4.

5 Conclusion

  • GPT-4 completed fewer than 50% of GTA tasks, which the authors characterize as challenging for current LLMs.The paper evaluates 16 LLMs on the benchmark.Source: §5 · p. 10
  • GTA combines real-world queries with implicit tool-use steps, multimodal inputs, and executable tools across perception, operation, logic, and creation.The benchmark also uses fine-grained metrics to evaluate tool-use capabilities.Source: §5 · p. 10
  • The authors expect GTA to support further research into identifying models’ tool-use capabilities and developing general-purpose tool agents.They also offer takeaways and suggestions for improving tool-use capability.Source: §5 · p. 10

6 Limitations

GTA has limited language diversity because all queries are in English, and creating human-written queries and tool chains costs more than using AI-generated counterparts.

6 Limitations

  • All GTA queries are in English, limiting the benchmark’s language diversity.The authors suggest adding multilingual queries to assess tool agents in non-English environments.Source: §6 · p. 10
  • Human authorship of queries and tool chains raises the cost per data item relative to AI-generated counterparts.The authors connect this choice to achieving high data quality.Source: §6 · p. 10
  • The authors identify multilingual query coverage as a possible future extension for evaluation in non-English environments.Source: §6 · p. 10

Checklist

The checklist points readers to the repository and experimental section for reproducibility details, while reporting that error bars were not applicable and that limitations were discussed.

Checklist

  • The authors report that the paper discusses its contributions, scope, and limitations.Source: Checklist · p. 14
  • The checklist directs readers to the GTA repository for code, data, instructions, and training details.It also points to Section 4.1 for compute information.Source: Checklist · p. 14
  • The checklist marks error bars as not applicable.Source: Checklist · p. 14

A Datasheet for Datasets

The datasheet describes GTA’s purpose, contents, construction, and intended use, along with its distribution and maintenance plans and details of its tool categories and query examples.

A.1 Motivation

  • GTA was created to evaluate general LLM tool use in real-world scenarios using human-written queries, executable tools, and authentic image inputs.The datasheet presents these features as bridging gaps between existing benchmarks and real-world tool-use scenarios.Source: §A.1 · p. 15

A.2 Composition

  • The dataset contains 229 instances and 252 images, with each instance including a query, image inputs, tool descriptions, a reference tool chain, and a final answer.The whole dataset is designated as a test set, and the authors note that writing errors may introduce noise.Source: §A.2 · p. 15

A.3 Collection Process

  • Queries were human-designed, while image inputs came from the internet or annotators; researchers and student annotators created the data, and images were reviewed for academic suitability.The authors say they will promptly remove images if requested by their authors.Source: §A.3 · p. 16

A.4 Preprocessing/cleaning/labeling

  • The dataset was created from scratch and manually verified; Excel and VSCode were used to create it.Source: §A.4 · p. 16

A.5 Uses

  • GTA is intended to evaluate general tool use, and its stated potential risks include image copyright and privacy concerns, harmful generated information, and malicious code.Source: §A.5 · pp. 16–17

A.6 Distribution / A.7 Maintenance

  • The dataset is released under the Apache License, and the authors plan updates based on user issues while primarily maintaining the newest version.Older versions may be kept for up to one year in specified circumstances.Source: §A.6 · p. 17, §A.7 · p. 17

B.1 Tool Definition

B.2 Examples of Three Query Types

  • GTA includes objective, subjective, and image-generation query types, with answer evaluation tailored to each type.Objective answers use whitelist and blacklist terms; subjective answers are compared by cosine similarity with three annotator responses, taking the highest score; image-generation evaluation measures argument accuracy.Source: §B.2 · pp. 18–19, Figure 7 · p. 19, Figure 8 · p. 19

C Additional Information for Data Design

The appendix documents query exemplars and their expansion, annotator instructions, and the JSON structure used to record executable tool chains.

C.1 Query Exemplars

  • Query exemplars seed new questions that preserve a tool chain while varying the scenario; examples span tasks such as circling a movie time or identifying a restaurant on a map.The appendix provides 15 query exemplars and expansion examples for most of them.Source: §C.1 · pp. 19, 20, 29

C.2 Diversified Expansion Approach

  • Each exemplar expands into six samples with unchanged tools, two with one tool replaced, and two with the tool set increased or decreased.The diversified expansion approach is illustrated in Figures 25 and 26.Source: §C.2 · p. 34, Figure 25 · p. 34, Figure 26 · p. 35

C.3 Instruction for Annotators

  • Annotators create multi-step, image-based queries and verify each tool-chain step by calling deployed tools, recording arguments and returns, and discarding queries a tool cannot solve correctly.GPT-4-generated chains serve as format examples, not as the final annotations.Source: §C.3 · pp. 36–37

C.4 Illustration of Executable Tool Chains

  • Each executable tool chain is JSON containing the involved tools, files, and a dialog among user, assistant, and tool roles.The assistant entry records the tool name and arguments, while the tool entry records its return value.Source: §C.4 · p. 38, Figure 29 · p. 38

D Additional Information for Experiments

The experimental agent uses ReAct to plan and act with deployed tools, while a prompt template specifies its interaction format.

D.1 Build an LLM-Based Agent System

  • The agent system combines an LLM, an action-and-planning schema, and tools, using ReAct and 14 AgentLego tools within Lagent and OpenCompass.Different LLMs are substituted into the Lagent framework for evaluation.Source: §D.1 · p. 39

D.2 ReAct-Style Prompts

  • Figure 30 presents the ReAct-style prompt template used by the agent system.The supplied passage identifies the template but does not describe its contents.Source: §D.2 · p. 39, Figure 30 · p. 39

D.3 Final Answer Evaluation of Subjective and Image Generation Queries

The section describes final-answer scoring for subjective and image-generation queries, using embedding similarity for subjective answers and an AnsAcc score for image answers.

D.3 Final Answer Evaluation of Subjective and Image Generation Queries

  • Subjective-answer scoring encodes predictions and ground truths with All-MPNet-Base-V2, computes cosine similarity, and considers only positive values to scale the score from 0 to 1.The formula passage accompanies this scoring description.Source: §D.3 · p. 40
  • For image-answer queries, the section formulates AnsAcc using image-generation tools in the ground-truth chain and the tools and arguments predicted by the model.The listed tools are AddText, DrawBox, TextToImage, and ImageStylization.Source: §D.3 · p. 40

D.4 Detailed Error Cases of Each Type

The appendix presents categorized error examples for two models and a same-query response comparison between two other models.

D.4 Detailed Error Cases of Each Type

  • The analysis categorizes seven primary error types in GPT-4-1106-Preview and Llama-3-8B-Instruct predictions, with detailed cases shown in Figure 31.The supplied passages do not report the relative frequency or severity of the error types.Source: §D.4 · p. 41, Figure 31 · p. 41

D.5 Comparison of Llama-2-Chat-7B and Agent-Flan-7B

  • The appendix compares Llama-2-Chat-7B and Agent-Flan-7B responses to the same query to examine instruction tuning on ReAct and JSON-format data.Figure 32 shows the response comparison; the passages do not state its outcome.Source: §D.5 · p. 42, Figure 32 · p. 42

Related papers

Loading 2407.08713v2…