PaperlayerView exact revision on arXiv

Source-linked AI summary

ACEBench: Who Wins the Match Point in Tool Usage?

Chen Chen, Xinlong Hao, Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Yuefeng Huang, Wulong Liu, Xinzhi Wang, Defu Lian, Baoqun Yin, Yasheng Wang, Wu Liu

arXiv:2501.12851v8cs.CL

TL;DR

Existing tool-use benchmarks offer limited evaluation of realistic multi-turn dialogue and fine-grained tool use. ACEBench evaluates models across normal, special, and agent scenarios; experiments indicate that models fine-tuned on specific tool-use datasets struggle with complex or imperfect instructions, while code capabilities enhance tool-use performance.

  • Problem

    Existing tool-use benchmarks lack evaluation of real-world multi-turn dialogue and fine-grained assessment of tool use.

    Source: §1 · p. 1

  • Method

    ACEBench evaluates tool use across normal scenarios, imperfect instructions, and manually abstracted real-world multi-turn, multi-step agent scenarios.

    Source: §1 · p. 2

  • Results

    Experiments indicate that models fine-tuned on specific tool-use datasets struggle to generalize to complex or imperfect instructions, while code capabilities enhance tool-use performance.

    Source: §5 · p. 8

  • Takeaways & Limitations

    ACEBench provides deeper insights into the tool-use abilities of various models.

    Source: §5 · p. 9

    ACEBench’s test data may differ from real-world applications, and manually constructed Agent scenarios limit evaluation diversity and coverage.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated significant potential in decision-making and reasoning, particularly when integrated with various tools to effectively solve complex problems. However, existing benchmarks for evaluating LLMs' tool usage face several limitations: (1) limited evaluation scenarios, often lacking assessments in real multi-turn dialogue contexts; (2) narrow evaluation dimensions, with insufficient detailed assessments of how LLMs use tools; and (3) reliance on LLMs or real API executions for evaluation, which introduces significant overhead. To address these challenges, we introduce ACEBench, a comprehensive benchmark for assessing tool usage in LLMs. ACEBench categorizes data into three primary types based on evaluation methodology: Normal, Special, and Agent. "Normal" evaluates tool usage in basic scenarios; "Special" evaluates tool usage in situations with ambiguous or incomplete instructions; "Agent" evaluates tool usage through multi-agent interactions to simulate real-world, multi-turn dialogues. We conducted extensive experiments using ACEBench, analyzing various LLMs in-depth and providing a more granular examination of error causes across different data types.

1 Introduction

ACEBench is proposed to address gaps in tool-use benchmarks by covering basic, imperfect-instruction, and interactive multi-turn scenarios. It combines fine-grained evaluation with an automated sandbox-based system and experimental validation.

1 Introduction

  • Existing tool-use benchmarks lack realistic multi-turn dialogue, fine-grained and special-case assessment, and low-cost evaluation methods (Guo et al., 2024).The introduction notes that some multi-turn dialogues use predefined content combinations and that many benchmarks rely on advanced LLMs for evaluation.Source: §1 · p. 1
  • ACEBench covers Normal scenarios, imperfect instructions in Special cases, and real-world multi-turn, multi-step tool invocation in Agent scenarios.Normal includes single- and multi-turn dialogues, personalized data, and atomic-level evaluation; Special includes incomplete or incorrectly formatted parameters and irrelevant questions.Source: §1 · p. 2
  • ACEBench aims to provide comprehensive tool-use evaluation across varied scenarios, finer-grained perspectives, imperfect instructions, and more stable metrics.The introduction states that these categories cover most LLM tool-use scenarios.Source: §1 · p. 2
  • The authors build an end-to-end automated evaluation system and a sandbox scheme for multi-turn, multi-step tool invocation based on real-world scenario abstraction.Source: §1 · p. 2
  • Extensive experiments are presented as demonstrating more comprehensive analysis, greater distinction, and clearer evaluation of LLM tool use.Source: §1 · p. 2

2 Related Works

Existing tool-use benchmarks have gaps in scenario breadth, realistic multi-turn evaluation, and evaluation cost. ACEBench broadens scenario coverage and simulates realistic dialogue with automated assessment to reduce costs and improve scalability.

2 Related Works

  • The growing use of LLM tool capabilities in real-world applications makes comprehensive evaluation of their performance and effectiveness increasingly necessary.Some benchmarks rely on LLMs for evaluation, bringing high costs and unstable operation.Source: §2 · p. 2
  • Stable ToolBench incurs costly large-model evaluation (Guo et al., 2024); BFCL and HammerBench use fixed-content dialogues (Wang et al., 2024a), while τ-Bench covers two scenarios (Yao et al., 2024).The paper says fixed-content dialogue construction does not capture adaptive real-world interactions and that narrow scenario coverage limits generalizability.Source: §2 · p. 2
  • ACEBench expands evaluation across tool-use scenarios and simulates realistic multi-turn dialogues with end-to-end automated assessment, reducing costs and improving scalability.The paper says its comparative analysis against recent benchmarks demonstrates ACEBench’s effectiveness in addressing these limitations.Source: §2 · pp. 2–3

3 ACEBench

ACEBench contains 2,000 entries across linguistically parallel Chinese and English versions, and evaluates Normal, Special, and Agent tool use through distinct procedures and an overall weighted accuracy.

3.1 Dataset

  • ACEBench’s Chinese and English versions have equal data-type distributions and comprise 2,000 annotated entries.Source: §3.1 · p. 3

3.1.1 Data Construction

  • Normal and Special data are LLM-generated, while experts construct Agent data using curated scenarios, a stateful sandbox, and iterative question annotation.Normal and Special generation uses synthetic APIs guided by real APIs, a hierarchical API context tree, and dialogue pipelines that select three to six candidate APIs per instance.Source: §3.1.1 · pp. 3–4

3.1.2 Multi-Stage Data Verification

  • ACEBench applies automated checks and LLM-assisted human review, followed by two-expert review, third-expert consolidation, re-evaluation, and three optimization rounds.Automated inspection checks API-definition clarity, call executability, dialogue accuracy, and sample consistency; model-based verification detects semantic errors using voting.Source: §3.1.2 · p. 4

3.1.3 Data Analysis

  • The benchmark spans 8 API domains, 68 sub-domains, and 4,538 APIs, with dialogue lengths from 1 to 8 turns.Its Normal, Special, and Agent categories cover simple single-turn calls, multi-step interactions with users and environments, and infeasible tool-call cases; samples also vary in API argument counts.Source: §3.1.3 · pp. 4–5, Figure 2 · p. 4, Figure 3 · p. 4

3.2.1 Normal Evaluation

  • Normal Data accuracy is binary, based on AST-parsed comparison of model function calls with the ground truth or any valid candidate answer.Source: §3.2.1 · p. 5

3.2.2 Special Evaluation

  • Special Data accuracy is binary and measures whether models identify missing or erroneous parameters and task-function mismatches.Source: §3.2.2 · p. 5

3.2.3 Agent Evaluation

  • Agent evaluation uses GPT-4o as a user simulator and reports binary End-to-End Accuracy plus Process Accuracy, the matched-call fraction n/m.End-to-End Accuracy requires exact matching of all target instance attributes; n is the degree of match between actual and ideal function-call processes, and m is the ideal process length.Source: §3.2.3 · p. 5

3.2.4 Overall Accuracy

  • Overall Accuracy is a weighted sum of Normal, Special, and Agent accuracies, with weights determined by the square roots of their sample sizes.Source: §3.2.4 · p. 5

4 Experiments

Experiments assess closed-source, open-source, and tool-learning-enhanced LLMs on ACEBench, with results reported for Chinese and English datasets.

4.1 Main results and analysis

  • Closed-source models such as the GPT-4 series remain overall best, while Qwen2.5-Coder-32B-Instruct, Qwen2.5-72B-Instruct, and DeepSeek-V3 narrow the gap with closed-source counterparts.The authors attribute this trend to advances in model architecture and training methodologies.Source: Table 2 · p. 6, §4.1 · p. 6
  • Models fine-tuned on specific datasets, including Watt-Tool-8B, xLAM-7B (Liu et al., 2024c), and Hammer2.1-7B (Lin et al., 2024), show a significant performance decline on the Special dataset.Source: §4.1 · p. 6
  • Most models achieve less than 50% end accuracy on Agent tasks, which require contextual integration across interdependent tool calls, long-term consistency, reasoning, and adaptability.These tasks simulate dynamic, real-world multi-turn interactions.Source: §4.1 · p. 6, Table 4 · p. 7

4.2 Error Analysis

  • Parameter-value errors dominate Normal data, output-format errors rank second, and function-name and parameter-type errors are less frequent.The authors associate the leading errors with limited contextual understanding and numerical generation, and format errors with difficulty following predefined syntax and rules.Source: §4.2 · pp. 6–7, Figure 6 · p. 8
  • On Special data, most errors stem from failure to detect instruction problems; other errors detect a problem but give unclear corrective feedback.Examples include failing to identify incorrect parameter values or missing critical information.Source: §4.2 · p. 7, Table 5 · p. 7
  • Agent errors arise from incorrect function calls or parameters, violations of scene rules, and mismanagement of context in multi-turn interactions.These failures reflect weaknesses in tool-use capability, task execution, and recording or processing contextual information.Source: §4.2 · pp. 7–8

4.3 Further Analysis

  • Performance improves as Qwen2.5-Coder and Qwen2.5-Instruct models scale up, especially on high-complexity tasks, but gains slow between 32B and 72B.The reported pattern indicates diminishing marginal benefits from further parameter increases.Source: Figure 7 · p. 8, §4.3 · p. 8
  • Standard prompts achieve the highest overall accuracy among the tested strategies, which also include condensed and minimal prompts.The paper attributes the standard prompt’s advantage to explicit formatting specifications that reduce interference from extraneous variables.Source: §4.3 · p. 8, Table 6 · p. 8

5 Conclusion

ACEBench evaluates LLM tool use across normal, special, and agent data, addressing gaps in multi-turn, fine-grained, and low-overhead evaluation. Experiments find that fine-tuning on specific tool-use datasets can hinder generalization to complex or imperfect instructions, while code capabilities enhance tool-use performance.

5 Conclusion

  • Fine-tuned models struggle to generalize to complex or imperfect instructions, while code capabilities enhance large models’ tool-use performance.The experiments also demonstrate ACEBench’s effectiveness in providing deeper insights into the tool-use abilities of various models.Source: §5 · pp. 8–9
  • ACEBench evaluates LLM tool use across normal, special, and agent data, addressing gaps in real-world multi-turn assessment, fine-grained function-call evaluation, and evaluation cost.The benchmark targets parameter-type function calls and the high costs of using large models for evaluation.Source: §5 · p. 8

Limitations · A Detailed Descriptions of Test Cases

ACEBench’s evaluation may not fully reflect real-world tool use: its LLM-generated test data differs from real-world application data, and manually designed Agent scenarios limit diversity and coverage.

Limitations

  • LLM-generated test data may differ from real-world application data, potentially affecting how well ACEBench performance reflects real-world scenarios.The authors note that measures were taken to ensure the data’s authenticity and diversity, but a gap remains.Source: Limitations · p. 9
  • Manually constructed Agent evaluation scenarios somewhat limit the diversity and coverage of ACEBench’s evaluation framework.Source: Limitations · p. 9

A.1 Data Categories Description … Agent Data

ACEBench organizes tool-use evaluation into Normal, Special, and Agent data, covering basic function calling, instructions that cannot be resolved as given, and complex interactive scenarios.

A.1 Data Categories Description

  • ACEBench divides its benchmark into Normal, Special, and Agent categories.These categories structure the benchmark’s data descriptions.Source: §A.1 · p. 11

Normal Data

  • Normal Data uses fixed question-answer pairs mapped to correct function calls and covers single- and multi-turn interactions, including parallel calls, topic switches, and query refinements.Single-turn cases vary by function-call count; multi-turn cases either change topics or refine the original question.Source: Normal Data · p. 11
  • Normal Data tests API selection among similar candidates and argument generation using supplementary user-profile information.Preference cases require mining factors such as past interactions and interests.Source: Normal Data · p. 11
  • Atom Data isolates parameter types to test their effect on data filling, with five types: number, enum, list, bool, and object.Its candidate APIs contain specific parameter types, such as numbers or lists.Source: Normal Data · p. 11

Special Data

  • Special Data covers cases that candidate functions cannot resolve: missing required information (Incomplete), invalid parameter formats or constraints (Error), and requests beyond function capabilities (Irrelevant) (Wang et al., 2024c).Error cases include values that must match a pattern or come from a predefined list.Source: Special Data · p. 11

Agent Data

  • Agent Data evaluates multi-step collaboration in real-world-modeled settings, using GPT-4o to simulate users; scenarios distinguish one user interaction from multiple interactions across a dialogue cycle.The distinction is between a single interaction throughout the flow and multiple interactions throughout the cycle.Source: Agent Data · p. 11
  • Agent Data’s scenarios simulate mobile-app operations, food delivery, banking, and end-to-end travel booking, from searches and transactions to changes or cancellations.The travel scenario includes inquiry, booking, payment, rescheduling, and cancellation; the banking scenario includes deposits, withdrawals, transfers, loans, and repayments. The team is working on adding more Agent Data scenarios.Source: Agent Data · pp. 11–12

B The Construction of Data … Error

The section describes how Agent data is built from real-world scenarios and how special data captures instructions that are incomplete, invalid, or beyond a function’s capabilities.

B.1 Agent Data

  • Agent data starts by extracting task requirements from real-world scenarios, dividing them into domain-specific sub-scenarios, and defining each module’s functional objectives.Examples include flight booking, food delivery, and financial services.Source: §B.1 · p. 12, Figure 8 · p. 13
  • Agent functions are implemented in code with processes for authentication, cost calculation, payment, and information recording, including exception handling.The passage says this design is intended to support scalability and robustness.Source: §B.1 · p. 12
  • Agent interaction rules standardize user-Agent exchanges, while scenario-based questions and solutions guide users through exceptions and target task outcomes.For food delivery, rules may verify the account, merchant information, and balance status.Source: §B.1 · p. 12

Irrelevant

  • Irrelevant data represents instructions beyond a function’s capabilities and is constructed by removing the correct API from the common data’s candidate APIs.Source: Irrelevant · p. 12

Incomplete

  • Incomplete-instruction data covers missing key information that can cause incorrect function calls; generated dialogues are converted into data with reference answers identifying the missing information.Source: Incomplete · p. 12

Error

  • Error-instruction data covers parameters or names that violate required formats or constraints, with generated conversations converted into data.Source: Error · p. 14

C Examples of Dataset … E Formula for Overall Accuracy

The appendices catalog dataset examples for Normal, Special, and Agent data, present evaluation inference prompts and prompt strategies, and introduce an Overall Accuracy formula.

C.1 Normal Examples

  • Normal examples cover Single-Turn, Multi-Turn, Preference, Similar APIs, and Atom data.The examples are shown in Figures 13–17, respectively.Source: §C.1 · p. 14

C.2 Special Examples

  • Special examples cover Incomplete, Error, and Irrelevant data.The examples are shown in Figures 18–20, respectively.Source: §C.2 · p. 14

C.3 Agent Examples

  • Agent data examples are shown in Figures 21 and 22.Source: §C.3 · p. 14

D.1 Normal Prompt

  • Normal Data evaluation inference prompts are shown in Figures 23 and 24.Source: §D.1 · p. 14

D.2 Special Prompt

  • The Special Data evaluation inference prompt is shown in Figure 27.Source: §D.2 · p. 14

D.3 Agent Prompt

  • Agent Data prompts include a scenario-specific example in Figure 28 and an inference prompt in Figure 29.Source: §D.3 · p. 14

D.4 Different Prompt Strategies

  • The prompt strategies are Standard, Condensed, and Minimal, shown in Figures 23, 25, and 26, respectively.Source: §D.4 · p. 14

E Formula for Overall Accuracy

  • The appendix gives an Overall Accuracy formula with coefficients A, B, and C, and defines sample sizes and accuracy for Normal, Special, and Agent data.Source: E Formula for Overall Accuracy · p. 14

F Detailed Results · G Error Examples · G.1 Error Examples of Normal Data

The detailed results section points to separate English- and Chinese-data accuracy tables. The normal-data error examples cover wrong function names, parameter types, parameter values, and output formats.

F Detailed Results

G.1 Error Examples of Normal Data

  • The normal-data examples include a wrong function name, shown in Figure 30.Source: §G.1 · p. 15
  • A wrong parameter type example is shown in Figure 31.Source: §G.1 · p. 15
  • A wrong parameter value example is shown in Figure 32.Source: §G.1 · p. 15
  • A wrong output format example is shown in Figure 33.Source: §G.1 · p. 15

G.2 Error Examples of Special Data

The section illustrates two Special Data error cases: error detection and error correction.

G.2 Error Examples of Special Data

Related papers

Loading 2501.12851v8…