Source-linked AI summary
ACEBench: Who Wins the Match Point in Tool Usage?
Chen Chen, Xinlong Hao, Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Yuefeng Huang, Wulong Liu, Xinzhi Wang, Defu Lian, Baoqun Yin, Yasheng Wang, Wu Liu
TL;DR
Existing tool-use benchmarks provide limited realistic multi-turn coverage, fine-grained assessment, and low-overhead evaluation. ACEBench addresses these gaps with Normal, Special, and Agent data and automated evaluation infrastructure. Experiments show deeper analysis of model tool-use abilities, while the benchmark remains bounded by synthetic data and manually constructed Agent scenarios.
Problem
Existing tool-use benchmarks lack realistic multi-turn scenarios, fine-grained assessment, and low-overhead evaluation.
Method
ACEBench combines Normal, Special, and Agent data with automated evaluation and sandboxed multi-turn, multi-step tool-use scenarios.
Results
Extensive experiments show that ACEBench provides more comprehensive analysis and deeper insights into LLM tool-use abilities.
Takeaways & Limitations
ACEBench supports broader and more granular evaluation of tool use across basic, imperfect-instruction, and interactive scenarios.
Takeaways & Limitations
The test data may differ from real-world application data, and manually constructed Agent scenarios limit evaluation diversity and coverage.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have demonstrated significant potential in decision-making and reasoning, particularly when integrated with various tools to effectively solve complex problems. However, existing benchmarks for evaluating LLMs' tool usage face several limitations: (1) limited evaluation scenarios, often lacking assessments in real multi-turn dialogue contexts; (2) narrow evaluation dimensions, with insufficient detailed assessments of how LLMs use tools; and (3) reliance on LLMs or real API executions for evaluation, which introduces significant overhead. To address these challenges, we introduce ACEBench, a comprehensive benchmark for assessing tool usage in LLMs. ACEBench categorizes data into three primary types based on evaluation methodology: Normal, Special, and Agent. "Normal" evaluates tool usage in basic scenarios; "Special" evaluates tool usage in situations with ambiguous or incomplete instructions; "Agent" evaluates tool usage through multi-agent interactions to simulate real-world, multi-turn dialogues. We conducted extensive experiments using ACEBench, analyzing various LLMs in-depth and providing a more granular examination of error causes across different data types.
1 Introduction
ACEBench addresses gaps in tool-use benchmarks by covering broader scenarios, finer-grained capabilities, imperfect instructions, and automated evaluation. It organizes evaluation into Normal, Special, and Agent data and reports extensive experimental analysis.
- Motivation: Existing benchmarks lack realistic multi-turn dialogue evaluation, fine-grained assessment, and coverage of imperfect instructions.These limitations include fixed dialogue combinations and insufficiently detailed evaluation of tool use.
- Benchmark design: ACEBench evaluates tool use through Normal, Special, and Agent categories spanning basic, imperfect-instruction, and multi-turn multi-step scenarios.Normal includes single-turn, multi-turn, personalized, and atomic-level cases; Special includes incomplete, malformed, or mismatched instructions; Agent models real-world interactions.
- Contributions: The benchmark provides finer-grained perspectives, imperfect-instruction assessments, and more stable evaluation metrics.Its stated goal is comprehensive evaluation across diverse tool-use scenarios.
- Evaluation infrastructure: The benchmark uses an end-to-end automated evaluation system and sandbox environments for multi-turn, multi-step tool invocation.The system is based on abstractions of real-world scenarios.
- Experimental validation: Extensive experiments provide more comprehensive and differentiated analysis of LLM tool usage.The authors present this as evidence that ACEBench enables clearer evaluation of tool-use abilities.
2 Related Works
ACEBench expands tool-use evaluation beyond costly or fixed-dialogue benchmarks by simulating realistic multi-turn interactions and enabling automated assessment. Its framework broadens scenario coverage while targeting scalability and evaluation cost.
- Existing benchmarks: Existing tool-use benchmarks have significant limitations in scenario realism, evaluation breadth, and evaluation cost.Prior work includes fixed-content multi-turn dialogues and dependence on large models or external APIs.
- ACEBench: ACEBench broadens evaluation across tool-use scenarios and supports realistic multi-turn dialogue processes.The framework is designed to address the restricted interaction patterns of earlier benchmarks.
- Evaluation framework: End-to-end automated assessment reduces evaluation costs and improves scalability.This framework simulates interactions while avoiding reliance on fully manual or costly evaluation procedures.
3 ACEBench
ACEBench combines automatically generated Normal and Special data with expert-constructed Agent scenarios, followed by quality verification and broad distribution analysis. Its evaluation uses exact function-call matching, problem-identification accuracy, and agent end-to-end and process metrics.
- Dataset construction: The dataset contains two linguistically parallel versions with equal data-type distributions and 2,000 annotated entries.The versions are Chinese and English.
- Dataset construction: Normal and Special data are automatically generated with synthetic APIs guided by real APIs, while Agent data is constructed by human experts.The API-generation process uses a hierarchical context tree to expand domain and functionality coverage.
- Agent data: Agent scenarios abstract real-world interactions into functional modules, state variables, sandbox environments, and multi-step tool-use cases.Examples include food delivery and telecommunications operations.
- Data verification: ACEBench applies automated rule-based and model-based checks followed by expert review to verify data quality.The checks cover API clarity, function-call executability, dialogue accuracy, sample consistency, and semantic errors.
- Data analysis: The benchmark covers 8 major domains, 68 sub-domains, and 4,538 APIs in Chinese and English.Its test cases span dialogue turns from 1 to 8 and varying numbers of API arguments.
- Evaluation: Normal evaluation compares function-call outputs with ground truth using AST parsing and accepts any matching candidate answer.Accuracy is binary: 1 for a full match and 0 for a mismatch.
- Evaluation: Special evaluation measures detection of missing or erroneous parameters and task-function mismatches, while Agent evaluation measures end-to-end and process accuracy.Agent process accuracy compares actual and ideal function-call processes.
4 Experiments
ACEBench evaluates LLM tool usage across diverse models and data types, revealing distinct weaknesses in imperfect instructions, complex multi-turn tasks, error handling, and scaling behavior.
- Main results and analysis: Closed-source models, including the GPT-4 series, achieve the strongest overall performance, while several open-source models are narrowing the gap.The cited comparison identifies Qwen2.5-Coder-32B-Instruct, Qwen2.5-72B-Instruct, and DeepSeek-V3 as progressively closer to closed-source counterparts.
- Main results and analysis: Most models achieve less than 50% end accuracy on Agent tasks, reflecting the difficulty of contextual, interdependent, multi-turn tool use.These tasks require maintaining consistency, integrating context, and adapting to evolving task states.
- Error Analysis: Param value errors dominate Normal-data failures, followed by output format errors, whereas function name and parameter type errors are less frequent.The pattern indicates comparatively stronger function-call matching and data-type handling than numerical generation and format compliance.
- Error Analysis: Most Special-data errors are caused by Error Detection failures rather than Error Correction failures.Models often fail to detect imperfect instructions or identify their specific problems before generating feedback.
- Error Analysis: Agent errors arise from function-call errors, rule violations, and information mismanagement during multi-turn interactions.These error types correspond to failures in selecting valid tools or parameters, following scene logic, and retaining contextual information.
- Further Analysis: Performance improves as model size increases, especially on high-complexity tasks, but gains slow between 32B and 72B models.The results indicate diminishing marginal benefits from further parameter scaling.
- Further Analysis: Standard prompts achieve the highest overall accuracy among the tested prompting strategies.The paper attributes this result to rigorous formatting specifications that reduce interference from extraneous variables.
5 Conclusion
The paper introduces ACEBench to evaluate LLM tool-use capabilities across normal, special, and agent scenarios. Extensive experiments show that it provides deeper insights into tool-use abilities and addresses limitations in existing benchmarks.
- ACEBench evaluates LLM tool use across Normal, Special, and Agent data types.
- Extensive experiments demonstrate ACEBench’s effectiveness in providing deeper insights into the tool-use abilities of various models.
Limitations
ACEBench’s evaluation has two stated limitations: generated test data may differ from real-world applications, and manually designed Agent scenarios constrain diversity and coverage.
- Generated test data may differ from real-world application data, potentially affecting evaluation of real-world model performance.
- Manual construction of Agent scenarios limits the diversity and coverage of the evaluation framework.
A.1 Data Categories Description
ACEBench divides tool-use evaluation into Normal, Special, and Agent data categories. These categories cover fixed-call competence, imperfect instructions, and multi-step collaboration in modeled real-world settings.
- Normal: Normal Data contains fixed question-answer pairs mapped to correct function calls across single-turn, multi-turn, similar-API, preference, and Atom categories.
- Normal: Atom Data isolates parameter types to test whether numbers, enums, lists, booleans, or objects affect data filling.
- Special: Special Data covers incomplete, erroneous, and irrelevant instructions that candidate functions cannot correctly resolve.
- Agent: Agent Data models tasks requiring multi-step collaboration in real-world-like environments, including mobile, food-delivery, financial-services, and travel-booking scenarios.
- Agent: Agent scenarios distinguish single-interaction multi-step flows from dialogues involving multiple user interactions.
B The Construction of Data
ACEBench constructs Normal and Special data through automated dialogue generation, while Agent data uses scenario abstraction, coded environments, interaction rules, and designed task formats.
- Agent Data Construction: Agent-data construction begins by extracting real-world requirements and modularizing them into domain-specific sub-scenarios.
- Agent Data Construction: Each Agent function is implemented in code with processes for authentication, calculation, payment, recording, and exception handling.
- Agent Data Construction: Scenario-specific interaction rules standardize user-Agent exchanges, including verification and guidance during exceptional cases.
- Special Data Construction: Irrelevant data removes the correct API from common-data candidate APIs so the instruction exceeds available function capabilities.
- Special Data Construction: Incomplete data is generated from dialogues where a meaningful tool request remains missing one or more required parameters.
- Special Data Construction: Error data contains parameters or names violating required formats or constraints and is constructed through dedicated prompts and examples.
C Examples of Dataset
ACEBench documents examples for every data category and provides corresponding evaluation prompts, including alternative prompt lengths and an overall-accuracy formulation.
- Dataset Examples: Examples are provided for Normal single-turn, multi-turn, preference, similar-API, and Atom data.
- Dataset Examples: Examples are provided for Special incomplete, error, and irrelevant data, as well as Agent data.
- Evaluation Prompts: Evaluation prompts are separately documented for Normal, Special, and Agent data.
- Evaluation Prompts: Normal evaluation prompts include Standard, Condensed, and Minimal versions.
- Overall Accuracy: Overall Accuracy combines Normal, Special, and Agent accuracies using coefficients defined from their respective sample sizes.
F Detailed Results
This section presents model accuracy evaluations for English and Chinese data and introduces four categories of tool-use errors in Normal Data.
- Accuracy Evaluation: Accuracy evaluations for different models are reported separately for English Data and Chinese Data.The English results appear in Table 7, while the Chinese results appear in Table 8.
- Normal Data Errors: Wrong Function Name describes an incorrect function-name selection in Normal Data.Figure 30 provides an example of this error type.
- Normal Data Errors: Wrong Param Type describes an incorrect parameter type in Normal Data.Figure 31 provides an example of this error type.
- Normal Data Errors: Wrong Param Value describes an incorrect parameter value in Normal Data.Figure 32 provides an example of this error type.
- Normal Data Errors: Wrong Output Format describes an incorrect output format in Normal Data.Figure 33 provides an example of this error type.
G.2 Error Examples of Special Data
The supplied examples cover error detection and correction in Special Data alongside Normal, Special, and Agent data formats, prompts, and application scenarios.
- G.2 Error Examples of Special Data: Special Data examples include error detection and error correction cases.Figure 34 illustrates error detection, while Figure 35 illustrates error correction.
- Data Examples: The benchmark examples distinguish Normal data formats, including single-turn, multi-turn, preference, similar-API, and atom-enum data.Figures 13–17 provide examples of these formats.
- Data Examples: Special data examples include incomplete, error-parameter, and irrelevant cases.Figures 18–20 provide examples of these Special Data types.
- Data Examples: Agent Data examples show partially correct and fully accurate representations.Figures 21 and 22 provide the corresponding examples.
- Application Setting: The examples include API-request formatting instructions and a simulated mobile-assistant setting for messaging, reminders, and takeout.The setting also specifies a current time of June 11, 2024, 16:00 Beijing Time.
- Inference Prompts: Inference prompts vary across Normal, Special, and Agent Data, with separate prompts for preference data and user simulation.The examples include condensed and minimal Normal prompts, a Special prompt, and Agent prompts for specific scenarios.