PaperlayerView exact revision on arXiv

Source-linked AI summary

StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models

Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, Yang Liu

arXiv:2403.07714v5cs.CL

TL;DR

Tool-learning benchmarks need large scale without sacrificing stability, but online APIs and automatic evaluation can vary over time. StableToolBench combines cached and LLM-simulated APIs with solvability-aware metrics and GPT-4 evaluation. The resulting benchmark provides more stable performance assessments while retaining API realism and diversity, although its GPT-4 dependence increases cost and creates upgrade sensitivity.

  • Problem

    Large-scale online-tool benchmarks improve real-world coverage but suffer from unstable API availability and unreliable automatic evaluation, weakening reproducibility and comparability.

    Source: §1 · p. 1

  • Method

    StableToolBench combines cached API outputs, LLM-based API simulation, solvability-aware pass and win rates, and GPT-4 automatic evaluation.

    Source: §1 · p. 2

  • Results

    StableToolBench provides more stable model performance evaluations under API changes and failures while retaining realism, diversity, and documentation-following reliability.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    StableToolBench offers a more reproducible large-scale basis for evaluating tool-learning models across changing API conditions.

    Source: §1 · p. 2, §6 · p. 9

    The benchmark depends on GPT-4 for evaluation and API simulation, increasing cost and making performance sensitive to future backbone upgrades.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) have witnessed remarkable advancements in recent years, prompting the exploration of tool learning, which integrates LLMs with external tools to address diverse real-world challenges. Assessing the capability of LLMs to utilise tools necessitates large-scale and stable benchmarks. However, previous works relied on either hand-crafted online tools with limited scale, or large-scale real online APIs suffering from instability of API status. To address this problem, we introduce StableToolBench, a benchmark evolving from ToolBench, proposing a virtual API server and stable evaluation system. The virtual API server contains a caching system and API simulators which are complementary to alleviate the change in API status. Meanwhile, the stable evaluation system designs solvable pass and win rates using GPT-4 as the automatic evaluator to eliminate the randomness during evaluation. Experimental results demonstrate the stability of StableToolBench, and further discuss the effectiveness of API simulators, the caching system, and the evaluator system.

1 Introduction

Large-scale tool-learning benchmarks improve real-world coverage but face unstable APIs and unreliable evaluation, undermining reproducibility. StableToolBench addresses these issues with cached and simulated APIs, a revised evaluator, and stable evaluation metrics.

Motivation

  • Large-scale online-tool benchmarks better reflect real-world scenarios but can produce non-reproducible and incomparable results as APIs change over time.ToolBench is cited as an example whose performance discrepancies could not be reproduced months after release.Source: §1 · p. 1
  • ToolBench instability arises from changing API availability and evaluators that may misjudge whether complex tool-use tasks are solvable.The paper specifically identifies inconsistent API status and limitations of gpt-3.5 as key sources of variability.Source: §1 · pp. 1–2

StableToolBench

  • StableToolBench replaces real APIs with a virtual system that first retrieves cached outputs and otherwise uses LLMs to simulate unavailable API behavior from documentation and examples.The cache covers repeated API-call scenarios, while simulation handles cache misses and unavailable real APIs.Source: §1 · p. 2
  • The benchmark redesigns evaluation with solvable pass and win rates and uses GPT-4 instead of prior automatic evaluators to reduce evaluation randomness.The revised virtual API and evaluation systems are intended to provide stable assessments despite API modifications.Source: §1 · p. 2

Empirical findings

  • Extensive experiments report more stable model performance under API failures while retaining realism, diversity, and documentation-following reliability comparable to real APIs.The contribution summary characterizes the benchmark as balancing stable simulated APIs with API reality and broader evaluation stability.Source: §1 · p. 2

2 Stability Analysis on ToolBench

ToolBench stability is assessed across performance, evaluation, and API status, revealing substantial variation in all three dimensions.

2.1 Stability of Performance

  • ToolBench performance declined over time across all tested methods, raising concerns about benchmark stability.The study reproduces reported settings with ChatGPT and ToolLLaMA using CoT and DFS strategies.Source: §2.1 · pp. 2–3

2.2 Stability of Evaluation

  • The original Pass Rate and Win Rate procedures produce inconsistent results because gpt-3.5-turbo-16k struggles to judge tool-learning answers reliably.Across repeated CoT and DFS runs, Pass Rate and Win Rate trends diverge, with DFS’s second Win Rate reaching 48.0.Source: §2.2 · p. 3

2.3 Stability of API Status

  • Only 44.4% of scanned ToolBench API calls succeeded, while the remainder were mostly unavailable or unauthorised.The API-status analysis classifies calls as successful, unavailable, or unauthorised and reports varied unavailability errors.Source: §2.3 · p. 3, Figure 3 · p. 3
  • Reducing the proportion of successful APIs substantially degraded model performance, demonstrating that API failures materially affect benchmark stability.Experiments tested 0%, 10%, 20%, and 50% sampled API failures on the I1 Instruction set using repeated evaluations.Source: §2.3 · p. 4, Figure 4 · p. 4

3 StableToolBench

StableToolBench improves benchmark stability through a virtual API server and a two-phase evaluation system. The server combines caching with API simulation, while evaluation isolates solvable tasks and uses GPT-4-based metrics.

3 StableToolBench

  • StableToolBench combines a virtual API server with a stable evaluation system to improve tool-learning benchmark stability.Source: §3 · p. 4

3.1 Virtual API Server

  • The caching system stores validated API-call responses keyed by category, tool, API name, and arguments, while continuously updating with new experiments.Source: §3.1 · p. 4
  • The API simulator uses gpt-4-turbo, documentation, and up to five cached real API calls as few-shot examples to mock unavailable APIs.Source: §3.1 · p. 4
  • The virtual API server first uses cached responses, then real APIs, and finally simulated APIs, saving returned responses back into the cache.Source: §3.1 · p. 5

3.2 Stable Evaluation System

  • The stable evaluator judges solvable tasks with three LLMs, then reports SoPR and SoWR using gpt-4-turbo instead of gpt-3.5-turbo-16k.SoPR assigns scores of 1, 0.5, and 0 to solved, unsure, and unsolved answers; SoWR directly resolves mixed outcomes and otherwise uses GPT-4 comparisons.Source: §3.2 · p. 5

4 Experiment

Experiments evaluate baseline performance, server stability, simulator realism and diversity, cache effectiveness, and evaluator quality. Across these tests, the proposed system remains stable while simulated APIs and GPT-4-based evaluation perform reliably.

4.1 Performance

  • GPT-4-series models outperform GPT-3.5 models, while DFS outperforms CoT across the tested LLMs under SoPR and SoWR.Source: §4.1 · p. 6

4.2 Stability of Virtual API Server

  • With 50% of APIs unavailable, performance changes remain insignificant within variance when unavailable calls are redirected to simulated APIs.Source: §4.2 · p. 6, Figure 7 · p. 6
  • Changing gpt-4-turbo simulator versions and temperatures produces performance changes within evaluation variance, indicating robust API simulators.Source: §4.2 · p. 6, Table 6 · p. 7

4.3 Turing Test of API Simulator

  • Human annotators cannot distinguish simulated from real API responses well, and simulated APIs are judged to behave more like real situations.The comparison covers overall realism, format accuracy, and answer relevance across 70 API pairs.Source: §4.3 · p. 7, Figure 8 · p. 7
  • 90% of sampled simulated outputs are judged reasonable against their documentation, while 6% are unreasonable and 4% uncertain.Source: §4.3 · p. 7

4.4 Diversity of API Simulator

  • Real and simulated API responses occupy similar embedding space, and only 2 of 60 APIs repeat exactly across two of five calls.Source: §4.4 · pp. 7–8, Figure 9 · p. 8

4.5 Effectiveness of Caching System

  • The cache achieves very high hit rates for in-domain methods and still mitigates instability substantially for newer out-of-domain methods.Source: §4.5 · p. 8, Table 7 · p. 8

4.6 Human Evaluation of Evaluator

  • Claude 2, Gemini, and GPT-4 outperform GPT-3.5 in solvability judgments, while GPT-4 is substantially better for answer evaluation and win-rate comparison.All GPT-3.5 evaluator accuracies are below 70%.Source: §4.6 · p. 8

5 Related Work

Related work covers tool-learning benchmarks, methods for integrating tools with LLMs, and evaluation strategies. The paper positions StableToolBench around the underexplored problem of evaluation stability.

Tool Learning Benchmarks

  • Tool-learning research studies LLM access to external tools for real-time knowledge, multimodal functions, and specialized domain skills, but rarely examines benchmark-environment stability.Source: §5 · p. 8

Tool Inference Methods

  • Tool inference methods use in-context tool descriptions and demonstrations or explicitly train LLMs on tool-interaction datasets.Source: §5 · pp. 8–9

Evaluation in Tool Learning

  • Tool-learning evaluation commonly measures response comparison, tool-call accuracy, solution paths, or multi-turn interaction, while evaluation stability remains a literature gap.Source: §5 · p. 9, Table 8 · p. 9

6 Conclusion

StableToolBench improves the stability of model-performance evaluation while retaining realism in its simulated APIs. Its caching system contributes substantially to the benchmark’s enhanced stability.

6 Conclusion

  • StableToolBench increases benchmark stability without declining reality, with simulated APIs offering realism and caching contributing greatly to enhanced stability.Source: §6 · p. 9

Limitations

StableToolBench improves evaluation stability but depends on costly, potentially changing closed-source GPT-4 components and a cache whose future effectiveness is uncertain.

  • GPT-4 is used both as the automatic evaluator and backbone server, increasing benchmark usage cost.Source: Limitations · p. 9
  • Future upgrades to the GPT-4 backbone may change API-simulation performance, while current open-source models are not yet capable enough for this role.Source: Limitations · p. 9
  • Although explored methods achieve high cache hit rates, future methods may reduce the cache’s effectiveness, motivating gradual cache updates.Source: Limitations · p. 9

A Comparison of Reported and Reproduced Performance

The supplementary material documents reproduced-versus-reported performance, API availability changes, and stability tests using virtual API systems across model versions.

A Comparison of Reported and Reproduced Performance

B Statistics of API change information

C Stability Test Scores with Virtual API Systems

D Call Error Identification and Cache Filtering Rule

The cache-filtering procedure identifies API-call errors using response and error-message patterns, then treats remaining calls as successful while excluding “Other Errors” from cache filtering.

E Configurations of API Diversity Analysis

The supplementary configurations specify the embedding and UMAP settings for API diversity analysis, while a separate table provides the prompt used for API simulation.

E Configurations of API Diversity Analysis

F Prompts of API simulation

E Configurations of API Diversity Analysis

G Prompt to Filter Solvable Task

This appendix identifies the prompts used to filter solvable tasks and to construct API calls for scanning available tools, with the prompts presented in Tables 14 and 15.

G Prompt to Filter Solvable Task

H Prompt Used to Make API Calls

G Prompt to Filter Solvable Task

  • Table 14 is labeled as the prompt used to filter solvable tasks.Source: Table 14 · p. 14

H Prompt Used to Make API Calls

Related papers

Loading 2403.07714v5…