PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Tool Manipulation Capability of Open-source Large Language Models

Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, Jian Zhang

arXiv:2305.16504v1cs.CLcs.AIcs.LG

TL;DR

Tool-manipulation research largely relies on closed LLM APIs, despite security and robustness risks from exposing enterprise information. The paper adapts model alignment, system prompts, and retrieved demonstrations for open-source models, evaluating them on ToolBench. The techniques boost success rates by up to 90% and are competitive with GPT-4 on 4 of 8 tools, with about one developer day of curation per tool.

  • Problem

    Tool-manipulation research mostly relies on closed LLM APIs, while exposing enterprise workflows and information to those services raises security and robustness risks.

    Source: §1 · p. 1

  • Method

    The paper adapts model alignment with programmatic data generation, system prompts, and in-context demonstration retrievers, and evaluates these techniques using ToolBench.

    Source: §1 · p. 2, §8 · p. 10

  • Results

    Open-source LLM success rates improved by up to 90%, with results competitive or better than GPT-4 on 4 of 8 ToolBench tools.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that the recipe typically takes about one developer day per tool to curate data, a practical amount of human supervision.

    Source: §1 · p. 2

    Boosted open-source models still have relatively low success rates on tasks requiring advanced reasoning, including Google Sheets, WebShop, and Tabletop.

    Source: §6.3 · p. 9

Abstract

from arXiv · show

Recent studies on software tool manipulation with large language models (LLMs) mostly rely on closed model APIs. The industrial adoption of these models is substantially constrained due to the security and robustness risks in exposing information to closed LLM API services. In this paper, we ask can we enhance open-source LLMs to be competitive to leading closed LLM APIs in tool manipulation, with practical amount of human supervision. By analyzing common tool manipulation failures, we first demonstrate that open-source LLMs may require training with usage examples, in-context demonstration and generation style regulation to resolve failures. These insights motivate us to revisit classical methods in LLM literature, and demonstrate that we can adapt them as model alignment with programmatic data generation, system prompts and in-context demonstration retrievers to enhance open-source LLMs for tool manipulation. To evaluate these techniques, we create the ToolBench, a tool manipulation benchmark consisting of diverse software tools for real-world tasks. We demonstrate that our techniques can boost leading open-source LLMs by up to 90% success rate, showing capabilities competitive to OpenAI GPT-4 in 4 out of 8 ToolBench tasks. We show that such enhancement typically requires about one developer day to curate data for each tool, rendering a recipe with practical amount of human supervision.

1 Introduction

The paper targets practical open-source LLM tool manipulation because closed APIs pose security and robustness risks when given enterprise information. It analyzes common failures, adapts alignment, prompts, and demonstrations, and evaluates the resulting recipe on ToolBench.

Motivation

  • The need for open-source tool manipulation is motivated by security and robustness risks from exposing enterprise workflows and information to closed LLM APIs.The introduction also reports that a leading open code-generation model failed every on-sale house-search test, while GPT-4 reached 77% success on the same 100 examples.Source: §1 · pp. 1–2

Challenges

  • Open-source models struggle with API selection, argument values without demonstrations, and generating executable code rather than surrounding natural language.These are the three challenges the paper identifies as hindering tool manipulation.Source: §1 · p. 2

Enhancement approach

  • The recipe combines instruction tuning on programmatically generated API-use examples, retrieved in-context demonstrations, and system prompts that guide executable API-call generation.Templates are instantiated with concrete values; demonstrations are retrieved by semantic similarity from a human-curated pool.Source: §1 · p. 2

Evaluation

  • ToolBench evaluates eight diverse software tools, with tasks requiring selection and combination of API functions from as many as one hundred candidates.The benchmark includes tasks ranging from Google Sheets manipulation to robot control.Source: §1 · p. 2

Results

  • On ToolBench, simple techniques improved open-source LLM success rates by up to 90%, achieving competitive or better results than GPT-4 on 4 of 8 tools; synthetic-data alignment primarily drove the gains.On average, a developer took one day to craft the in-context demonstrations and curate templates for generating alignment data.Source: §1 · p. 2

2 Background

The background frames software tool manipulation as translating a natural-language goal into API calls, either directly or through iterative interaction with an environment. Its motivating comparison reports a large open-versus-closed model gap and illustrates the potential value of in-context demonstrations.

2 Background

  • The background situates the study as an investigation of open-source LLM capability, using API calls as the software interface for carrying out natural-language goals.The authors choose APIs because they are a prevalent abstraction for developers and users in modern software systems.Source: §2 · p. 3, §2.1 · p. 3

2.1 Tool manipulation setup

  • An action generator maps a goal to API calls using relevant API documentation, optionally retrieved from a larger documentation pool, and optional prompt information.The setup uses a retrieved subset when the full API pool exceeds the model's input-length limit.Source: §2.1 · p. 3
  • Single-step manipulation produces API calls directly, whereas multi-step manipulation alternates API calls with environment observations until an exit state.The paper uses the single-step setup for illustration but evaluates both single- and multi-step cases.Source: §2.1 · p. 3

2.2 Motivating Observation

  • In the motivating home-search comparison, open-source LLMs had a 70% success-rate gap relative to zero-shot GPT-4.Figure 2 also contrasts API selection without documentation and shows improved OpenWeather success with an oracle one-shot demonstration.Source: §2.2 · p. 4, Figure 2 · p. 4

3 Challenges for open-source LLMs

Open-source models show a substantial tool-manipulation gap and fail in API selection, argument population, and executable generation. Demonstrations can improve performance, though the strongest examples are hand-picked for individual test cases.

3 Challenges for open-source LLMs

  • For the house-searching task, the reported comparison found that open-source LLMs lag behind OpenAI GPT-4 by 70% on success rate.Source: Table 1 · p. 4
  • Table 3 categorizes typical tool-manipulation error types for a weather-query tool.When a failure has multiple errors, the authors classify it by the first triggered category: non-executable generation, wrong API selection, then wrong argument population.Source: Table 3 · p. 4, §3 · p. 4
  • Without documentation or demonstrations, open-source models struggle to select APIs that GPT-4 can identify from the goal alone.The authors suggest GPT-4 may internalize API usage during training.Source: §3 · p. 4
  • Wrong argument values account for up to 63% of open-source model failures.Hand-picked oracle demonstrations improved success rates by up to 45%, but were selected separately for each test case.Source: §3 · pp. 4–5
  • Open-source models produce non-executable output in 23% of one hundred weather-query cases, including verbosity around API calls and failures to follow format guidelines.The authors identify code-only generation as a needed regulation.Source: §3 · p. 5

4 Boosting Open-source LLMs for Tool Manipulation

The section adapts model alignment, demonstration retrieval, and generation regulation to address tool-manipulation failures with limited human curation. Together, these methods use programmatic examples, relevant retrieved demonstrations, and code-only instructions.

4.1 Multi-tool model alignment with programmatic data curation

  • Human-written templates paired with randomly sampled keyword values generate alignment examples, with O(n) templates per tool and about one developer day of curation on average.Each API is encouraged to appear in at least one template, and the resulting data tunes one model jointly across tools.Source: §4.1 · p. 5, Figure 3 · p. 5

4.2 Demonstration retrieval

  • A retriever selects semantically similar examples from a repository requiring each API to appear in only one human-curated demonstration.This design uses O(n) examples to support goals involving unseen API combinations without manually selecting examples per request.Source: §4.2 · pp. 5–6
  • With 10 demonstrations for a 15-API home-search task, retrieved examples boost open-source success rates by up to 79% and make GPT-4 nearly perfect.The demonstrations do not precisely match any test case's API combination.Source: §4.2 · p. 6, Figure 4 · p. 6

4.3 Generation regulation with system prompts

  • A shared system prompt defines the goal, demonstration, and generation format, instructs code-only output, and inserts the goal and retrieved APIs for each request.The task-specific content is populated at inference, leaving one-time prompt development.Source: §4.3 · p. 6, Figure 5 · p. 6

5 ToolBench: A New Tool Manipulation Benchmark

ToolBench provides predefined test cases for quantitative evaluation across diverse software tools and real-world tasks. It assesses API-selection complexity and advanced reasoning challenges, using execution-based evaluation and task-specific metrics.

5 ToolBench: A New Tool Manipulation Benchmark

  • ToolBench is presented as the first open-source benchmark with predefined test cases for quantitative evaluation of tool manipulation.Its tasks cover generalization to unseen API combinations as well as advanced reasoning.Source: §5 · p. 6

5.1 Software tools and evaluation infrastructure

  • The benchmark combines five collected tasks and three tasks derived from VirtualHome, WebShop, and Tabletop, spanning single- and multiple-step action generation across 2 to 108 APIs.Each task has approximately 100 test cases, with a limited number of demonstration examples.Source: §5.1 · pp. 6–7, Table 4 · p. 7
  • Evaluation executes generated API calls and assesses final outcomes; success rate is used for most tasks, rewards for WebShop, and executability and LCS for VirtualHome.This permits evaluation without requiring generated calls to match the ground-truth calls exactly.Source: §5.1 · p. 7

5.2 Level of challenges

  • API complexity averages each test sample's minimum distance to a demonstration and reflects difficulty generalizing to unseen API combinations; higher scores mean greater complexity.The score does not capture advanced reasoning, so a low-complexity task such as WebShop can still be challenging.Source: §5.2 · p. 7
  • More than O(n) WebShop demonstrations can improve success rates, and the examples can be acquired through programmatic software operations without heavy human curation.This is a task-specific finding about WebShop.Source: §5.2 · p. 7
  • Advanced reasoning includes non-API coding in Google Sheets and Tabletop, as well as decisions based on WebShop observations.WebShop has zero API-selection complexity because its two one-argument APIs are both covered by demonstrations, yet it can still require reasoning from observations.Source: §5.2 · pp. 7–8

6 Experiment

ToolBench experiments show that the combined recipe substantially improves open-source models, while ablations identify model alignment as the main contributor and show remaining weaknesses on reasoning-heavy tasks.

6.2 Capability Gap

  • In zero-shot settings, open models were up to 74% lower than GPT-4 on simpler tasks, and generally had no meaningful accuracy on other tasks except WebShop.The study compared LLAMA-30B, StarCoder, and CodeGen-16B-mono with GPT-4; the zero-shot setup provided API documentation without demonstrations.Source: §6.2 · p. 8, §6.1 · p. 8

6.3 Boosting open-source LLMs

  • Up to 90% success-rate improvement brought open-source models to competitive or better GPT-4 performance on 4 of 8 tasks, though Google Sheets and Tabletop still lagged.The comparison enhanced GPT-4 with the same system prompt and demonstration retriever; Tabletop was evaluated only few-shot.Source: §6.3 · pp. 8–9, Table 6 · p. 9
  • One developer day on average provided the demonstrations and alignment templates used for each task’s enhancement.Demonstration sets contained 10–83 examples per task, while alignment data typically used fewer than 100 templates.Source: §6.3 · p. 9
  • Boosted open-source models still had relatively low success on advanced-reasoning tasks such as Google Sheets, WebShop, and Tabletop.The authors identify further enhancement of open models’ reasoning capabilities as a remaining challenge.Source: §6.3 · p. 9

6.4 Ablation Study

  • Both demonstrations and model alignment improved success across tasks individually, while system prompts benefited simpler tasks with fewer API calls.In the full system, removing alignment reduced success on up to 7 tasks, versus up to 5 when removing demonstrations and up to 3 when removing system prompts.Source: §6.4 · p. 9, Table 7 · p. 9

7 Related work

The related work contrasts tool execution with both program synthesis and tool-result-augmented generation, then positions this study as an open-model approach with a real-execution benchmark.

7 Related work

  • Unlike rule-based compilation, autoregressive code models generate programs as sequences; CodeGen represents this approach for API-call generation in this study.The cited program-synthesis models include CodeGen [34], SantaCoder [36], and StarCoder [33].Source: §7 · p. 10
  • Prior tool-augmented work includes using tool outputs in generation and executing tools; this study focuses on the latter stream [1, 38, 39, 26, 40, 41, 42].Examples of the execution stream include embodied robotic learning [20] [43] [44] and automation for other tools [46] [47] [48].Source: §7 · p. 10
  • Where recent studies mostly enhance closed-API in-context learning [1] [2] [3] [5], this work adapts established NLP techniques for open models and provides predefined real-execution test cases.The adapted techniques are cited as [19] [26].Source: §7 · p. 10

8 Conclusion

The paper proposes practical techniques to improve open-source LLM tool manipulation and evaluates them on a benchmark of diverse software tasks. The techniques achieve GPT-4-competitive results on four of eight tasks with practical human-labeling effort.

8 Conclusion

  • The paper adapts model alignment with programmatic data generation, system prompts, and in-context demonstration retrievers to improve open-source tool manipulation.The approach draws on observations of common tool-manipulation failures and conventional NLP methods.Source: §8 · p. 10
  • ToolBench evaluates the techniques on diverse software tools for real-world tasks.Source: §8 · p. 10
  • The techniques make leading open-source LLMs competitive with GPT-4 on 4 of 8 ToolBench tasks with practical human-labeling effort.Source: §8 · p. 10

A Benchmark Details

The appendix describes ToolBench task construction and evaluation, then outlines the models included in its broader comparison.

A.1 OpenWeather; A.2 The Cat API

  • OpenWeather and Cat API tests use 100 queries each to assess REST API calls, with generation checked for executability and task-specific correctness.OpenWeather covers nine API-call types and Cat API covers six query types.Source: §A.1 · p. 15, §A.2 · p. 15

A.3 Home Search

  • Home Search evaluates 100 queries over 15 functions, requiring location and buy-or-rent calls before criteria and search; three criterion functions are withheld from demonstrations.Executable searches are scored by F1 against the ground-truth criteria.Source: §A.3 · pp. 15–16

A.4 Trip Booking

  • Trip Booking tests 120 queries across transportation and hotel scenarios, where required calls and call ordering determine whether a search is executable.Some calls are optional; omitting them leads to an unsuccessful search.Source: §A.4 · p. 16

A.5 Google Sheets

  • Google Sheets tests common spreadsheet operations across 70 cases, counting a result as correct only when both cell values and formatting match.The setup provides 100 gspread calls and also encourages Pandas DataFrame and gspread-dataframe use.Source: §A.5 · p. 16

A.6 Virtual Home

  • VirtualHome uses 100 held-out activities from 183 unique tasks and evaluates action executability plus normalized longest-common-subsequence similarity to reference programs.The remaining 83 tasks supply demonstration examples; the task API has 40 action functions.Source: §A.6 · p. 16

A.7 WebShop

  • WebShop provides 100 test instructions in simple mode with 1,000 random products, defining success as a positively rewarded purchase within 25 steps.Demonstrations come from human trajectories, with a shortened set of 200 trajectories for input-length constraints.Source: §A.7 · pp. 16–17

A.8 Tabletop

  • Tabletop evaluates generated pick-and-place programs across 15 task types, with success requiring all objects to reach their target positions within a small threshold.Test setups involve up to four colored blocks and four bowls.Source: §A.8 · p. 17

B Comprehensive Model Evaluation on the ToolBench; B.1 Models

  • The broader evaluation examines 27 models in a 3-shot setting, including representative closed and open models; some model families are excluded when their tokenizers hinder executable code generation.The evaluated model descriptions cover families including GPT, StarCoder, LLaMA, OPT-IML, and NeoX-derived models.Source: B Comprehensive Model Evaluation on… · p. 17, §B.1 · pp. 17–18

B.2 Evaluation

The section describes ToolBench evaluation and baseline patterns, then details how alignment data are created, how all-shot loss is applied, and how models are trained.

B.2 Evaluation

  • Baseline evaluation uses task-specific prompts and API retrieval, executes generated calls against ground truth or environments, and scores only the top sampled action.Models receive sufficient generation tokens; conversation-oriented models also use prompt markers matching their training format.Source: §B.2 · pp. 19–20

B.3 ToolBench performance of different models

  • Open-source models lag GPT-4 and GPT-3.5 especially on challenging Google Sheets and Tabletop tasks, despite competitiveness on simpler tasks.Source: §B.3 · p. 20
  • Instruction-tuned model variants show no significant ToolBench accuracy gains over their paired base models, while untuned LLaMA performs relatively well among public models.Source: §B.3 · p. 20
  • Larger models tend to perform better within the GPT, LLaMA, OPT, Pythia, and StableLM families when training-data quantity and quality are held comparable.Source: §B.3 · p. 20
  • StarCoder and CodeGen outperform similarly sized models, and CodeGen-16B-mono exceeds its base and multi-language variants on almost all tasks.StarCoderBase is reported as on par with LLaMA-65B, which is more than four times larger.Source: §B.3 · p. 20

C.1 Training data

  • Training data for four tasks expand demonstration examples into templates with variable values; the other four use formatted demonstrations, with test overlap minimized but unavoidable for Open Weather.Source: §C.1 · p. 20

C.2 All-shot loss

  • All-shot loss trains API-call generation for every goal in concatenated examples and empirically improves success over last-goal-only loss, especially with demonstrations.Table 12 reports mean (standard deviation) performance across technique combinations.Source: §C.2 · pp. 20, 22, Figure 6 · p. 22, Table 12 · p. 22

C.3 Training details

  • Models are fine-tuned for eight epochs on shuffled task data, using 2048-token sequences, batch size 16, and a constant 1e-5 learning rate.Training uses four 80GB A100 GPUs, and validation accuracy selects the checkpoint.Source: §C.3 · p. 22

C.4 Extended results for Section 6

The appendix reports extended technique results and specifies how task complexity is estimated from the likelihood that tests derive from demonstrations, alongside a reversed-F1 comparison measure.

C.4 Extended results for Section 6

  • Table 12 reports mean and standard deviation for model performance across combinations of the three proposed techniques; the authors say randomness barely changes final results.Source: §C.4 · p. 23

D.1.1 The likelihood of a test being derived from an example

  • Test-to-example likelihood assumes each example API call is retained or deleted with equal probability and each correct uncovered call is inserted with probability 1/|D|.Repeated calls to the same API are treated as distinct, and API-call order is ignored.Source: §D.1.1 · p. 23

D.1.2 The distance between a test and example pair

  • The pairwise distance is the logarithm of the reciprocal test-generation likelihood, making larger values correspond to greater complexity while compressing magnitude differences.Source: §D.1.2 · pp. 23–24

D.1.3 Complexity score of a task

  • Task complexity is a score based on tests, demonstration examples, and API functions; it ranges from zero to infinity, with higher scores indicating harder API selection.The score is computed for original and alignment-training sets, which share tests and API functions but differ in examples.Source: §D.1.3 · p. 24

D.2.1 Computation details

  • For computation, Open Weather and The Cat API treat each valid URL with parameters as a distinct API option; WebShop receives complexity zero because its examples cover both available APIs.Tabletop examples are divided into 65 single-step samples because test cases lack predefined correct answers.Source: §D.2.1 · p. 24

D.2.2 Reversed-F1 Score

  • The comparison measure uses reversed F1 distance between each test-example pair, scaled by 100 to match the complexity score’s range.Source: §D.2.2 · p. 24

D.2.3 Measurements

The measurements compare task complexity and reversed-F1 scores with error rates across five tasks. Complexity scores correlate near-perfectly with error rates for three models, though GPT-4’s low-complexity results are sensitive to experimental randomness.

D.2.3 Measurements

  • Near-perfect Spearman correlation links complexity score with error rate for LLaMA-30b, CodeGen-16b, and StarCoder across the evaluated tasks.Figure 7 and Table 13 report the separate complexity/error-rate and reversed-F1/error-rate comparisons.Source: §D.2.3 · p. 25, Figure 7 · p. 25, Table 13 · p. 25
  • For GPT-4, complexity-score correlation is 0.2 because near-perfect accuracy on low-complexity tasks makes SCC sensitive to experimental randomness.The cited low-complexity tasks have complexity below 12 and accuracy above 93%.Source: §D.2.3 · p. 25
  • Despite GPT-4’s sensitivity, the complexity score is reported as a more effective indicator of task difficulty than reversed F1.Source: §D.2.3 · p. 25
  • The authors interpret high SCC values as empirical support that the score captures benchmark difficulty across models and tasks.Source: §D.2.3 · p. 25

Related papers

Loading 2305.16504v1…