PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error

Boshi Wang, Hao Fang, Jason Eisner, Benjamin Van Durme, Yu Su

arXiv:2403.04746v1cs.CLcs.AIcs.LG

TL;DR

Existing LLMs show limited correctness when using tools, despite tools’ role in obtaining up-to-date information and taking actions. STE combines simulated tool-use scenarios, execution feedback, and memory; on ToolBench, it improves Mistral-Instruct-7B and supports continual tool learning through experience replay.

  • Problem

    Existing LLMs remain far from reliable tool use, despite tools’ role in acquiring up-to-date information and taking consequential actions.

    Source: §1 · pp. 1–2

  • Method

    STE has an LLM imagine plausible tool-use scenarios, learn from API execution feedback, and use short- and long-term memory to improve exploration.

    Source: §1 · p. 2

  • Results

    Mistral-Instruct-7B reaches 76.8% correctness with STE, a 46.7% absolute boost, and outperforms GPT-4 with ICL.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Simple rehearsal-based fine-tuning could enable continual learning of new tools while preserving previously acquired skills.

    Source: §1 · p. 3, §6 · p. 9

    Positive tool-use examples alone make it difficult to teach models when not to use a tool.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Tools are essential for large language models (LLMs) to acquire up-to-date information and take consequential actions in external environments. Existing work on tool-augmented LLMs primarily focuses on the broad coverage of tools and the flexibility of adding new tools. However, a critical aspect that has surprisingly been understudied is simply how accurately an LLM uses tools for which it has been trained. We find that existing LLMs, including GPT-4 and open-source LLMs specifically fine-tuned for tool use, only reach a correctness rate in the range of 30% to 60%, far from reliable use in practice. We propose a biologically inspired method for tool-augmented LLMs, simulated trial and error (STE), that orchestrates three key mechanisms for successful tool use behaviors in the biological system: trial and error, imagination, and memory. Specifically, STE leverages an LLM's 'imagination' to simulate plausible scenarios for using a tool, after which the LLM interacts with the tool to learn from its execution feedback. Both short-term and long-term memory are employed to improve the depth and breadth of the exploration, respectively. Comprehensive experiments on ToolBench show that STE substantially improves tool learning for LLMs under both in-context learning and fine-tuning settings, bringing a boost of 46.7% to Mistral-Instruct-7B and enabling it to outperform GPT-4. We also show effective continual learning of tools via a simple experience replay strategy.

1 Introduction

The introduction identifies accurate use of already-trained tools as an understudied need, given that tool calls can enable consequential actions. It proposes simulated trial and error (STE), which combines imagined scenarios, tool interaction, and memory, and reports gains in tool-use performance and continual learning.

  • Prior tool-augmented LLM work emphasizes tool coverage and flexibility, while much existing work on fine-tuning targets generalization to unseen tools rather than accuracy on trained tools.Source: §1 · p. 1
  • Accuracy on tools for which an LLM was trained is understudied, although tool calls can enable consequential actions and errors may cause harmful outcomes or undermine trust.Source: §1 · p. 1
  • STE draws on biological tool learning by combining trial and error, imagined scenarios, and short- and long-term memory in a method for tool-augmented LLMs.The LLM imagines plausible instructions, interacts with the API and observes feedback, then reflects; exploration examples can support ICL or fine-tuning.Source: §1 · p. 2
  • Mistral-Instruct-7B reaches 76.8% correctness with STE, a 46.7% absolute boost, outperforming GPT-4 with ICL; rehearsal also supports continual tool learning while preserving prior skills.Source: §1 · pp. 2–3

2 Simulated Trial and Error

STE first explores each API through imagined queries, tool interactions, and reflection, using memory to improve exploration; it then turns the resulting trials into examples for fine-tuning or in-context learning.

2.1 Exploration

  • For each API, exploration uses a budget of trials in which the LLM imagines a relevant query, interacts with the API, and reflects to support subsequent exploration.Source: §2.1 · p. 3
  • Execution feedback supports iterative correction: the LLM alternates thought, API action, and observation, then answers the query and reflects on whether it succeeded.The feedback can help correct syntactic and semantic errors in API calls.Source: §2.1 · p. 3
  • Short-term memory retains recent fine-grained trajectories within an episode, while long-term memory stores distilled past queries and outcomes to encourage exploration of less explored scenarios.Source: §2.1 · pp. 3–4

2.2 Exploitation

  • For exploitation, STE extracts queries, final API calls and results, and responses from trials, filters examples for validity, and paraphrases them for each API.The examples are used either for fine-tuning or as demonstrations retrieved for in-context learning.Source: §2.2 · p. 4

3 Experimental Setup

The experiments use selected ToolBench APIs, several LLM baselines, and tool-use metrics focused on API-call correctness; the setup also notes a constraint on evaluating query success with dynamic real-world information.

  • The study uses 50 free, low-latency APIs from ToolBench, spanning search, domain-specific information-seeking, and problem-solving tools.Source: §3 · p. 4
  • Exploration uses ChatGPT for 15 episodes of four trials per API, yielding 60 pre-filtering examples per API and about 7,000 examples after filtering and paraphrasing.Each trial allows at most four API calls; 15 examples per API are selected for testing and manually checked.Source: §3 · p. 4
  • Baselines include Llama-2-Chat, Mistral-Instruct, GPT-3.5-turbo, GPT-4, and ToolLLaMA-v2; ICL uses eight nearest examples, with an oracle API retriever for models whose context is too short.Fine-tuned STE models do not need API documentation in context.Source: §3 · pp. 4–5
  • Correctness measures API name and arguments, with separate metrics for well-formed calls and API matching; evaluating query success is left to future work because many APIs return dynamic real-world information.Source: §3 · p. 5

4 Results

STE improves tool-use performance under both in-context learning and fine-tuning. Ablations identify execution feedback and both memory mechanisms as useful, while continual-learning experiments show rehearsal largely mitigates forgetting.

4.1 Effectiveness of STE

  • 46.7%: STE fine-tuning boosts Mistral-Instruct-7B in correctness and enables it to outperform GPT-4.Source: §4.1 · p. 6

4.2 Ablation Studies

  • Without execution feedback, exploration produces ill-formed examples; short- and long-term memory support effective trial and error, and reflection helps maintain informative long-term memory.Source: §4.2 · p. 6
  • 78.3%→51.7%: disabling short-term memory lowers the share of positive tool-use examples; with it, forecast-weather queries cover 16 attributes more specifically and comprehensively.Source: §4.2 · p. 6
  • Without long-term memory, only 71.7% of trials concern distinct subjects, with a more skewed attribute distribution; long-term memory maintains exploration diversity across episodes.Source: §4.2 · p. 6

4.3 Error Analysis

  • In sampled GPT-4 errors, STE examples reduce wrong-API, missing-or-wrong-argument, and hard-to-evaluate cases from 36.7%, 26.7%, and 36.7% to 19.0%, 10.0%, and 16.7%, respectively.Source: §4.3 · p. 6

4.4 Continual Tool Learning

  • Rehearsal largely mitigates catastrophic forgetting as tools arrive in four batches, allowing continual learning while preserving previously acquired skills.Source: Table 3 · p. 7, §4.4 · p. 8

4.3 Error Analysis

  • For the most performant fine-tuned model, Mistral-Instruct-7B, notable error causes were commonsense/world knowledge (47.4%), language understanding (31.6%), and grounding (21.1%).Around one-third of error examples were difficult to evaluate because overlapping tool functionalities can make ground truth non-unique and time-sensitive tools can prevent consistent ground truths.Source: §4.3 · p. 7

5 Related Work

Related work studies expanding tool access through in-context learning and fine-tuning, as well as feedback-based improvement, synthesized training data, and dynamic memory. STE builds on these lines of work with trial-and-error tool learning and short- and long-term memory.

Tool-augmented language models

  • Tool-augmented LLMs use fine-tuning on tool-specific demonstrations or in-context tool descriptions and examples; this work focuses on improving tool-use abilities.Prior work also explores trained tool embeddings and API-description use.Source: §5 · p. 8

LLMs can learn from feedback

  • Feedback research finds that LLMs can improve or correct predictions, and this work uses tool-execution feedback and self-reflection for progressive tool learning.The cited studies include Shinn et al. (2023), Madaan et al. (2023), and others.Source: §5 · p. 8

Data synthesis & bootstrapping with LLMs

  • LLM-synthesized data has been used for tool-specific training, and this approach aims to make synthesized tool-use examples more comprehensive and diverse.The passage situates this within broader dataset synthesis and bootstrapping work.Source: §5 · p. 8

Augmenting models with dynamic memory

  • Dynamic-memory research includes growing stores of user and environment feedback; STE uses fine-grained short-term and distilled long-term memory for progressive tool learning.The passage cites prior memory work including Shinn et al. (2023) and Zhong et al. (2023).Source: §5 · p. 8

6 Conclusions

The paper proposes simulated trial and error (STE), a progressive memory-based tool-learning method motivated by how humans master tools through continual interaction and reinforcement. ToolBench experiments report its effectiveness and show rehearsal-based fine-tuning can support continual learning of new tools while preserving previous skills.

  • STE is a progressive memory-based tool-learning method motivated by how humans master tools through continual interaction and reinforcement.Source: §6 · p. 9
  • Experiments on APIs drawn from ToolBench show the effectiveness of the proposed method.Source: §6 · p. 9
  • Rehearsal-based fine-tuning could enable continual learning of new tools while preserving previous skills.Source: §6 · p. 9

Limitations

The paper identifies limits and open directions involving exploration, tool composition, memory capacity, unlearning, and example-based teaching of when not to use a tool. In particular, positive tool-use examples alone make that behavior difficult to teach.

  • The augmented memory is limited by the LLM's context length, with retrieval modules or hierarchical and compressed representations suggested for scaling it.Source: Limitations · p. 9
  • Composing or planning multiple tool calls is orthogonal to the paper's focus on learning individual tools.Source: Limitations · p. 9
  • The paper explores continual learning of new tools but identifies unlearning tools that become unloaded or outdated as an important open problem.Source: Limitations · p. 9
  • The exploration-exploitation process could be made iterative, gradually reducing reliance on strong models as the enhanced models improve.Source: Limitations · p. 9
  • Example-based tool learning has difficulty teaching when not to use a tool through positive tool-use examples alone.The authors suggest negative examples or carrying relevant API parts alongside example-based training as future directions.Source: Limitations · p. 9

A API Selection

The study selects 50 APIs from BMTools and RapidAPI. BMTools APIs are manually selected, while RapidAPI candidates are filtered by access, rate limit, latency, and exploratory call success.

  • The study selects 50 APIs from ToolBench, drawing from BMTools and RapidAPI.Source: A API Selection · p. 14
  • For BMTools, the authors manually select 10 high-quality APIs.Source: A API Selection · p. 14
  • For RapidAPI, they retain free APIs, rank them by rate limit and latency, explore the top 400, and select 40 by API-call success rate.Source: A API Selection · p. 14

B Experimental Details

The fine-tuning setup specifies AdamW optimization and reports the rehearsal data source and hardware used.

Fine-tuning

  • Fine-tuning used AdamW for one epoch, with learning rate 2 × 10^-5, batch size 64, and a cosine scheduler with 0.03 warmup ratio.The optimizer used β = (0.9, 0.999), ϵ = 10^-8, and no weight decay.Source: B Experimental Details · p. 14
  • Training examples were truncated to 2,048 tokens, and runs used four NVIDIA A100/A6000 GPUs.Source: B Experimental Details · p. 14

Flan-V2 data & rehearsal

C Example queries for all explored trials

The appendix lists explored forecast_weather queries across settings and reports that STE's memory mechanisms improve exploration diversity and comprehensiveness.

Exploration results

  • Long-term memory substantially improves the diversity and comprehensiveness of exploration, compared with STE without long-term memory.Figure 4 groups explored trials by the attribute inquired about.Source: C Example queries for all explored… · p. 14

Explored queries

Exploration results

D Error Examples of GPT-4 and Mistral-Instruct-7B

This section points readers to the appendix tables containing error examples for GPT-4 and fine-tuned Mistral-Instruct-7B.

GPT-4

Mistral-Instruct-7B

E Prompts

This appendix presents STE prompts for refinement, memory, filtering, and paraphrasing, alongside explored weather-query examples across locations, forecast horizons, requested details, and memory settings. It also reports that long-term memory broadens the attributes explored.

E Prompts

  • The appendix identifies prompts for self-refinement with execution feedback and short-term memory, long-term memory, example filtering, and query paraphrasing.These prompts are presented in Tables 10–13.Source: E Prompts · p. 14, Table 10 · p. 18, Table 11 · p. 19
  • Long-term memory substantially improves the diversity of attributes explored in simulated trials.Source: Figure 4 · p. 15
  • Table 14 lists explored trials for STE with both short- and long-term memory, including weather queries across locations, forecast attributes, and time ranges.Examples ask about UV index, sunrise and sunset, and rain forecasts.Source: E Prompts · pp. 20–21, Table 14
  • Other prompts ask about specific conditions and weather measures, including rain, cloudiness, temperature, visibility, humidity, and wind speed.The lists include both general forecasts and requests for hourly forecasts or chances of rain.Source: E Prompts · pp. 21–22
  • The prompt sets span cities and forecast horizons, from tomorrow or specific weekdays to multi-day forecasts, as illustrated in the explored-trial query lists.Examples include Sydney and Tokyo forecasts, with horizons ranging from three to ten days.Source: E Prompts · pp. 21–22, Table 15, Table 16
Loading 2403.04746v1…