PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models

Zhengliang Shi, Yuhan Wang, Lingyong Yan, Pengjie Ren, Shuaiqiang Wang, Dawei Yin, Zhaochun Ren

arXiv:2503.01763v2cs.CLcs.AIcs.IR

TL;DR

Tool retrieval is important for LLM agents, but existing benchmarks often preselect a small set of tools and retrieval performance across diverse scenarios remains unclear. The paper introduces TOOLRET and a large training set for tool retrieval. Even strong conventional-IR models struggle on TOOLRET, while low retrieval quality is associated with lower end-to-end task pass rates.

  • Problem

    Existing tool-use benchmarks often manually preselect a small set of relevant tools, leaving tool retrieval across diverse scenarios underexplored despite its importance for LLM agents.

    Source: §1 · p. 1

  • Method

    The paper constructs TOOLRET from collected query-tool datasets and introduces a large-scale training set for tool retrieval.

    Source: §1 · p. 2, §9 · p. 9

  • Results

    33.83 nDCG@10 for the best model, NV-embedd-v1, shows that even strong conventional-IR models struggle on TOOLRET, and low retrieval quality significantly degrades tool-use LLM task pass rates.

    Source: §1 · p. 2, §9 · p. 9

  • Takeaways & Limitations

    The authors report that TOOLRET-train enhances IR-model performance and their ability to augment tool-use LLMs.

    Source: §1 · p. 2

    The benchmark is limited to English-language text retrieval, and prompt sensitivity has not been sufficiently investigated.

    Source: Limitation · p. 10

Abstract

from arXiv · show

Tool learning aims to augment large language models (LLMs) with diverse tools, enabling them to act as agents for solving practical tasks. Due to the limited context length of tool-using LLMs, adopting information retrieval (IR) models to select useful tools from large toolsets is a critical initial step. However, the performance of IR models in tool retrieval tasks remains underexplored and unclear. Most tool-use benchmarks simplify this step by manually pre-annotating a small set of relevant tools for each task, which is far from the real-world scenarios. In this paper, we propose ToolRet, a heterogeneous tool retrieval benchmark comprising 7.6k diverse retrieval tasks, and a corpus of 43k tools, collected from existing datasets. We benchmark six types of models on ToolRet. Surprisingly, even the models with strong performance in conventional IR benchmarks, exhibit poor performance on ToolRet. This low retrieval quality degrades the task pass rate of tool-use LLMs. As a further step, we contribute a large-scale training dataset with over 200k instances, which substantially optimizes the tool retrieval ability of IR models.

1 Introduction

Tool retrieval is a critical step for LLM agents, yet existing evaluations often preselect small toolsets rather than testing retrieval across large, similar tool collections. TOOLRET evaluates this gap, finding that strong conventional IR models struggle, while training on TOOLRET-train improves retrieval and end-to-end task pass rates.

1 Introduction

  • Most existing tool-use benchmarks manually preselect about 10–20 relevant tools per task, unlike large, functionally similar real-world toolsets.In a ToolBench pilot, agent performance dropped when officially annotated tools were replaced by retrieved tools, and even strong retrievers struggled to find targets.Source: §1 · p. 1
  • TOOLRET is introduced as a large-scale benchmark with 7.6k diverse retrieval tasks and a corpus of 43k tools.It aggregates query-tool datasets, standardizes them as retrieval tasks, and supports instructional retrieval with target-aware instructions.Source: §1 · pp. 1–2
  • 33.83 nDCG@10 for the best model, NV-embedd-v1, shows that even strong conventional-IR models struggle on TOOLRET.The paper identifies low query–tool term overlap and a shift from conventional information-seeking tasks as factors in the performance gap.Source: §1 · p. 2
  • TOOLRET-train contains more than 200k retrieval tasks, and models trained on it improve retrieval and end-to-end task pass rates with tool-use LLMs.Training examples pair tasks with instructions, target tools, and negative tools.Source: §1 · p. 2

2 Related work

Prior tool-learning approaches struggle to handle large, frequently updated toolsets within LLM context limits, while tool retrieval remains underexplored. Conventional IR benchmarks and ad-hoc retrievers do not provide systematic evaluation across diverse tool-retrieval scenarios.

Tool learning with foundation models

  • Tool-learning approaches that prepend tool descriptions or train models on synthetic data both face challenges with massive, frequently updated toolsets and limited LLM context.The cited examples include platforms with over 52k tools and package repositories with over 600k packages.Source: §2 · p. 2
  • Semantic retrievers have been used for tool selection, but existing solutions are typically ad hoc and lack systematic evaluation across diverse retrieval scenarios.The passage notes that retraining LLMs to memorize frequently updated tools is cost-intensive.Source: §2 · p. 2

Information retrieval benchmark

  • Tool retrieval remains underexplored in IR research, despite its greater complexity when tasks require combining multiple tools and benchmarks often preselect a small tool set.The paper positions its diverse evaluation tasks and over 200k training examples as addressing this gap.Source: §2 · p. 3

3 Benchmark construction

The benchmark aggregates and standardizes tool-use datasets, samples tasks and tools to manage imbalance and redundancy, and pairs retrieval tasks with generated instructions. The resulting TOOLRET has 7.6k tasks and a corpus of 43k tools.

3.1 Data collection

  • The authors collect more than 30 datasets released from August 2023 to December 2024, clean and normalize them, and align tasks as queries with target tools as labels.Sources include tool-use benchmarks, conference resources, and public datasets; tools have unique identifiers and documentation.Source: §3.1 · p. 3

3.2 Data sampling

  • TOOLRET contains 7.6k retrieval tasks and a corpus of 43k tools after sampling tasks and merging overlapping toolsets.Tasks are clustered by text embeddings and sampled; tool documentation is reviewed to identify and merge identical toolsets.Source: §3.2 · p. 3

3.3 Instruction construction

  • For instructional retrieval, three experts craft 100 seed instructions, then GPT-4o generates an instruction for each task using in-context examples.The instructions bridge query intent and target-tool functionality; the resulting benchmark pairs each of its 7.6k tasks with an instruction.Source: §3.3 · pp. 3–4

4 Benchmark statistic

TOOLRET characterizes tool-retrieval tasks by their documentation formats, complexity, length, instruction quality, and diversity. Its queries have low lexical overlap with target tools, while generated instructions are reviewed by experts and often differ from the seed instructions.

4 Benchmark statistic

  • TOOLRET groups tool documentation into code snippets, structured Web APIs, and free-form descriptions of customized applications.Source: §4 · p. 4

4.1 Complexity

  • Low query–tool lexical overlap and a higher number of target tools make TOOLRET more challenging than conventional IR benchmarks.The comparison includes HotpotQA and MTEB; the lower overlap suggests greater reliance on semantic representation than lexical matching.Source: §4.1 · p. 4, Figure 2 · p. 4

4.2 Length statistics

4.3 Quality

  • 89.2% of generated instructions correctly cover target-tool features and remain grounded in the original queries; experts revised the remaining 10.8%.Five experts reviewed instruction quality; Table 3 reports a Kappa statistic of 0.743.Source: §4.3 · p. 5, Table 3 · p. 5

4.4 Instruction diversity

  • Many generated instructions have low ROUGE-L overlap with the 100 handcrafted seed instructions.Each generated instruction is compared with its highest-overlap seed.Source: §4.4 · p. 5, Figure 4 · p. 5

5 Benchmark evaluation setup

The evaluation measures ranking quality, target-tool retrieval, retrieval accuracy, and completeness, and compares query-only retrieval with retrieval using an added instruction. It covers sparse, dense, embedding, reranking, and LLM-agent approaches.

5.1 Evaluation protocol

  • The protocol uses NDCG@K, Recall@K, Precision@K, and Completeness@K to assess ranking, target retrieval, retrieved-tool accuracy, and whether all target tools appear in the top K.Completeness@K is 1 only when all target tools are retrieved within the top K.Source: §5.1 · p. 5
  • Models are evaluated with either the query alone or the query concatenated with an instruction; the comparison isolates the effect of adding instructions.Source: §5.1 · p. 5

5.2 Model to Evaluate

  • The evaluated approaches include sparse lexical retrieval, single- and multi-task dense models, cross-encoder rerankers, and zero-shot LLM reranking.The LLM and reranking baselines use NV-Embed-v1 to retrieve their initial tools.Source: §5.2 · pp. 5–6

6 Experiment result

Existing retrievers perform poorly on TOOLRET, and common reranking methods offer limited or negative gains. Adding instructions improves retrieval, while TOOLRET scores remain lower than conventional IR scores despite a positive correlation between the benchmarks.

6.1 Tool retrieval performance

  • Every evaluated retriever scores below 35% on Completeness@10 and below 52% on Recall@10; even NV-Embed-v1 remains below 45% on Completeness@10.ColBERT underperforms simple lexical BM25; the paper identifies low query–target overlap and domain shift as potential reasons for the performance gap.Source: §6.1 · p. 6

6.3 Compare with conventional IR tasks

  • TOOLRET and MTEB scores have Pearson’s coefficient β = 0.790, but TOOLRET scores are lower; conventional relevance-trained models such as Contriever perform poorly on TOOLRET.The authors interpret this pattern as indicating a need for more target-aware reasoning.Source: Figure 5 · p. 6, §6.3 · p. 7

6.1 Tool retrieval performance

  • MonoT5 reranking lowers NV-Embed-v1's average NDCG@10 from 33.83 to 28.92, while bgeranker-v2-gemma improves Completeness@10 by 4.7%.Other commonly used rerankers provide limited or negative improvements.Source: §6.1 · p. 7

6.2 Substantial gains from instruction

  • All IR models perform better when given an additional instruction, with the clearest gains reported for instruction-tuned embeddings such as NV-embed-v1 and e5-mistral.Source: §6.2 · p. 7

7 Retrieval affect downstream task

Replacing oracle tools with IR-retrieved tools reduces tool-use agents’ pass rates on ToolBench. TOOLRET-train provides instructional retrieval data, and models trained on it improve both retrieval performance and downstream pass rates.

7.1 Poor retrieval leads to poor tool-use agents

  • In ToolBench-G1, GPT-3.5 achieves a pass rate of 50.60 with bge-large-retrieved tools, 11.40 lower than with oracle tools.Figure 6 plots IR retrieval performance against the end-to-end pass rate of tool-use LLMs.Source: §7.1 · p. 7, Figure 6 · p. 8

7.2 Towards better retrieval

  • TOOLRET-train contains over 200k instances from ToolACE, ToolBench, and APIGen, pairing queries with target tools and instructions.Training examples use 10 negative tools retrieved by NV-embed-v1; the model is optimized to maximize the likelihood of target tools.Source: §7.2 · p. 8
  • All IR models trained on TOOLRET-train improve NDCG@10, and GPT-3.5 and ToolLlama achieve substantial pass-rate gains with the improved retrievers.Source: §7.2 · p. 8
  • Removing instructions from the training loss improves retrieval over non-tuned models but underperforms instruction-tuned counterparts.The paper presents this ablation as evidence for the effectiveness of its instructional training data.Source: §7.2 · p. 8

8 Discussion

The discussion addresses evaluation concerns from overlapping tools in a merged corpus, explains its format-based taxonomy, and proposes further retrieval-task dimensions. It emphasizes selecting semantically and functionally appropriate tools and distinguishing the most suitable tool among similar alternatives.

8 Discussion

  • Overlapping functionality among tools merged from existing benchmarks may make evaluations unreliable.The authors argue that the benchmark treats the original dataset’s ground truth as the most appropriate tool and expects fine-grained discrimination.Source: §8 · pp. 8–9
  • The benchmark targets semantic and functional tool matching rather than similarity of surface forms.The authors frame this as choosing the most appropriate tool when multiple functionally similar tools exist.Source: §8 · p. 9
  • The format-based taxonomy divides TOOLRET into Web APIs, Code Functions, and Customized Applications.The authors say these categories reflect tool structures across diverse datasets and support analysis across tool representations.Source: §8 · p. 9
  • The authors propose adding query length, number of target tools, and query–documentation lexical overlap as evaluation dimensions.The overlap categories are low (<0.05), medium (0.05–0.2), and high (≥0.2); lower overlap indicates greater need for deep semantic matching.Source: §8 · p. 9

9 Conclusion

TOOLRET benchmarks tool retrieval at scale and finds that strong conventional-IR models still struggle on this task, with retrieval quality affecting tool-use LLM pass rates. The authors also report that training on TOOLRET-train improves retrieval and raises pass rates by 10%-20%.

9 Conclusion

  • Training IR models on TOOLRET-train substantially improves retrieval and raises tool-use LLM pass rates by 10%-20%.TOOLRET-train contains over 200k retrieval tasks.Source: §9 · p. 9
  • Even models strong on conventional IR benchmarks struggle on TOOLRET, and low retrieval quality degrades tool-use LLM task pass rates.The benchmark comprises 7.6k queries, each paired with an instruction, and a corpus of 43k tools.Source: §9 · p. 9
  • TOOLRET is a heterogeneous tool-retrieval benchmark built by aggregating existing tool-use datasets into a unified format.The benchmark contains 7.6k queries and 43k tools.Source: §9 · p. 9

Limitation

The paper’s limitations include untested multilingual retrieval and insufficient investigation of prompt sensitivity. Its evaluation also focuses on retrieval once before tool use, while interleaved retrieval-and-calling is left for future work.

Limitation

  • The benchmark is limited to English text retrieval and does not investigate multilingual retrieval or prompt sensitivity sufficiently.The authors plan to expand language coverage and annotate more instruction variants to study prompt phrasing.Source: Limitation · p. 10
  • The authors identify instruction sensitivity as an area requiring further study across prompt formulations and instruction styles.They note LLM sensitivity to prompt wording and propose annotating a broader range of instructions.Source: Limitation · p. 10
  • The paper evaluates retrieval-then-calling, with retrieval triggered once before the top-k tools are passed to the tool-use LLM.Interleaved retrieval-and-calling, where the IR model can receive the LLM’s reasoning output, is proposed for future work.Source: Limitation · p. 10

Ethics Statement

This section documents TOOLRET’s purpose, composition, task and instruction formats, and ethical safeguards, including human review of generated instructions. It also describes TOOLRET-train construction and the retrieval-model baselines evaluated on the benchmark.

Ethics Statement

  • The authors say they follow the ACM Code of Ethics, use publicly available resources and licensed datasets, prioritize reproducibility, and seek to avoid harm, deception, or misuse.They note that LLMs can generate harmful, offensive, or biased content, and report experiments with both commercial and open-source LLMs.Source: Ethics Statement · p. 10

A Data Card

  • TOOLRET reveals that IR models strong on conventional benchmarks still struggle with tool retrieval, and that poor retrieval quality degrades tool-use LLMs’ end-to-end task pass rate.The datacard presents TOOLRET as a comprehensive, fair benchmark to facilitate tool-retrieval model development.Source: A Data Card · p. 15

B.1 Dataset collections

  • TOOLRET unifies heterogeneous tool-use datasets into three subsets—web APIs, code functions, and customized apps—to support fine-grained evaluation across differing tool formats and retrieval difficulty.The benchmark aligns the collected datasets in a format modeled on conventional IR benchmarks.Source: §B.1 · p. 15

B.2 TOOLRET-Web

  • The TOOLRET-Web subset integrates datasets containing Web API tools for retrieval tasks.The collected APIs span varied tool-use scenarios and domains.Source: §B.1 · p. 15, §B.2 · p. 16

B.3 TOOLRET-Code

  • The TOOLRET-Code subset integrates datasets in which code functions serve as tools, including functions for deep-learning tasks and other operations.The collected functions include PyTorch, TensorFlow, and Hugging Face model tools.Source: §B.3 · pp. 16–17

B.4 TOOLRET-Customized

  • The customized-app subset uses tools documented in free-form natural language rather than structured formats and incorporates datasets from multiple sources.Its collected datasets include ToolACE, GPT4Tools, TaskBench, ToolAlpaca, ToolBench-sam, ToolEmu, and TooLink.Source: §B.4 · pp. 17–18

B.5 Task format

  • TOOLRET reformats test cases into a unified IR-style format, pairing each task’s query and instruction with labeled target tools, each identified and documented by functionality.The benchmark is intended to support tool-retrieval evaluation in a format similar to BEIR and MTEB.Source: §B.5 · p. 18

B.6 Details of instruction construction

  • Three NLP and IR experts craft seed instructions; GPT-4o samples from the evolving pool to generate a task-tailored instruction, then adds it back to increase instruction diversity.The generated instruction is intended to emphasize query requirements and target-tool characteristics to aid retrieval.Source: §B.6 · p. 19

B.7 Human annotation

  • Human annotators assess generated instructions for hallucinations, tool-feature comprehensiveness and accuracy, and relevance to the input query.They mark instructions meeting all four criteria as correct; otherwise, they explain the issue and revise the instruction to ensure benchmark instructions are grounded in queries and target tools.Source: §B.7 · p. 20
  • Instructions judged incorrect receive a brief explanation and manual revision to improve accuracy and clarity.The process is presented as a means of ensuring high-quality instructions faithfully grounded in the original queries and target tools.Source: §B.7 · p. 20

C Large-scaling training dataset: TOOLRET-train

D.1 Baselines

  • The evaluation includes sparse BM25s and single-task dense retrievers trained on MS-MARCO, alongside COLT, trained on an ad-hoc tool-retrieval dataset.The dense baselines include gtr, contriever, and colbertv2.0.Source: §D.1 · p. 21
  • The evaluated baselines also include multi-task embedding models, cross-encoder re-rankers, and zero-shot LLM agents for re-ranking.LLM agent and re-ranking baselines use NV-embedd-v1 to retrieve their initial tools.Source: §D.1 · p. 23

D.2 Compare with conventional IR tasks

Across comparisons with conventional IR and controlled retrieval settings, TOOLRET reveals challenges for existing models; training on TOOLRET-train and improving retrieval are associated with better tool-use task pass rates. The experiments compare performance across conventional benchmarks, dataset-specific toolsets, and TOOLRET subsets.

D.2 Compare with conventional IR tasks

  • Pearson’s correlation between TOOLRET and MTEB retrieval scores is β = 0.790, yet TOOLRET scores are consistently lower than conventional IR scores.Figure 7 presents the benchmark correlation.Source: §D.2 · pp. 23–24, Figure 7 · p. 24
  • IR models trained with relevance-oriented criteria, including Contriever, degrade substantially on TOOLRET, whose multiple potential targets and lower query-tool term overlap demand target-aware reasoning.The paper describes these characteristics as making retrieval more complex and requiring stronger reasoning and understanding capabilities.Source: §D.2 · pp. 24–25

D.3 Results of controlled experiment

  • Controlled experiments evaluate retrieval within each integrated dataset’s toolset, separately for query-only and query-plus-instruction inputs.The request includes Table 10 and Table 11 captions, but no outcome values for these experiments.Source: Table 10 · p. 24, §D.3 · p. 25, Table 11 · p. 25

D.4 Results of in-subset retrieval

  • In-subset experiments restrict retrieval to the toolsets of TOOLRET-web, TOOLRET-code, or TOOLRET-customized, with query-only and query-plus-instruction settings.The passages specify experiment scope and tables, but do not report the outcome values.Source: §D.4 · p. 25, Table 12 · p. 26, Table 13 · p. 27

D.5 Results of trained IR mdels

  • TOOLRET-train contains more than 200k retrieval tasks, and trained IR models are evaluated on TOOLRET after learning to distinguish target tools from negative tools.The supplied passages state the dataset scale and training/evaluation design, but not the Table 14 results.Source: §D.5 · pp. 25–26, Table 14 · p. 27

D.6 Improved IR enhances tool-use LLMs

  • Improved IR models yield substantial gains in tool-use LLM pass rates, with Figure 6 showing a positive correlation between retrieval performance and task pass rate.The evaluation replaces ToolBench’s pre-annotated tools with tools retrieved from TOOLRET’s 43,000-tool corpus.Source: §D.6 · p. 26, Table 15 · p. 28

Related papers

Loading 2503.01763v2…