PaperlayerView exact revision on arXiv

Source-linked AI summary

ToolSearcher: Optimizing Tool Selection at Scale via Reinforcement Learning

Zhenlong Dai, Xujie Song, Zitong Wang, Tong Niu, Jian liu, Weiqiang Wang, Xiu Tang, Sai Wu, Chang Yao, Jingyuan Chen

arXiv:2609.30906v1cs.CL

TL;DR

Large-scale tool selection is difficult because repositories are diverse, context limits make exhaustive documentation impractical, and tool composition requires compatibility reasoning. ToolSearcher combines category-constrained discrimination with event-level search modeling and trajectory-aligned credit allocation. It outperforms baselines on StableToolBench and AppWorld, while its synthesized training data and limited interaction rounds constrain aspects of the evaluation setting.

  • Problem

    Large-scale tool selection is underexplored: repositories contain diverse, similar tools that cannot all fit in context, and selection requires reasoning about compatible compositions.

    Source: §1 · p. 2

  • Method

    ToolSearcher uses category-constrained tool discrimination, event-level search modeling, and trajectory-aligned credit allocation to optimize multi-turn search and selection.

    Source: §1 · pp. 2–3

  • Results

    ToolSearcher outperforms reinforcement-learning baselines on StableToolBench and out-of-distribution AppWorld scenarios.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The reported results support ToolSearcher’s effectiveness in large-scale and out-of-distribution tool-selection settings.

    Source: §1 · p. 3, §6 · p. 10

    Synthesized training data mostly uses parallel tool combinations and lacks state-awareness, whereas AppWorld APIs are mostly stateful.

    Source: A Limitations · p. 14

Abstract

from arXiv · show

Large language models (LLMs) excel at natural language processing but struggle to interact with external environments. Tool learning provides a promising way to extend LLMs into actionable agents, where tool selection is a critical prerequisite for successful tool use. Existing work often assumes a small or predefined set of tools, leaving large-scale tool selection underexplored. Real-world repositories contain a vast and diverse array of tools, making it difficult for LLMs to effectively search, distinguish, and compose tools under context-length constraints. We identify large-scale tool selection as a new challenge for agentic reinforcement learning, highlighting that existing RL methods for knowledge-based question answering are inadequate for selecting tools while considering compatibility. To address this challenge, we propose ToolSearcher, a novel RL framework for effective multi-turn search and fine-grained optimization in large-scale tool selection. Specifically, we introduce category-constrained tool discrimination to improve the model's ability to distinguish functionally similar tools, event-level search modeling to explicitly optimize the discovery of target tools during multi-turn search, and trajectory-aligned credit allocation to provide fine-grained reward signals for different stages of the search-selection process. Extensive experiments on large-scale tool selection benchmarks demonstrate that ToolSearcher consistently outperforms a set of strong baselines in challenging settings involving iterative search and complex tool composition.

1 Introduction

Large-scale tool selection is underexplored despite repositories that are too large and diverse to fit fully in context. ToolSearcher addresses the challenge with multi-turn search and fine-grained learning signals for selecting compatible tool compositions.

1 Introduction

  • Tool learning includes tool selection, tool calling, and response generation, but prior work has mainly focused on calling rather than searching and selecting tools.Selection means finding appropriate candidates in a large, diverse repository for a task.Source: §1 · p. 2
  • Prior studies typically assume a predefined or small candidate set, while real repositories contain many diverse tools and cannot all be documented within context limits.Similar semantics can mask different functions, while similar functions can have different interfaces and constraints.Source: §1 · p. 2
  • Existing RL methods for knowledge-intensive search do not explicitly model tool search or the compatibility reasoning needed to compose tools.Tool-search trajectories must plan around a hypothesized tool chain and adjust that plan as results arrive; trajectory-level rewards also provide sparse supervision.Source: §1 · p. 2
  • ToolSearcher combines category-constrained discrimination, event-level search modeling, and trajectory-aligned credit allocation for large-scale tool selection.The components target distinguishing similar tools, optimizing discovery of previously unsearched target tools, and rewarding progress at different trajectory stages.Source: §1 · pp. 2–3
  • Experiments report that ToolSearcher consistently outperforms baselines on challenging benchmarks requiring iterative search and complex tool composition.The reported settings involve large-scale tool-selection benchmarks.Source: §1 · p. 3

2 Related Work

Related work covers retrieval-augmented generation and agentic reinforcement learning, while identifying large-scale tool selection as a distinct gap. Existing retrieval and RL approaches do not address the combined challenge of distinguishing similar tools and learning compatible combinations.

2.1 Retrieval-Augmented Generation

  • RAG typically retrieves information before generation, and later methods extend retrieval from single-document answers to multi-hop reasoning across small document sets.Top-k retrieval can limit corpus coverage; tool-integrated agents interleave reasoning and search actions but remain dependent on prompting and lack task-specific feedback.Source: §2.1 · p. 3

2.2 Agentic Reinforcement Learning

  • Agentic RL supports sequential decisions, and recent work applies it to multi-step retrieval, including global retrieval requiring broad document coverage.The cited approaches include GRPO [34], GSPO [52], GDPO [22], and MARAG-R1 [24].Source: §2.2 · p. 3
  • RL for large-scale tool selection remains largely unexplored, particularly for distinguishing similar tools and learning compatible tool combinations.This gap differs from optimizing retrieval over document subsets or broad document coverage.Source: §2.2 · p. 3

3 Method

ToolSearcher frames selection as finding a tool set for a user requirement and uses a search engine in a multi-turn RL process. Its methods constrain searches by category, optimize discovery events, and align rewards with progress through search and selection.

3.1 Task Definition

  • Tool selection seeks a tool set that satisfies a user requirement, with the model maximizing its conditional probability given the repository and search engine.Each tool document includes a function description and interface specifications for input constraints and output schema.Source: §3.1 · p. 4

3.2 Preliminary

  • The search-engine interface is included in the prompt, and returned results provide feedback for subsequent search rounds.The LLM issues structured tool calls, after which the search engine executes queries and returns retrieved results.Source: §3.2 · p. 4

3.3 Category-Constrained Tool Discrimination

  • Category-constrained discrimination changes global retrieval into search within a functionally similar subspace, then trains first on subcategory search before global search.The first phase uses 30% of the data to distinguish functionally similar tools; the second trains fine-grained semantic search across the repository.Source: §3.3 · pp. 4–5

3.4.1 Event-level Search Modeling.

  • Event-level search modeling represents a trajectory as search events and optimizes events that discover previously unsearched target tools.Events are tool calls paired with retrieved tool sets; retrieved tokens are loss-masked, and events that re-search already found tools are not optimized.Source: §3.4.1 · p. 5
  • Event-level advantages measure each event’s significance using target tools first found by that event, adjusting its gradient within the trajectory.The advantage is the maximum advantage among target tools first searched by the event.Source: §3.4.1 · p. 6

3.4.2 Trajectory-Aligned Credit Allocation.

  • Trajectory-aligned credit allocation focuses learning on unmastered steps by removing rewards for selection events tied to unfinished searches and for already-mastered search events.Rewards are aligned with progress across trajectories at different stages.Source: §3.4.2 · p. 6

4 Experimental Setup

The experiments evaluate tool selection on StableToolBench and AppWorld, compare retrieval and reinforcement-learning baselines, and specify the models and search settings used.

4.1 Datasets and Metrics.

  • StableToolBench covers 16k tools across 49 categories, while AppWorld provides an out-of-distribution evaluation environment.StableToolBench includes single-tool, intra-category multi-tool, and intra-collection multi-tool instructions; AppWorld includes 9 apps with 457 APIs and simulated activities for approximately 100 users.Source: §4.1 · p. 7

4.2 Baselines.

  • Comparisons include retrieval-augmented generation, multi-turn search, and several reinforcement-learning methods.The baselines include RAG, RAGSFT, Multi-turns, Search-R1, GSPO, GDPO, and MARAG-R1, with Qwen2.5-7B-Instruct and Qwen3-4B-Instruct used as search-engine-calling baselines.Source: §4.2 · p. 7

4.3 Implementation Details

  • Experiments use Qwen2.5-7B-Instruct and Qwen3-4B-Instruct, with up to 8 interaction turns and 5 rollouts per prompt.All multi-turn methods retrieve 5 tool documents per turn, and Qwen3-Embedding-0.6B is the search retriever.Source: §4.3 · p. 7

5 Experimental Analysis

ToolSearcher substantially improves tool selection on StableToolBench, including across scenarios and model scales, and maintains strong tool-selection and execution performance on AppWorld under distribution shift. Ablations report contributions from category-constrained discrimination, event-level search modeling, and trajectory-aligned credit allocation.

5.1 Main Results

  • On StableToolBench, ToolSearcher lifts Qwen2.5-7B-Instruct overall F1 from 9.8% to 51.3% and Match from 4.6% to 27.8% versus Multi-turns.It also exceeds GDPO by 1.7% in overall F1 and leads F1-Inst at 59.5%, 48.5%, and 29.4% on I1, I2, and I3.Source: §5.1 · pp. 7–8, Table 1 · p. 8

5.2 Tool-Selection Capabilities Across Diverse Scenarios

  • Across scenarios, ToolSearcher improves Qwen-4B search and final-selection recall (I3 Recall 26.91%) and raises Qwen-7B Recall by 43.08%, 39.58%, and 21.09% on I1, I2, and I3.For Qwen-7B, it slightly reduces SRecall while improving final selection.Source: Figure 2 · p. 8, §5.2 · p. 8

5.3 Generalization Capability to Out-of-Distribution

  • On AppWorld, ToolSearcher leads both model scales in F1, Recall, SRecall, TGC, and SGC; for Qwen2.5-7B-Instruct, TGC and SGC exceed GDPO by 5.7% each.It also outperforms all baselines in Difficulty-2, involving up to 8 unique APIs, where several baselines achieve near-zero SGC.Source: Table 2 · p. 9, §5.3 · p. 9

5.4 Ablation Study and Method Analysis

  • Removing any of CCTD, ESM, or TCA hurts overall performance; without TCA, StableToolBench F1 falls from 51.3% to 46.1% and SRecall from 68.0% to 53.8%.Removing ESM reduces SRecall by 22.4% on StableToolBench and 39.4% on AppWorld; CCTD's constrained search setting shows lower metrics than global search.Source: Table 3 · p. 9, §5.4 · pp. 9–10

6 Conclusion

ToolSearcher combines multi-turn search with category-constrained discrimination and trajectory-aligned credit allocation for large-scale tool selection. It outperforms reinforcement-learning baselines on StableToolBench and out-of-distribution AppWorld tasks involving iterative search and complex tool composition.

6 Conclusion

  • ToolSearcher outperforms several reinforcement-learning baselines on StableToolBench and out-of-distribution AppWorld tasks requiring iterative search and complex tool composition.Source: §6 · p. 10
  • Category-constrained discrimination and event-level search modeling help distinguish similar tools and optimize tool compositions through multi-turn search.Source: §6 · p. 10
  • Trajectory-aligned credit allocation provides fine-grained optimization by aligning rewards with progress relative to overall search-selection progress.Source: §6 · p. 10

A Limitations

ToolSearcher’s synthesized training data mostly represents parallel tool combinations, while AppWorld APIs are largely stateful. The method is also constrained by an eight-round training interaction limit, below AppWorld’s average API count.

A Limitations

  • Synthesized training data mostly contains parallel rather than sequential tool combinations, weakening progression in complex combinations and limiting improvement on stateful AppWorld APIs.Source: A Limitations · p. 14
  • Training interaction is limited to 8 rounds, or 5,000 tokens, while AppWorld scenarios involve an average of more than 8 APIs.Source: A Limitations · p. 14
  • Effectively combining stateful data and APIs with tight cross-category links remains an open problem identified for future work.Source: A Limitations · p. 14

B.1 StableToolbench Dataset details.

StableToolBench draws on 16,464 RapidAPI REST APIs organized into coarse categories and fine-grained collections. Its data covers single- and multi-tool instructions, with training examples divided across three instruction types.

B.1 StableToolbench Dataset details.

  • StableToolBench includes 765 samples and 16,464 RapidAPI REST APIs, grouped into 49 coarse categories and more than 500 fine-grained collections.Source: §B.1 · p. 14
  • The benchmark tests generalization to unseen instructions, unseen tools within a category, and unseen tools from a different category across its instruction scenarios.Source: §B.1 · p. 14
  • Training data contains 14,418 samples: 11,329 for I1, 2,253 for I2, and 836 for I3, after filtering non-English data and unnatural instructions.Source: §B.1 · p. 14

B.2 Appworld Dataset details.

AppWorld tasks involve complex, multi-application workflows, averaging 9.5 APIs and reaching 26. Tool selection is evaluated using labeled training tasks, while downstream tool calling is assessed through execution outcomes.

B.2 Appworld Dataset details.

  • AppWorld tasks average 9.5 APIs and reach a maximum of 26, and may have multiple valid tool sequences.Source: §B.2 · p. 15
  • Tool selection uses labeled AppWorld training tasks as test data; Test-N is evaluated through execution outcomes because it lacks tool-selection labels.Source: §B.2 · p. 15
  • Tool-selection evaluation uses 147 tasks across 49 scenarios, while downstream tool-calling evaluation uses 105 D-1 and D-2 tasks across 34 scenarios.Source: §B.2 · p. 15

B.3 Metric details.

The section defines tool-selection and AppWorld evaluation metrics, then describes baseline methods and shared comparison settings.

B.3 Metric details.

  • F1, Recall, Precision, Match, and SRecall assess selection quality, exact-set agreement, and target-tool discovery during search.F1 balances precision and recall; SRecall measures recall during the search phase.Source: §B.3 · p. 15
  • AppWorld uses FullCodeRefl with gpt-5-mini 3 to generate and execute code from selected tools, retrying after errors; TGC and SGC measure task- and scenario-level completion.TGC requires all scenario-agent tests to pass for a task; SGC requires all tests to pass for a scenario.Source: §B.3 · p. 15

C Baselines.

  • SearchR1 combines multi-turn search with retrieved-token masking and an outcome reward, while GSPO uses sequence-likelihood importance ratios and normalized response rewards.The GSPO baseline follows a token-level objective variant for multi-turn RL.Source: C Baselines · pp. 15–16
  • GDPO decouples normalization across tool-calling rewards; the comparison uses search recall and final-selection exact match as its two rewards.The final-selection reward follows Search-R1 practice.Source: C Baselines · p. 16
  • MARAG-R1 uses supervised fine-tuning followed by RLOO reinforcement learning; here, the authors use the same search engine as ToolSearcher after other retrieval choices performed poorly.Its reported reward design includes token-level F1, Document Coverage Reward, and Tool Exploration Reward.Source: C Baselines · p. 16
  • RL methods share training configurations, while SFT and testing use top-100 retrieved documents in the RAG comparison.The top-100 setting reflects the authors’ observation that smaller single-round retrieval sets often miss most target tools.Source: C Baselines · p. 16

D Implementation Details

Implementation evaluates in-domain and out-of-domain performance, selects a compact retriever, and specifies matched multi-turn training and inference settings.

D Implementation Details

  • Experiments use Qwen2.5-7B-Instruct, with Qwen3-4B-Instruct added to test robustness on smaller models, and evaluate StableToolBench and AppWorld separately.StableToolBench represents in-domain evaluation and AppWorld out-of-domain evaluation.Source: D Implementation Details · p. 16
  • Qwen-Embedding-0.6B is selected as the search retriever because it performed best among tested models while keeping parameter size relatively small.The tested alternatives include BM25, UniXcoder, GIST-Large, Arctic-Embed 2.0, and larger Qwen-Embedding models.Source: D Implementation Details · p. 16
  • Training uses five retrieved tool documents per turn and switches from category-constrained discrimination to global search after 30% of training data; inference retains five documents and allows eight turns.Training uses one epoch of 56 steps on eight A100 GPUs; inference increases retrieved-content allowance to 4,096 tokens per turn.Source: D Implementation Details · p. 17

E Detailed Settings of Ablation Study

The ablation study removes each of ToolSearcher’s three components separately to assess their contributions.

E Detailed Settings of Ablation Study

  • The CCTD ablation removes category-level constraints and uses global search throughout training instead of restricting search to subcategories.This isolates the effect of category-constrained tool discrimination.Source: E Detailed Settings of Ablation Study · p. 17
  • The ESM ablation replaces intermediate event-level rewards with trajectory-level supervision based only on the final outcome.This tests event-level search modeling.Source: E Detailed Settings of Ablation Study · p. 17
  • The TCA ablation removes trajectory-level constraints on repeated discoveries and reward correction when target tools remain unidentified.This tests trajectory-aligned credit allocation.Source: E Detailed Settings of Ablation Study · p. 17

F Gradient Analysis

This section analyzes how event-specific optimization signals target stages of tool search and selection relative to coarser trajectory-level signals. It also provides the experiment prompt, experimental details and disclosures about data and code availability, error bars, and fixed seeds.

F Gradient Analysis

  • Event-specific gradient coefficients use corresponding event-level advantages to target optimization signals at different stages of the search-selection process.The analysis says this assignment is intended to help the model improve the capabilities associated with each stage.Source: F Gradient Analysis · p. 18
  • GRPO applies a group-wise advantage from the binary final-selection reward to all trajectory actions, while GSPO’s token-wise adjustment still uses trajectory-level rewards.The analysis notes that GSPO can give similar signals to non-critical trajectory parts, producing redundant or noisy optimization.Source: F Gradient Analysis · pp. 18, 20
  • GDPO combines search-recall and final-selection advantages across trajectory actions, but may misassign credit when different search steps contribute unequally.Trajectories with the same search-recall score can retrieve different target tools, while all steps receive the same signal.Source: F Gradient Analysis · p. 18

G Prompt

  • The prompt permits repeated API-document searches, limits each turn to one search, and requires final API names to come from search results without fabrication.The final list uses the format [category_name].[tool_name].[api_name].Source: G Prompt · p. 20

H Broader Impact

  • The broader-impact discussion identifies benefits for complex tool orchestration and risks including misuse, unsafe tools, unreliable outcomes, and privacy leakage.It recommends access controls, sandboxing, security and privacy filtering, and human supervision in high-stakes scenarios.Source: H Broader Impact · pp. 21–22

NeurIPS Paper Checklist

  • The checklist affirms that the paper’s stated claims match its contributions, and says assumptions, gradient analysis, and reproducibility details are provided in specified sections and appendices.It points to Section 3 and Appendix F for assumptions and gradient analysis, and Section 4 and Appendices B and D for experimental reproducibility.Source: NeurIPS Paper Checklist · p. 23, §2 · p. 24

2. Limitations

  • The authors state that their limitations are discussed in Appendix A.Source: §2 · p. 23
  • The datasets are publicly available, and the code release is linked in the abstract.Source: §2 · p. 25
  • Training and test details are elaborated in Sections 4 and Appendices B and D.Source: §2 · p. 25
  • Error bars are not reported because they would be computationally expensive, and all experiments use a fixed seed.Source: §2 · p. 25
  • The authors provide the training GPU device and setup in Appendices B and D.Source: §2 · p. 26
  • The paper discusses societal impacts in Appendix H and states that RL training for tool selection does not pose high-risk misuse concerns.Source: §2 · pp. 26–27
  • The method uses an LLM as the policy model for multi-turn tool search and selection, optimized with reinforcement learning.Source: §2 · p. 29
Loading 2609.30906v1…