PaperlayerView exact revision on arXiv

Source-linked AI summary

Separate Field Scoring Protects Unused Tools When Agents Pool Tool-Use Experience

Abhijit Chakraborty

arXiv:2610.30867v1cs.SEcs.AI

TL;DR

Tool-use benchmarks do not report separately on tools with shared usage and tools without it, although pooled examples can move those groups in opposite directions. The paper compares folding, a slot-reserving router, and SYNAPSE, which scores documentation and usage separately. SYNAPSE has higher held-out Hit@5 than the router across three catalogs, while labeled-query retrieval advantages depend on the catalog.

  • Problem

    Benchmarks do not separately report tools with shared usage and tools with none, even though shared usage moves the groups in opposite directions.

    Source: Introduction · p. 1

  • Method

    SYNAPSE separately scores each tool’s documentation and best-matching usage example, then mixes the z-normalized scores using a development-selected weight.

    Source: Introduction · p. 1

  • Results

    Against the router, SYNAPSE is higher on held-out Hit@5 by +10.14 points on StableToolBench, +2.75 on ToolRet, and +8.71 on TOUCAN; labeled-query advantages vary.

    Source: Results · p. 3

  • Takeaways & Limitations

    When many requests target unused tools, score usage as its own field; when requests target labeled tools, the router retrieves more on TOUCAN and ToolRet in specified settings.

    Source: Conclusion · pp. 4–5

    The study does not test task success, tool-trained retrievers, Re-Invoke, or privacy mechanisms, and the agents are simulated partitions of one usage pool.

    Source: Limitations · p. 4

Abstract

from arXiv · show

Agents that call tools from a shared catalog can pool what they learn about which tool serves which request. We measure what pooled usage examples do to tool retrieval, reporting separately the held-out tools, for which no agent has usage, and all other tools. Adding usage examples to the index as extra entries for used tools raises Hit@5 (whether a gold tool is in the top five) for those tools and lowers it for held-out tools, by a large margin on two of three catalogs and with both embedders we tested. Giving used tools the same number of equally long entries filled with other tools' text lowers held-out Hit@5 about as much, but gives little or none of the gain. We call our method of scoring usage as its own field SYNAPSE. On StableToolBench, the held-out gold tool appears in its top five more often than with a router that reserves slots for usage (0.785 against 0.683), and this held-out lead holds on all three catalogs with both embedders. This protection persisted when we replaced each tool's usage with other tools' text, and a constant score offset for tools with usage gives similar held-out Hit@5 in most settings. The gain for tools with usage, in contrast, needs usage about the right tool. The router keeps more of the gain on labeled queries on TOUCAN with both embedders and on ToolRet with one, so the better choice depends on the workload. Splitting the usage among ten simulated agents shows that pooling helps exactly as much as giving one agent the same records.

Introduction

The paper asks how pooling usage examples changes retrieval for tools with usage and tools without it, and introduces separate-field scoring as an alternative to folding usage into the index. Its results show that the effects differ by tool group and catalog.

Introduction

  • In the motivating Shopify example, folding usage entries removes the correct scraper from the top five, although documentation alone retrieves it.The authors selected this query to illustrate a failure, not to estimate its frequency.Source: Introduction · p. 1
  • Benchmarks do not report tools with shared usage separately from tools with none, although shared usage moves the groups in opposite directions.The paper defines held-out tools as those with no contributed usage and labeled tools as all other tools; Hit@5 measures retrieval, not task completion.Source: Introduction · p. 1
  • SYNAPSE scores documentation and best-matching usage as separate fields, z-normalizes both across tools, and combines them with a development-selected weight.Tools without usage retain their documentation score.Source: Introduction · p. 1
  • Folding usage raises labeled Hit@5 and lowers held-out Hit@5 on StableToolBench and TOUCAN, while both changes are small on ToolRet.Length-matched text from other tools causes a similar held-out loss; SYNAPSE beats the router on held-out queries, but the router retrieves more labeled tools in some settings.Source: Introduction · p. 1

Setup

The setup evaluates retrieval on three tool catalogs with held-out and labeled test queries. It compares retrieval arms using frozen settings and reports both query-group Hit@5 and the Figure 1 visual comparison.

Setup

  • StableToolBench’s held-out set contains 172 G1-tool and G1-category gold tools; the filtered catalog has 161 held-out-test and 149 labeled-test queries.ToolRet and TOUCAN use hash-based held-out assignments; their test-query counts are 255 and 728, and 769 and 2,595, respectively.Source: Setup · pp. 1–2
  • Figure 1 plots held-out-test and labeled-test Hit@5 for each arm using Jina embeddings and three seeds; the untyped grouped arm was not run on TOUCAN.The caption notes different x-axis ranges.Source: Figure 1 · p. 2
  • All arms return five distinct tools using Jina-v2 embeddings; the main comparisons are repeated with Nomic Embed v1.5.SYNAPSE’s weight and the router’s documentation-slot count are selected on development tools and frozen before test scoring.Source: Setup · p. 2
  • The held-out Hit@5 intervals are paired 95% percentile bootstrap intervals with 10,000 draws, clustered by the first gold tool across all three seeds.Intervals including zero are reported as no difference detected, not as equivalence.Source: Setup · p. 2

Results

Folding usage can improve retrieval for tools with examples while damaging held-out tools; SYNAPSE protects held-out retrieval better than a slot-reserving router, while labeled-query advantages vary by catalog and workload.

RQ1: Who gains and loses

  • Folding raises labeled-test Hit@5 by 14 points on StableToolBench and 29 on TOUCAN, while held-out Hit@5 falls from 0.733 to 0.400 and 0.486 to 0.203; ToolRet changes little.With Nomic embeddings, folding again lowers held-out Hit@5 on all three catalogs.Source: Results · p. 2
  • Length-matched other-tools text leaves held-out Hit@5 about unchanged but lowers labeled-test Hit@5 by 33 and 27 points, indicating labeled gains need relevant usage.The figure reports Hit@5 over three seeds; the control preserves usage-entry count and tokenizer length.Source: Results · p. 2, Figure 2 · p. 3
  • One usage entry per used tool shrinks but does not remove the held-out loss; that control cannot separate entry count from text length, query-like style, or within-index competition.For example, held-out Hit@5 is 0.553 against 0.733 on StableToolBench with Jina.Source: Results · p. 3

RQ2: Combining usage with documentation

  • The router exceeds SYNAPSE on labeled-test Hit@5 on TOUCAN with both embedders and on ToolRet with Jina; the Nomic ToolRet labeled difference is unresolved.On fully covered ToolRet queries, the router leads by 17.15 points with Jina and 10.47 points with Nomic.Source: Results · p. 3
  • Held-out protection persists when other tools’ usage text replaces the correct text, while labeled Hit@5 falls by 6.2 to 54.6 points; a constant offset gives similar held-out results in most settings.These controls do not isolate which part of the scoring rule protects held-out tools.Source: Results · p. 3
  • On StableToolBench, results favor SYNAPSE when fewer than about 59% of queries go to labeled tools and do not decide above that share.This workload comparison was computed only for StableToolBench.Source: Results · p. 3
  • SYNAPSE beats the router on held-out Hit@5 across all three catalogs with Jina (+10.14, +2.75, +8.71 points) and Nomic (+7.66, +3.14, +11.05 points).Figure 3 compares held-out-test with labeled-test Hit@5; no SYNAPSE weight reaches the router’s points, and no router quota reaches SYNAPSE’s curve.Source: Results · p. 3, Figure 3 · p. 4

RQ3: Pooling usage across agents

  • At ten simulated agents, pooling beats the average single agent by 2.48 held-out points and 13.18 labeled points; one agent with the same records returns identical lists.The fixed StableToolBench usage pool was split among up to ten agents; the experiment tests no benefit specific to federation.Source: Results · p. 3, Figure 4 · p. 4

Related Work

Prior work expands tool representations with generated or observed queries, or studies sharing knowledge across clients. This paper focuses on retrieval effects for tools left without usage expansion and separately scores fields.

Expansion and tool representations

  • Query expansion has been studied for documents receiving expansion, including predicted queries, usage-query embeddings, and LLM-expanded tool documentation.The cited approaches include Nogueira et al. (2019), Moon et al. (2024), and Lu et al. (2026).Source: Related Work · p. 4
  • Re-Invoke expands every tool with generated queries, while this paper studies tools left without expansion when they compete with expanded tools.The authors report that their folding arm expands used tools only and that their length control suggests costs fall on tools left out.Source: Related Work · p. 4

Field scoring and federated learning

  • Separate-field scoring has prior retrieval precedents, while federated agent-learning work shares free-text knowledge, in-context examples, or router parameters across clients.The paper uses a simple form of field-wise scoring to measure which tools it protects.Source: Related Work · p. 4

Limitations

The evaluation is limited to retrieval Hit@5 with two embedders, and several settings remain untested or constrained.

Limitations

  • Results measure retrieval Hit@5, not task success, and do not cover tool-trained retrievers or Re-Invoke.The authors also make no claim about those systems.Source: Limitations · p. 4
  • The Nomic training-data audit is incomplete, so the authors make no contamination-free claim for that embedder.Source: Limitations · p. 4
  • Agents are simulated partitions of one usage pool, with no privacy mechanism applied.The length control covers two catalogs and does not separate token length from query-like style or index competition.Source: Limitations · p. 4

Conclusion

Shared usage helps tools it describes, but folding it into one index can push unused tools out of the candidate list. The conclusion favors field-wise usage scoring when many requests target tools without usage, while the better approach depends on workload.

Conclusion

  • Folding shared usage into one index can push tools with no usage out of the candidate list.Source: Conclusion · p. 4
  • When many requests target tools without usage, score usage as its own field.Source: Conclusion · p. 5
  • When nearly all requests target labeled tools, the router retrieves more on TOUCAN with both embedders and ToolRet with Jina.Source: Conclusion · p. 5

Appendix A: Splits and query accounting

The appendices specify how tools and queries are split, how retrieval comparisons are measured, and how results change with candidate budget and tool renaming.

Appendix A: Splits and query accounting

  • StableToolBench’s held-out set comprises 172 gold tools, while ToolRet and TOUCAN use hash-based held-out splits.ToolRet’s covered-only comparisons rest on about a fifth of its labeled rows.Source: Appendix A · p. 6

Appendix B: Metric and inference

  • Intervals use first-gold-tool clusters and keep all seeds and queries in a cluster together; they are conditional on frozen settings.The clustering convention does not capture every dependence from non-first gold tools shared across clusters.Source: Appendix B · p. 6

Appendix C: Retrieval budget

  • SYNAPSE’s held-out lead over the router narrows with more candidates on StableToolBench and ToolRet, but not TOUCAN.Its covered-query gain over documentation stays positive at 5, 10, and 30 candidates; on StableToolBench its interval at 30 touches zero.Source: Appendix C · p. 6

Appendix D: Tool renaming

  • After tool identifiers and explicit name mentions were masked, SYNAPSE still beat the router on StableToolBench held-out queries by 10.97 points.The labeled difference remained uncertain.Source: Appendix D · p. 6

Appendix E: Length-control construction

The length control preserves entry counts and tokenizer lengths while substituting other tools’ text. The appendices also describe the budget-sweep figure and identify tables containing values behind figures.

Appendix E: Length-control construction

  • The length control replaces each usage entry with other-tools text of exactly the same tokenizer length while preserving tool identifiers, documentation, and entry counts.It covers 113,110 StableToolBench entries and 6,617 TOUCAN entries.Source: Appendix E · p. 6
  • Figure 5 compares paired Hit@k differences with tool-clustered 95% intervals across 5, 10, and 30 candidates.The router’s usage-slot share stays at its k=5 proportion, and the dotted line marks settings frozen at k=5.Source: Figure 5 · p. 7

Appendix F: Exact values behind the figures

  • Tables 3–6 give the values plotted in Figures 1, 2, 4, and 5; Table 3 reports held-out and labeled Hit@5.Table 4 gives Figure 2 values, and Table 5 covers StableToolBench Hit@5 for Figure 4.Source: Appendix F · p. 7, Table 3 · p. 7, Table 4 · p. 7

Appendix G: Second embedder (Nomic Embed v1.5)

Appendix G evaluates Nomic Embed v1.5 with settings selected on development tools, then compares its selected settings with Jina’s and reports the scope of Table 7.

Appendix G: Second embedder (Nomic Embed v1.5)

  • On StableToolBench, Nomic’s selected settings equal Jina’s; carrying Jina’s settings over unchanged gives the same router results and SYNAPSE results within 1.3 points.Settings were selected on development tools, with development and test gold tools disjoint.Source: Appendix G · p. 7
  • Table 6 reports SYNAPSE-minus-router and SYNAPSE-minus-documentation-only Hit@k differences at k ∈ {5, 10, 30}, with paired tool-clustered 95% intervals.H denotes held-out-test; C denotes labeled-test queries whose every gold tool has usage.Source: Table 6 · p. 8
  • Table 7 reports Nomic Hit@5 for held-out-test, labeled-test, and fully covered labeled queries across three seeds, with settings reselected on development tools.Its other-tool text length control was run on StableToolBench only.Source: Table 7 · p. 8

Appendix H: Workload break-even and a mechanism diagnostic

Appendix H estimates when SYNAPSE’s StableToolBench advantage holds across workload mixes and diagnoses whether usage scores distinguish relevant from irrelevant labeled tools.

Appendix H: Workload break-even and a mechanism diagnostic

  • SYNAPSE gains 4.30 points of Hit@5 ([1.29, 7.45]) on StableToolBench at a 48.1% labeled-query mix, and is better below an estimated 83.4% labeled share ([59.2%, 100%]).If every gold tool on a labeled query must have usage, the cutoff is 78.4% ([55.3%, 100%]); above about 59%, the data do not decide.Source: Appendix H · p. 8
  • The StableToolBench break-even cutoffs assume each stratum’s Hit@5 carries over to the target workload, and were not computed for ToolRet or TOUCAN.An upper endpoint of 100% includes bootstrap draws with no crossing.Source: Appendix H · p. 8
  • Usage-minus-documentation scores separate gold tools from distractor labeled tools at the same documentation rank with AUC 0.817 ([0.780, 0.850]) over 340 StableToolBench queries.This pooled diagnostic includes development queries and is not a test-only result.Source: Appendix H · p. 8

Appendix I: Entry-budget, offset and other-tool controls

Appendix I tests entry-budget, score-offset, and other-tool controls to distinguish the effects of adding usage entries from the effects of usage relevance.

Appendix I: Entry-budget, offset and other-tool controls

  • The other-tool control permutes usage texts and vectors across tools, preserves entry counts and global length distribution, and is evaluated at SYNAPSE’s frozen weight.The offset control adds a development-tuned constant to standardized documentation scores for tools with usage; results are descriptive on test sets seen in earlier work.Source: Appendix I · p. 8, Table 8 · p. 9, Table 9 · p. 9
  • With Jina, one-entry folding lowers held-out Hit@5 versus documentation-only by 18.01 points on StableToolBench, 1.57 on ToolRet, and 19.98 on TOUCAN.Intervals are respectively [12.80, 23.33], [0.38, 3.19], and [10.93, 28.82].Source: Appendix I · p. 8
  • With Nomic, one-entry folding lowers held-out Hit@5 versus documentation-only by 9.52 points on StableToolBench, 1.70 on ToolRet, and 13.78 on TOUCAN.Intervals are respectively [4.60, 15.40], [0.39, 3.40], and [7.94, 19.75].Source: Appendix I · p. 8
  • Held-out offset minus SYNAPSE was −1.45 (Jina/StableToolBench), −1.18 (Jina/ToolRet), +0.35 (Jina/TOUCAN), +0.41 (Nomic/StableToolBench), 0.00 (Nomic/ToolRet), and −13.18 (Nomic/TOUCAN).The corresponding paired 95% intervals are reported for each comparison.Source: Appendix I · p. 8

Related papers

Loading 2610.30867v1…