PaperlayerView exact revision on arXiv

Source-linked AI summary

Tool Retrievers Are Underestimated: Annotation Expansion Reveals True Capability

Yanyu Zhu, Chenheng Zhang, Shaoshen Chen, Hoilam Pao, Yufei zhang, Jiajun Chai, Dongnian Wang, Zhaoyu Hu, Guojun Yin, Wei Lin, Hai-Tao Zheng

arXiv:2609.08327v1cs.SEcs.IR

TL;DR

Tool retrieval benchmarks assign one ground-truth combination even when queries can be solved by multiple functionally equivalent combinations, leaving the one-to-many mapping unrepresented. ToolEX automatically expands the annotations into ToolEq; the expanded benchmark raises scores for base retrievers and reduces the apparent fine-tuning gains, with the same issue also found in skill retrieval.

  • Problem

    Existing retrieval benchmarks label only one tool combination per query despite the presence of functionally equivalent alternatives.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    ToolEX automatically finds equivalent tool combinations and uses the expanded valid references in ToolEq, which credits retrievals matching any labeled alternative.

    Source: §1 · pp. 1–2, §3.3 · p. 4

  • Results

    ToolEq raises NDCG@10 by 5–7 pp on average for eight base retrievers over Tool-DE, while 30–47% of apparent fine-tuning gain vanishes when equivalents are credited.

    Source: §1 · p. 2, §5 · p. 7

  • Takeaways & Limitations

    Applying the same pipeline to SkillRet confirms that the one-to-one annotation problem extends beyond tool retrieval.

    Source: §1 · p. 2, §5 · p. 7

    When a query has multiple equivalent combinations, standard metrics count every combination other than the single annotated reference as a false negative.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

In open-world scenarios with massive and evolving tool repositories, tool-augmented large language models rely on a retriever to surface relevant tools for a given query. Because such repositories often contain many tools that implement the same functionality, a single query can often be resolved by several distinct but functionally equivalent tool combinations, making the natural query-to-tool mapping inherently one-to-many. However, existing tool retrieval benchmarks annotate each query with a single relevant tool combination, collapsing this one-to-many mapping into a rigid one-to-one annotation and causing valid retrieved tools to be misjudged as failures. To address this, we propose ToolEX (Tool Equivalent eXpansion), a framework that automatically discovers and annotates the tool combinations functionally equivalent to the labeled ones. Applied to the 7,360-query Tool-DE benchmark, ToolEX finds that 67.9% of sub-queries admit equivalent alternatives, expanding the singular ground truth to an average of 5.3 valid combinations per query. Using the expanded benchmark ToolEQ, we re-evaluate eight base retrievers and two fine-tuned variants; metrics on ToolEQ rise substantially over Tool-DE, showing that one-to-one annotation systematically underestimates retrievers and that 30--47% of the reported fine-tuning gain is an evaluation artifact rather than genuine improvement. Applying the same pipeline to skill retrieval on SkillRet further confirms that the one-to-one problem extends beyond tool retrieval.

1 Introduction

Existing tool retrieval benchmarks impose a one-to-one mapping even though queries can have multiple functionally equivalent tool combinations, causing valid alternatives to be scored as misses. ToolEX expands these annotations automatically; on Tool-DE, the resulting ToolEq benchmark raises retrieval scores and reveals that some apparent fine-tuning gains reflect the evaluation protocol.

1 Introduction

  • The query-to-tool mapping is naturally one-to-many, but existing benchmarks label only one combination, so equivalent retrieved tools can be scored as misses.For a historical-events query, four equivalent tools are unannotated alongside the single labeled QueryHistoryToday.Source: §1 · p. 1, Figure 1 · p. 1
  • ToolEX automatically discovers functionally equivalent tool combinations without human labeling.It decomposes composite queries, retrieves and verifies candidate tools for each sub-query, then combines and filters the verified sets.Source: §1 · pp. 1–2
  • On the 7,360-query Tool-DE set, ToolEX expands annotations from one to an average of 5.3 combinations per query; 67.9% of sub-queries gain an equivalent alternative.ToolEq credits retrievals matching any valid equivalent combination.Source: §1 · p. 2
  • Eight base retrievers gain 5–7 pp NDCG@10 on average over Tool-DE under ToolEq, and correcting evaluation narrows the fine-tuned-to-base gaps.For the 0.6B and 4B models, the gaps change from +6.4 pp to +4.5 pp and from +7.2 pp to +3.8 pp, respectively.Source: §1 · p. 2
  • Applying the same pipeline to SkillRet also confirms that the one-to-one annotation problem extends beyond tool retrieval.The paper describes the issue as ecosystem-wide.Source: §1 · p. 2

2 Related Work

Related work covers tool-use benchmarks and retrieval methods for open-domain agents, where retrieval is needed because repositories are too large to fit directly into context. Existing retrieval approaches improve representation, reformulation, or policies, but evaluate against a single target; this paper instead changes the evaluation paradigm.

Tool Learning Ecosystem: Use and Retrieval Benchmarks

  • Tool-use benchmarks cover diverse execution scenarios, while dedicated retrieval benchmarks address open-domain repositories too large for direct inclusion in an LLM context window.Benchmark construction commonly samples seed tools and prompts LLMs to synthesize corresponding queries, coupling each query to its seeded tools.Source: §2 · p. 2

Tool Retrieval Methodologies

  • Retrieval methods optimize query-tool matching through dense-retriever fine-tuning, specialized vector spaces, generative identifiers, query reformulation, decomposition, graph modeling, and reinforcement learning.These approaches are evaluated against a single rigidly annotated target, which can count an equivalent alternative as a false negative.Source: §2 · p. 2
  • This work changes evaluation rather than retrieval algorithms, expanding single ground truths into sets of functionally equivalent tool combinations.It is presented as orthogonal to methods that improve query and tool encoding, reformulation, or retrieval.Source: §2 · p. 2

3 Equivalent Ground-truth for Tool Retrieval and Skill Retrieval

In open-world tool libraries, queries may have multiple valid tool combinations, but benchmarks label only one. Section 3 presents ToolEX for expanding those ground truths and ToolEq for evaluating retrieval against valid alternatives.

3.1 Limitations of Current Retrieval Benchmarks

  • Open-world libraries contain functionally equivalent tools, so valid combinations form a one-to-many relation that single-answer benchmarks score as false negatives.During fitting, unlabeled positives can also enter negative batches, suppressing generalization to equivalent tools.Source: §3.1 · p. 3

3.2 Annotation Pipeline

  • ToolEX decomposes queries into atomic sub-queries, retrieves and verifies equivalent tools, then assembles and audits combinations for function coverage and dependency consistency.Stage 1 uses benchmark-provided ground-truth tools, making decomposition an annotation-time component rather than a deployable test-time planner.Source: §3.2 · pp. 3–4
  • 67.9% of sub-queries admit an equivalent tool beyond the annotated one, and ToolEq averages 5.3 valid combinations per query.Table 1 reports expansion statistics for ToolEq and SkillEq.Source: §3.2 · p. 4, Table 1 · p. 4

3.3 Benchmark and Evaluation Metric

  • ToolEq stores multiple valid combinations per query and scores each standard metric against each reference separately, retaining that metric’s best score.This best-match evaluation credits retrievals that match a valid alternative rather than only the originally labeled combination.Source: §3.3 · p. 4

4 Experiments

The experiments compare retrieval under one-to-one and equivalent-aware annotations, then test the expanded evaluation on skill retrieval and downstream ToolBench tasks. The results show that one-to-one evaluation misses credit for valid equivalents, while the extent of recovered credit varies across categories and settings.

4.2 Main Results

  • All eight base retrievers score higher on ToolEq; their mean NDCG@10 gain over Tool-DE ranges from +4.6 pp for NV-Embed-v1 to +8.5 pp for Qwen3-Embedding-8B, averaging +6.5 pp.ToolEq credits any equivalent combination, and the old–new gap measures the underestimation recovered by the expanded annotations.Source: A Systematic Underestimation · p. 5, Table 2 · p. 5
  • ToolEq recovers the most NDCG@10 credit for Qwen3-Embedding-0.6B in Customized queries (+9.2 pp), followed by Web (+7.9 pp) and Code (+5.4 pp).The passage attributes this gradient to greater functional redundancy in customized and web tool libraries than in code repositories.Source: Category-Level Heterogeneity · p. 6
  • Under ToolEq, Qwen3-Embedding-0.6B gains 12.6 pp NDCG@10 with oracle-assisted sub-query decomposition, while Tool-Embed-4B reaches 70.2 NDCG@10.The diagnostic reuses benchmark-provided decomposition and tool references, so it is an oracle-assisted upper bound rather than a deployable setting.Source: §4.1 · p. 5, Unlocking Retrieval Gains with Query… · p. 6
  • 47% of Tool-Embed-4B's reported fine-tuning gain vanishes under ToolEq, and its gap over Qwen3-Embedding-4B shrinks from 7.2 pp on Tool-DE to 3.8 pp.At 0.6B, the gap shrinks from 6.4 pp to 4.5 pp; on Code, the 4B comparison changes from +2.1 pp to −0.3 pp.Source: The Illusion of Fine-Tuning Improvements · p. 6, Figure 3 · p. 6
  • Equivalent tools rank in the top 10 for 79.4% of eligible Qwen3-Embedding-4B queries, and in 14.5% the ground-truth tool ranks outside the top 10 while an equivalent appears within it.A random negative reaches the top 10 on 9.2% of the same query set.Source: Where Do Equivalent Tools Appear in… · p. 6

4.3 Extended Experiment: Skill Retrieval

  • On SkillEq, SkillRet-0.6B improves from 53.2 to 64.6 NDCG@10 (+11.4 pp), extending the one-to-one annotation problem beyond tool retrieval.The section also reports that fine-tuned SkillRet-8B remains above zero-shot Qwen3-8B after expansion: 82.93 vs. 61.89 NDCG@10 (+21.0 pp).Source: §4.3 · p. 6, Table 3 · p. 7

4.4 Downstream Task Evaluation on ToolBench

  • ToolEX-expanded achieves the highest pass rate across all six ToolBench instruction categories, exceeding Oracle by +3.5–6.5 pp and ToolBench-IR by +1.0–6.9 pp.The authors treat this as evidence that many discovered equivalents are practically usable, not proof that ToolEX surpasses a fully specified execution oracle.Source: §4.4 · pp. 6–7

4.5 Discussion: Benchmark Implications

  • ToolEq restores credit for valid repository solutions omitted by one-to-one annotation; in open-world tool ecosystems with substantial redundancy, exact-reference evaluation should not be the only correctness measure.The authors recommend reporting both exact-reference and equivalent-aware scores.Source: §4.5 · p. 7

5 Conclusion

ToolEX identifies one-to-many annotation as a problem in tool retrieval and expands equivalent positives; under ToolEq, 30–47% of apparent fine-tuning gain on Tool-DE vanishes. The authors also report limitations in validation and generalization, and outline runtime verification and active learning as future directions.

5 Conclusion

  • 30–47% of the apparent fine-tuning gain on Tool-DE vanishes under ToolEq once equivalent tool combinations are credited.The same framework’s application to SkillRet indicates that the one-to-many problem extends beyond tool retrieval.Source: §5 · p. 7
  • ToolEX verifies equivalence semantically rather than by practically invoking tools, which may introduce false positives.The expanded labels have only partial human spot-checking, not exhaustive full-scale validation.Source: §5 · p. 7
  • Generalization beyond Tool-DE, SkillRet, and ToolRet remains to be validated.Future work includes runtime verification on executable tools and active-learning loops to prioritize high-uncertainty candidates.Source: §5 · p. 7

A LLM Prompt Templates

The appendix prompts specify how LLMs decompose queries, verify individual tools, and audit combinations. The decomposition prompt is oracle-assisted because it uses benchmark-provided tool counts and reference tools, so it is not a deployable test-time planner.

Stage 1: Multi-Tool Query Decomposition

  • Stage 1 uses DeepSeek-V3.2 to produce exactly k atomic sub-queries, one per ground-truth tool, while minimizing the semantic gap to tool documentation.The prompt focuses each sub-query on the operation and entity, and excludes tool, API, and function names and specific parameter or return-value details.Source: A LLM Prompt Templates · p. 9

Stage 2: Tool Verification

  • Stage 2 uses GPT-4o-mini to judge whether a candidate tool directly completes an atomic sub-query with the same core operation and sufficient output as the reference tool.The prompt returns no when functionality mismatches or significant extra steps are needed, and also defaults to no when genuinely uncertain.Source: A LLM Prompt Templates · p. 9

Stage 3: Combination Audit

  • Stage 3 uses Claude-Sonnet-5 to assess whether a tool combination can fully complete the query, checking required operations against the sub-query breakdown.The audit treats tools with the same core operations as the reference as sufficient despite differences in names, interfaces, or parameter formats; it assumes intermediate values pass between tools at runtime.Source: A LLM Prompt Templates · pp. 9–10

B Human Validation Guidelines

Human validation audits accepted query-level tool combinations using stratified sampling and a two-part validity rule. Two reviewers judge each item independently, with disagreements resolved by a third reviewer.

B Human Validation Guidelines

  • The audit samples accepted query-level combinations, stratified by domain and query complexity.The sampling unit is the final equivalent combination after Stage 3.Source: B Human Validation Guidelines · p. 10
  • Reviewers receive the query, task instruction, candidate and reference combinations, model rationale, and any available sub-query breakdown..Source: B Human Validation Guidelines · p. 10
  • A combination is valid only if it covers every required operation and its tools are dependency-compatible when one tool’s output must feed another.Two reviewers judge each item independently; a third reviewer resolves disagreements without seeing model scores.Source: B Human Validation Guidelines · p. 10
  • The failure taxonomy includes decomposition ambiguity, non-substitutable near-synonyms, incomplete functional coverage, and cross-platform dependency mismatch.Source: B Human Validation Guidelines · p. 10

C Justification of Max-Aggregation

Max-aggregation evaluates retrieval against all equally valid equivalent combinations and takes the best score for each metric. This makes the score independent of which equivalent was arbitrarily recorded as the single reference, while the resulting gap from Tool-DE measures annotation-induced underestimation.

C Justification of Max-Aggregation

  • The expanded set contains m equally valid combinations, each scored using the single-reference definition of NDCG@K, Recall@K, or Comp@K.The one-to-one benchmark selects one combination as its sole reference, whereas ToolEq asks how well retrieval does across all valid solutions.Source: C Justification of Max-Aggregation · p. 10
  • Taking the maximum across equally valid references is label-agnostic and credits a retrieval that matches any valid equivalent in full.A mean would discount retrievals matching some but not all equivalents; when there is only one valid combination, max-aggregation reduces to the standard single-reference metric.Source: C Justification of Max-Aggregation · p. 10
  • For every query and metric, ToolEq’s max score is at least the Tool-DE single-reference score, and their non-negative gap is exactly the score restored by crediting a retrieved equivalent.A positive gap occurs when an unannotated equivalent scores higher than the recorded reference; the gap remains non-negative after macro-averaging.Source: C Justification of Max-Aggregation · p. 10
  • The maximizing reference may differ across NDCG@K, Recall@K, and Comp@K for the same query.The metrics reward different aspects of ranking, so forcing one reference to optimize all three would distort at least one.Source: C Justification of Max-Aggregation · pp. 10–11
Loading 2609.08327v1…