PaperlayerView exact revision on arXiv

Source-linked AI summary

When Tool-Backed Skill Retrieval Fails: Source-Style Collapse in Executable Capability Retrieval

Yiqi Liu, Joseph James, Yang Wang, Chenghao Xiao, Chenghua Lin

arXiv:2608.16502v1cs.LGcs.IR

TL;DR

Tool retrieval can collapse across source styles even when the underlying tool corpus is unchanged, blocking capabilities before planning or execution. The paper introduces ToolScout, which uses query-side TF-IDF compatibility to route mismatched traffic, and finds that the failure persists beyond raw API-schema formatting while routing addresses it.

  • Problem

    The paper asks whether retrievers adapted to one source-defined slice remain reliable on other source styles over the same unchanged tool corpus.

  • Method

    ToolScout uses a query-side TF-IDF compatibility predictor to route likely mismatched queries to an aggregate-trained retriever and support matched continuation fine-tuning.

  • Results

    ToolScout addresses source-style collapse, while skill-card rerendering shows the same failure and routing behavior persists beyond raw API-schema formatting.

  • Takeaways & Limitations

    Reliable tool-using agents require reliable capability retrieval before reranking, planning, or execution.

  • Takeaways & Limitations

    The detailed analyses focus on a controlled 1100-query normalized web subset, while independent replication on another multi-source tool corpus remains a next test.

Abstract

from arXiv · show

Large-scale agents increasingly rely on retrieval to access external capabilities. We study this retrieval gate in structured tools and APIs, a measurable class of tool-backed executable skills that must be surfaced before an agent can plan, incorporate, or act. In this setting the retrieval layer can silently fail even when the capability corpus is fixed: on ToolRet, a retriever fine-tuned on one source-specific slice collapses on another source-specific slice of the same benchmark, with FT-1100 despite its higher lexical overlap with the gold tools. We call this failure mode source-style collapse. Query-side TF-IDF fingerprints flag source styles on which the fine-tuned retriever is likely to fail better than semantic or length-based proxies, giving a cheap signal for mismatch over a fixed tool corpus. We propose ToolScout, a source-aware routing method that uses this signal as a routing guard: on the mixed 4,996-query stream, TF-IDF-based routing raises coverage from 22.3% to 86.1%, and across five collapsed sources 20 matched examples raise the coverage-weighted global top-1 proxy from 1.3% to 53.9%. The same failure and routing behaviors persist when tools are rerendered as executable skill cards, which rules out raw API-schema format as the sole cause.

1 Introduction

The paper studies source-style collapse at the retrieval gate for executable tools and APIs: a retriever trained on one source slice can fail on other source styles despite a fixed tool corpus. ToolScout uses query-side source-style signals to route mismatched traffic and tests whether the behavior persists under executable skill-card representations.

  • Motivation: Retrieval determines which executable capabilities enter the candidate set before reranking, planning, incorporation, or execution, making missed tools unavailable to downstream agents.The paper renders each tool or API as an executable skill card with a name, description, input schema, and implementation.
  • Problem: 0.7% coverage on APIGen, 0.0% on ToolACE, and 8.3% on UltraTool show collapse for a toolbench-style 1100-query checkpoint on other source styles within the same benchmark.The tool corpus remains fixed, and lexical overlap alone does not explain the failure.
  • Motivation: About 172k tokens of full-catalog exposure compress to about 407 tokens at retrieved top-5, making retrieval essential for large heterogeneous tool catalogs.Downstream rerankers and planners operate only on the small candidate pool selected by candidate generation.
  • Method: ToolScout routes batches likely to mismatch a source-specific retriever to an aggregate-trained checkpoint, then adapts it with matched supervision to recover remaining coverage.The method treats source style as conventions in query wording, verbosity, schema-reference patterns, and query-tool pairing.
  • Robustness: The same failure pattern, detector, and switch policy persist after tools are rerendered as executable skill cards, so raw API-schema formatting is insufficient to explain the collapse.This supports studying source-style sensitivity as a retrieval behavior rather than only a representation-format artifact.

2 Capability Retrieval and Tool Routing

Capability retrieval treats reusable skills, tools, and APIs as external units that agents must surface before incorporation or execution. This section situates ToolScout at the retrieval gate, distinguishing it from candidate-pool reranking, hierarchical retrieval, agent orchestration, and adaptation to new target distributions.

  • Skill and capability augmentation: Skill Retrieval Augmentation frames scalable capability access as retrieving reusable skills from a large external corpus before agents decide whether and how to incorporate them.SRA-Bench decomposes evaluation into retrieval, incorporation, and end-task execution.
  • Tool and API retrieval: Tools and APIs form the structured, executable end of the capability spectrum, with benchmarks evaluating large-scale retrieval and multi-hop tool use.Toolformer and Gorilla established the general tool-use setting, while API-Bank, ToolBench / ToolLLM, StableToolBench, ToolHop, and ToolRet sharpened retrieval evaluation.
  • Tool and API retrieval: Tool2Vec studies tool-specific representations, whereas ToolRerank addresses hierarchy-aware reranking and adaptive truncation after a candidate pool is supplied.AnyTool and DeepAgent use hierarchical retrieval inside broader agent stacks.
  • Retrieval adaptation under capability shift: Dense retrieval and reranking baselines established the standard retrieval toolkit, while BEIR made cross-domain transfer a central concern and modern encoders provide off-the-shelf backbones.The cited baselines include DPR, ANCE, ColBERT, and BERT reranking; modern encoders include BGE-M3 and Qwen3-Embedding.
  • Retrieval adaptation under capability shift: Domain-adaptation methods use pseudolabeling or synthetic query generation to adapt retrievers to new target distributions, aligning with ToolScout’s few-shot repair stage.ToolScout instead keeps the capability corpus fixed while changing the source slice that generates queries.

3 ToolScout Routing Policy

ToolScout is a source-aware, two-stage retrieval-and-reranking policy that returns a context-budgeted candidate set while preserving coverage for downstream execution. It routes queries using TF-IDF compatibility with retriever training sources and supports direct, staged, and hierarchical retrieval variants.

  • Core pipeline: ToolScout uses a direct query → candidates → reranking pipeline with a fixed candidate budget tied to retrieved tool cards and rendered token length.The budget is operationalized by the number of retrieved cards and their rendered token length rather than monetary cost.
  • Source-aware routing: Source style captures query wording, verbosity, schema-reference patterns, and query-tool pairing conventions introduced by upstream generation or annotation.These conventions affect both query surface forms and the target tool texts exposed to the retriever.
  • Staged and hierarchical extensions: Staged retrieval decomposes queries into sub-queries, retrieves each independently, aggregates candidate scores, and reranks the resulting top-k pool.The reranker must match the candidate distribution of the retrieval path because merged staged and direct pools differ.
  • Source-aware routing: ToolScout computes source compatibility in TF-IDF space and calibrates safe and unsafe distance bands on natural ToolRet source splits.The bands are calibrated once in leave-one-config-out form and then fixed for routing experiments.
  • Source-aware routing: Queries inside the safe band use the matched fine-tuned retriever, whereas clearly unsafe queries switch to an aggregate-trained checkpoint or retain staged routing while matched supervision is collected.The policy can continue operating as matched supervision becomes available.

4 Experimental Setup

The experiments test retrieval under controlled, broader same-corpus, and large-scale ToolRet settings, using standard lexical and dense baselines with fixed candidate depth and held-out reranking evaluation. They also define a coverage-weighted top-1 proxy and rerender tools as executable skill cards to test whether retrieval-gate failures persist beyond raw API-schema formatting.

  • Benchmarks: ToolRet uses a controlled 1,100-query normalized web subset, a 4,996-query aggregate spanning 19 source-query generators, and a merged build with 44,453 tools and 7,726 queries.The subsets probe retrieval stress, same-corpus source-style sensitivity, and larger-scale behavior across web, code, and customized corpora.
  • Baselines and evaluation: BM25, bge-small-en-v1.5, and BGE-M3 provide lexical, lightweight dense, and general-purpose dense baselines, with candidate depth fixed at 20 for main dense runs.Reranker datasets use deterministic query-hash train/validation/test splits, and reranking metrics are reported on held-out candidate-covered test queries.
  • Evaluation metric: The coverage-weighted top-1 proxy counts uncovered queries as failures while evaluating covered queries using conditional top-1 accuracy within the retrieved top-m pool.It combines top-m candidate coverage, Cm, with P@1|cov, the top-1 accuracy among queries whose candidate pool contains at least one gold tool.
  • Skill-card rendering: Each tool is rerendered as an executable skill card with normalized capability, conditions, inputs, and output or effect fields, while tool identities, labels, splits, retrievers, and routing remain unchanged.This isolates whether retrieval-gate failure persists under skill-like representation rather than raw API-schema formatting.

5 Results

Results show that source-specific retriever fine-tuning can catastrophically collapse across ToolRet styles despite fixed tools, while TF-IDF routing and small matched-source updates substantially recover coverage. The pattern persists with executable skill-card rendering, indicating raw API-schema format is not the sole cause.

  • Source-style transfer: Depth-20 coverage collapses to 22.3% when the 1100-trained checkpoint transfers to the broader 4996-query aggregate, while aggregate training retains 87.3% on the held-out 1100 split.The checkpoint remains strong on the embedded toolbench source at 91.8% coverage but collapses on APIGen at 0.7% and ToolACE at 0.0%.
  • Source-style diagnostics: TF-IDF distance tracks transfer collapse at query-count-weighted r=-0.85 and routes unsafe sources with 78.9% accuracy and 0.88 F1.These results outperform semantic-centroid distance, which reaches 73.7% accuracy and 0.85 F1; lexical overlap does not explain the collapse because APIGen has higher overlap than the toolbench slice.
  • ToolScout routing: 86.1% coverage results from TF-IDF routing on the mixed 4996-query stream, compared with 22.3% for always using FT-1100 and 85.2% for the fixed aggregate checkpoint.Routing sends high-coverage styles to FT-1100 and low-coverage transfer sources to the aggregate checkpoint.
  • Matched-source repair: 20 matched examples raise the weighted MLP global proxy from 1.3% to 53.9% across five collapsed sources, approaching the 59.0% aggregate-trained reference.The shot-count sweep shows that 5-shot repair generally fails, whereas short continuation fine-tuning with 20 examples substantially repairs source-specific loss.
  • Executable skill cards: Skill-card rendering reproduces the collapse: matched-source fine-tuning raises 1100-slice coverage from 72.2% to 89.2%, while FT-1100 reaches only 21.7% on the mixed 4996-query stream.Tool identities, labels, splits, retrievers, and routing policy remain unchanged, isolating rendering format as insufficient to explain the behavior.

6 Conclusion

The paper identifies source-style collapse as a same-corpus tool-retrieval failure and proposes ToolScout, which uses query-side TF-IDF compatibility to route mismatched traffic. Collapse and routing persist when tools are rendered as executable skill cards.

  • 6 Conclusion: Source-style collapse occurs when a retriever adapted to one source-defined slice loses coverage on another slice of the same tool corpus.This establishes the failure mode as a corpus-fixed retrieval mismatch rather than a change in available tools.
  • 6 Conclusion: ToolScout uses a query-side TF-IDF compatibility predictor to route likely mismatched traffic to an aggregate-trained checkpoint.The routing guard targets source-style incompatibility before retrieval proceeds.
  • 6 Conclusion: ToolScout applies short continuation fine-tuning when matched examples become available.This supplements routing with adaptation from newly available matched examples.
  • 6 Conclusion: Collapse and the switch policy persist after visible tool text is rewritten into normalized executable skill cards.The skill-card rendering keeps the same tools, labels, splits, retrievers, and routing policy while using a more skill-like representation.

Limitations

The study’s comparisons primarily support routing-mechanism claims under shared protocols, with detailed analyses concentrated on a controlled 1,100-query subset. Generalization is limited by corpus scope, supervision availability, and the quality and coverage of hierarchy metadata.

  • Shared evaluation protocol: Shared retrieval, exposure, and downstream-selection pipelines make routing variants comparable but do not reproduce their original prompts, execution policies, or model choices.The findings therefore support claims about routing mechanisms under a shared protocol rather than full reproductions of original agent stacks.
  • Corpus and annotation coverage: Detailed ToolRet reranking, depth, and failure-bucket analyses focus on the controlled 1,100-query normalized web subset, while the full 7.6k-task build offers less detailed error annotation.A positive replication on another independently generated multi-source tool corpus remains an important next test.
  • Scope and supervision: Executable skill-card results cover only structured tools and APIs, excluding procedural skills, non-tool resources, and end-task skill incorporation; few-shot repair depends on supervision availability.Matched examples may come from failed-query telemetry or tool-provider onboarding, but the rate at which such supervision accumulates in real deployments remains open.
  • Hierarchical routing: Manual categories remain stronger than automatic clusters in the first ToolRet hierarchy extension, although both hierarchy variants substantially reduce candidate context.Hierarchical routing is evaluated most strongly on StableToolBench, and broader tests on category-structured tool corpora are needed because category-metadata quality matters.

A Appendix

The appendix presents reranker, retrieval, source-diagnostic, rendering, normalization, and execution follow-ups. It also identifies priority tables for readers focused on the main collapse claim.

  • Appendix A.1–A.3 covers reranker checks, larger-scope ToolRet runs, and retriever adaptation details.
  • Tables 21–36 collect source diagnostics, synthetic controls, MetaTool, skill-card rendering, and MCP-lite normalization.
  • Appendix A.6–A.9 covers depth, staged retrieval, hierarchy, and local execution follow-ups.

A.1 Reranker Splits and Checks

The reranker evaluation separates candidate coverage from held-out covered-query quality and verifies that stronger rerankers improve the latter without resolving uncovered-query failures. Feature analysis shows retrieval evidence dominates StableToolBench, while ToolRet also relies heavily on metadata structure, with no execution-feedback contribution.

  • Evaluation splits: 129 held-out candidate-covered ToolRet queries supply the reranking metrics, while candidate coverage is computed over the full 1100-query benchmark slice.The reported P@1, R@5, and nDCG@5 metrics use the reranker protocol’s held-out candidate-covered split.
  • Evaluation splits: 635 / 126 / 129 and 551 / 94 / 108 are the train / val / test counts for ToolRet and StableToolBench, respectively.The splits skip 210 uncovered ToolRet queries and 12 uncovered StableToolBench queries.
  • Feature analysis: StableToolBench is dominated by retrieval-side evidence, whereas ToolRet also relies heavily on metadata-structure features; execution-feedback features carry zero weight in both dense runs.The learned reranker is linear, enabling grouped absolute coefficient-mass analysis.
  • Stronger-reranker controls: MLP and cross-encoder controls improve covered-query quality, especially on ToolRet, but uncovered-query failures remain binding because reranking begins only after candidate generation succeeds.Both controls use the same feature sets or candidate-covered reranker datasets and held-out evaluation protocol as the linear reranker.

A.2 Larger-Scope ToolRet Aggregate

The larger ToolRet aggregate merges 19 query configurations into a 4,996-query benchmark over the same web tool corpus, with substantial upstream candidate-generation failures under the depth-20 protocol.

  • Benchmark construction: 4,996 deduplicated queries comprise 2,242 single-tool and 2,754 multi-step queries in the larger aggregate benchmark.The aggregate merges 19 ToolRet query configurations over the same web tool corpus.
  • Upstream retrieval difficulty: 1,316 queries fail to place any gold tool into the candidate pool under the same depth-20 dense candidate-generation protocol.This indicates the larger aggregate is harder at the upstream candidate-generation stage.

A.3 Task-Adapted Dense Retriever Fine-Tuning

Task-adapted dense-retriever fine-tuning directly targets candidate-generation failure, but its benefits depend on source-style alignment and hard-negative quality. The intervention improves retrieval robustly across evaluation slices and encoder capacities, while aggregate-training controls do not resolve collapsed-source transfer.

  • Direct dense-retriever adaptation: BGE-M3 fine-tuning with MultipleNegativesRankingLoss and retrieval-confusing hard negatives directly repairs the candidate-generation bottleneck before reranking or hierarchy changes.The method trains query-tool pairs with retrieval-confusing hard negatives, aligning the intervention with the diagnosis that candidate generation is the first-order failure mode.
  • Hard-negative policy: Retrieval-confusing semantic negatives are strongest, BM25 negatives are close, random negatives are weaker, and same-category negatives are materially worse.All ablation rows use the same BGE-M3 + MNRL protocol on the 1100 split, with the negative source varied.
  • Mixed-training controls: Balanced and temperature-reweighted aggregate training leave collapsed-source margins largely unchanged while reducing toolbench margins relative to natural aggregate fine-tuning.The representation probe shows that the collapsed-source failure persists under these mixed-training controls, while compatible toolbench-style traffic also degrades.
  • Cross-source failure: 22.3% candidate coverage on ToolRet-4996 versus 73.7% for the aggregate off-the-shelf baseline explains the especially low 10.9% global score of FT on 1100.The remaining covered queries also have weaker alignment with the 1100-trained reranker, despite 48.7% covered-query MLP P@1.
  • Statistical validation: +19.0 points on the 1100 linear comparison and +19.3 points on the 4996 MLP comparison remain cleanly separated from zero under paired bootstrap.The 95% intervals are [10.8, 27.2] and [15.4, 23.1], respectively.
  • Capacity control: MNRL fine-tuning on bge-small-en-v1.5 still raises depth-20 candidate coverage and substantially improves linear and MLP direct dense reranking despite a lower absolute ceiling.The same direction across a smaller backbone suggests the intervention is broader than one encoder family or capacity point.

A.4 Additional Boundary Experiments

Additional boundary experiments show that query-only rewriting produces only partial movement toward collapsed source styles, whereas paired changes to queries and tool renderings reproduce stronger cross-source transfer loss. A MetaTool comparison further bounds the effect: arbitrary task-family partitions do not reproduce the large collapse.

  • Query-only rewrite control: 37.8–142.0% of the natural target-source TF-IDF distance gap was closed by rewriting, but downstream coverage changed much less than under natural ToolRet collapse.The rewrite also moved queries in BGE-M3 semantic space, indicating partial style transfer without reproducing the full failure.
  • Query-only rewrite control: 87.5% accuracy and 81.0 macro-F1 show that a query-only TF-IDF classifier can recover source footprints from query-side features.The classifier covered 12 source styles with at least 50 queries each.
  • Paired synthetic-source transfer: Paired synthetic-source transfer produces stronger loss than query-only rewriting: fine-tuning improves matched-source coverage but loses coverage cross-source.The Qwen/MirrorAPI cross-generator variant also falls below the off-the-shelf retriever in the mismatched direction.
  • MetaTool task-family comparison: 79.0% overall coverage for off-the-shelf BGE-M3 and at least 88.5% in every reported cross-family cell show no ToolRet-style collapse across MetaTool task families.This supports attributing the large ToolRet collapse to independent source-generation effects over a shared tool corpus rather than arbitrary task partitions.

A.5 Skill-Card Rendering Check … A.9 Hierarchy Efficiency and Benchmark-Native Execution

The appendix shows that source-style transfer collapse persists across skill-card and stricter MCP-lite renderings, while routing and hierarchy methods improve retrieval coverage or execution efficiency under specific conditions. Depth, staged-retrieval, and benchmark-native checks further characterize quality–coverage and efficiency tradeoffs.

  • A.5 Skill-Card Rendering Check: Skill-card rerendering preserves matched-source usefulness, severe FT-1100 transfer collapse, aggregate-checkpoint stability, and effective TF-IDF switching while changing only visible tool text.The check keeps tool identities, labels, splits, retriever backbones, fine-tuning, and routing fixed; only tool text is rerendered.
  • A.5 Skill-Card Rendering Check: 85.2% mixed-4996 coverage follows from a one-threshold TF-IDF switch, versus 21.7% under FT-1100; adding a base fallback gives 86.0%.Weighted TF-IDF distance correlates with FT-1100 coverage at r=-0.851, compared with r=-0.576 for weighted semantic distance.
  • A.5 Skill-Card Rendering Check: MCP-lite paired retrain/evaluation leaves narrow-checkpoint collapse in place and lowers off-the-shelf coverage, indicating schema normalization alone does not explain the failure.MCP-lite retains name, cleaned description, and fixed-order arguments fields while removing source-specific metadata and boilerplate.
  • A.6 Depth Sweep Details: Increasing retrieval depth raises coverage, but covered-query quality decays enough that the global top-1 proxy peaks at the 20-candidate operating point.These are the exact depth-sweep values underlying Figure 3.
  • A.7 Qualitative Case Studies: Same-corpus source-style shifts can cause collapse despite nearly literal gold-tool names, while compositional requests motivate staged retrieval.The qualitative cases contrast compact API-template phrasing with longer task-description supervision and show a Yosemite request bundling multiple tasks.
  • A.7 Qualitative Case Studies: 33.5% versus 33.2% is the aggregate improvement from substituting the aggregate checkpoint in the staged MLP, while held-out 1100 staged evaluation falls from 58.9% to 22.2%.Rerankers were retrained on corresponding staged merged-pool datasets, so the weak transfer is deeper than an unfitted second stage.
  • A.8 Split-and-Merge Follow-Up Metrics: Learned decomposition and adaptive first-stage-trigger variants are directionally consistent with the main split-and-merge result, which remains a conditional extension around the main collapse result.The follow-up uses the same semantic plus semantic_rerank setup and fixed 40-candidate budget, changing only the reranker.
  • A.9 Hierarchy Efficiency and Benchmark-Native Execution: 32.8% versus 31.7% overall execution accuracy favors auto_skill_tool over the official exposure baseline, while token cost per solved task is 17.7k versus 18.3k.The hierarchy uses 46 clusters matching the benchmark’s 46 manual categories; the paired common subset contains 259 queries across six groups.
Loading 2608.16502v1…